When moving a file rclone removed the file from its old parent by
looking the parent ID up from the path using the directory cache. When
the source contained two directories with the same name and path, the
cache could resolve to the wrong duplicate, so the removeParents
request didn't match the file's real parent.
This left the file with both its old and new parent, which fails on
shared drives with:
A shared drive item must have exactly one parent., teamDrivesParentLimit
This uses the source object's actual parent ID instead when it is
known, falling back to the path lookup only when the object has zero
or multiple parents.
At least on my macOS Sequoia 15.7.4, the system automatically adds a
com.apple.provenance xattr to files created by processes. This xattr
lacks the "user." prefix so getXattr filters it out, but the metadata
map was already allocated, resulting in an empty non-nil map being
returned instead of nil.
This caused TestMetadata/Symlink/Xattr and TestMetadata/File/Xattr
to fail because they assert the return value is nil when no user
xattrs are present.
The fix checks if the metadata map is empty after filtering and
returns nil if so.
Drime's folder rename returns success immediately but the children
listing of the renamed folder briefly returns empty before settling,
which made VFS see an empty directory right after a successful rename
and broke subsequent operations on its contents.
After a successful DirMove, poll the renamed folder's listing until it
shows the pre-rename child count, with a 30s timeout and exponential
backoff. The backend integration test already tolerated this via its
own list-consistency retries; this brings VFS into line.
Fixes#9450
Large files (sent as multipart uploads) were placed in the wrong folder
for two reasons:
- the parent folder was sent as "parent_id", but the API ignores that
and expects "parentId", so the parent was never honoured
- relativePath was sent as the full path from the drive root, which made
the server build folders from it and silently drop any "0" path
segment (e.g. ".../data/0/file" lost the "0")
Send the parent as "parentId" and use just the leaf as relativePath,
matching the working single-part upload. This also lets us remove the
now-unneeded absolute-path resolution code.
Fixes#9392
Co-authored-by: Brian King <BrianDKing@gmail.com>
The drime origin returns a malformed response (reported by Cloudflare as
a 520 error) for a literal PUT request to the file-entries update
endpoint, which broke renaming, and so server-side copy and move.
Use a POST with the X-HTTP-Method-Override: PUT header instead - the API
routes this to the same handler and it works reliably.
Also retry Cloudflare 520-524 errors which may occur transiently.
shouldRetry was a stub returning false unconditionally, which makes
protondrive the only rclone backend that disables pacer-level retries
entirely. Every other backend at minimum falls back to
fserrors.ShouldRetry(err) so genuine transport-level transients (TCP
resets, brief 5xx) get retried.
- Use errors.As to unwrap proton.APIError instead of string matching
- Retry transient storage block errors (Code=200501)
- Retry server errors (5xx, except 503)
- Skip 429 and 503 (handled by go-proton-api's resty retry layer
via catchTooManyRequests / catchRetryAfter, which honours Retry-After)
- Fall back to fserrors.ShouldRetry for non-API errors
Co-authored-by: tomholford <tomholford@users.noreply.github.com>
MEGA S4 has launched a new dedicated domain megas4.com with three new
endpoints, including their first Asia-Pacific location in Tokyo, plus
new endpoints in Paris and Barcelona. The existing Amsterdam,
Luxembourg, Montreal and Vancouver locations have been renamed onto
the new domain.
All current s4.mega.io endpoints continue to be supported and are
retained as legacy options. New accounts use the megas4.com endpoints.
Add tenant_url configuration option for accessing OneDrive/SharePoint
without admin privileges. When tenant_url is set, rclone uses the
SharePoint v2.0 API endpoint instead of the standard Microsoft Graph
API.
Co-Authored-By: KyokoMiki <161209740+kyokomiki@users.noreply.github.com>
Phonero is a Norwegian mobile provider focusing on the enterprise
market. They provide rebranded Jottacloud service through their «Phonero
Sky» (Norwegian for «Phonero Cloud»).
The Client ID `desktop` used by most of the other whitelabel services
does not work for Phonero Sky. Both `desktop-win` and `desktop-mac`
work, though. Authentication does not work when the `jotta-default`
scope is present, so drop it. This results in a Client ID and scope
setup that is identical with that of Let's Go Cloud (Germany).
When using the drime backend to access a folder shared from another
Drime account (via root_folder_id config option), listing the folder's
children fails with HTTP 500. This makes cross-account read-only
mounts non-functional.
This fixes the problem by changing listAll to use `folderId` instead
of `parentIds[]`
Fixes#9420
When using rcat to upload a new version of a file that already existed,
the file upload would succeed. The subsequent deletion of the old file
is attempted after the upload. Drime appears to handle the deletion of
the old file automatically and returns HTTP status code 422, stating
the "The selected entry ids is invalid."
The deletion and the rcat would fail before this change. This is with
file history enabled on my Drime account.
This change detects the error and ignores it since the file has
already been deleted.
At some point Drime recommended 200M for the upload cutoff for
switching to multipart upload. However uploads have stopped working
using single part upload for 100..200Mish files.
Their docs now recommend 5M as the cutoff for multipart upload so this
changes the default.
The /s3/multipart/create and /s3/entries endpoints interpret relativePath
as an absolute path from the drive root, not relative to parent_id. When
root_folder_id was set to a non-root folder, files larger than
upload_cutoff ended up at the user's drive root instead of the configured
folder.
Resolve the absolute path of the Fs root once via GET /folders/{hash}/path
(cached on first OpenChunkWriter call) and use that to build the correct
relativePath.
Fixes#9392
When enabled, an out-of-space error during a local write returns a
fatal error that aborts the run, instead of being retried.
Without this option, ENOSPC errors are treated as retryable and
rclone may spin through the retry loop many times on a full disk
before giving up. That is fine for transient network errors but
unhelpful when the disk is genuinely full and the operator wants
the run to fail loudly. Default is off so existing behaviour is
unchanged.
Implementation follows the pattern suggested in the issue: a defer
at the top of Update wraps the error with fserrors.FatalError when
the option is on and the error is disk-full. Detection covers both
file.ErrDiskFull from the preallocate path and syscall.ENOSPC from
io.Copy or Close, via a small helper that uses fserrors.IsErrNoSpace.
Add three new regions and their endpoints for Fastly Object Storage:
- eu-west-1 (Paris)
- us-east-1 (Virginia)
- us-west-1 (Oregon)
These are distinct from the existing us-east, us-west and eu-central
endpoints, which are kept in place.
shouldRetry treated every non-nil error as retryable, so permanent
failures (auth, 4xx, not-found) burned through the LowLevelRetries
budget instead of returning fast.
This also fixes the pacer sleeps: pacer.MinSleep(1000) and
MaxSleep(10000) are time.Duration values, so they were 1µs and 10µs -
almost certainly intended as 10ms and 2s.
The test set the short idle timeout before creating the test Fs, which
made fs.NewFs fail to read the FTP welcome banner within 1s on slow CI
hosts. Restore the long timeout while NewFs dials the control
connection, then apply the short idle timeout before the upload so the
data connection still exercises the close race that shut_timeout fixes.
The previous wording "Already have a token - refresh?" was misleading
because answering yes triggers a full re-authorization flow, not an
OAuth2 refresh token grant. Updated to "Token already configured -
replace it?" to accurately describe what happens.
Also updated the SugarSync backend which has its own copy of the prompt,
and the docs for box, drive, and onedrive that reference it.
The stscreds.AssumeRoleProvider from AWS SDK Go v2 does not cache
credentials by itself. The SDK only auto-wraps providers with
aws.CredentialsCache when they are loaded via
config.LoadDefaultConfig; when assigned directly to
aws.Config.Credentials it must be wrapped manually, as documented on
stscreds.NewAssumeRoleProvider.
Without the cache, configurations using role_arn would call AssumeRole
once per S3 request, flooding STS and CloudTrail.
See: https://forum.rclone.org/t/aws-iam-roles-credentials-arent-cached/53732
When a Proton Drive file has no active revision attributes,
readMetaDataForLink returns a nil FileSystemAttrs and Object.originalSize
is left as nil. Object.Open then dereferenced this nil pointer when
calling fs.FixRangeOption, causing a SIGSEGV during copy.
Use Object.Size() instead, which already implements the correct fallback
to the link size when originalSize is unavailable.
This updates the github.com/rclone/Proton-API-Bridge package to fix a
segfault when reading files with no metadata.
Fixes#9377Fixes#9117
Previously all log output produced by Proton-API-Bridge (stdlib log)
and go-proton-api (logrus + resty's logger) bypassed rclone's
logging: it ignored -v / -vv levels and didn't reach --log-file.
Add a small adapter implementing the resty.Logger / bridge Logger
shape that calls fs.Errorf / fs.Logf / fs.Debugf, and pass it via
the new Config.Logger hook. The bridge in turn forwards the same
value to go-proton-api's WithLogger option, so HTTP-layer warnings
and the formerly-hardcoded logrus warnings inside go-proton-api
also surface through rclone's log levels.
The Proton Drive backend constructed the upstream Proton-API-Bridge
without ever passing rclone's HTTP transport. As a result none of
rclone's HTTP flags reached Proton: --dump headers, --dump bodies,
--no-check-certificate, --user-agent, --bind, --ca-cert, --header,
--tpslimit etc. all silently did nothing for this remote, and HTTP
traffic was invisible to -vv.
Pass fshttp.NewTransport(ctx) through the new Config.Transport hook on
the bridge, which forwards it to the updated go-proton-api's
WithTransport option and so to the underlying resty client.
next_page is not currently being returned on listings which is causing
the rclone listing code to go wrong. This was returned so is likely a
regression in Drime.
This changes the page counter to calculate using current_page and
last_page. last_page on the first page request is just current_page+1.
drime appears to be capping per_page to 200. as more pages are
requested, last_page increments by 1 until current_page = last_page
Add read-only iCloud Photos support to the existing iclouddrive
backend via `service = photos` config option.
Also includes auth improvements on top of #9209's SRP authentication.
**Photos features:**
- 3-level hierarchy: libraries (Personal + Shared Photo Library) →
albums → photos/videos
- server-side smart albums (All Photos, Videos, Favorites,
Screenshots, Live, Bursts, Panoramas, Slo-mo, Time-lapse, Portrait,
Long Exposure, Animated, Hidden, Recently Deleted)
- User-created albums and nested album folders
- Live Photo `.MOV` companions as first-class entries
- Edited photo versions (`-edited` suffix) and RAW alternatives
- Duplicate filename dedup for camera counter wrap collisions
- Parallel cold listing for large albums
- Delta sync via CloudKit `changes/zone` - warm listings near-instant from disk cache
- Disk cache (libraries, albums, photos) with atomic writes for crash safety
- `ChangeNotify` support for FUSE mounts via `changes/zone` polling
- `ListR` support for `--fast-list` and recursive operations
- `--metadata` support - width, height, added-time, favorite, hidden
- Fresh download URLs per file - no stale URL failures on long copies
- FUSE mount documentation with recommended flags
**Auth improvements over #9209:**
- SMS 2FA fallback for users without trusted Apple devices
- Explicit push notification request - fixes iOS/macOS 26.4+ where 409
no longer auto-pushes
- Thread safety for concurrent FUSE callers (mutexes on session and client state)
- Session endpoint caching - skips ~5s `/validate` round-trip on warm start
- `Disconnect` support - clears auth state + disk cache
- PCS cookie support for Advanced Data Protection accounts, including
trusted-device approval for PCS cookies
Built on @coughlanio's Photos PoC (Closes#8734) and @mikegillan's SRP auth (#9209).
Fixes#7982
Co-authored-by: Chris Coughlan <chris@coughlan.io>
Implement multipart upload support with configurable chunk size and concurrency options
Enable OpenChunkWriter with per-chunk encryption
Enhance multipart upload handling with new upload cutoff and error management for small files
The fix for #8241 set FileRequestIntent=Backup on
service.ClientOptions so the azfile SDK emits the
x-ms-file-request-intent header was inadvertently dropped in this
commit (released in v1.73.0)
846f193806 azureblob,azurefiles: factor the common auth into a library
This broke azurefiles with OAuth (service principal secret,
certificate, MSI, etc.) with:
400 MissingRequiredHeader: x-ms-file-request-intent
This restores it in the azurefiles SetClientOptions callback. The SDK
only emits the header for TokenCredential auth, so shared-key and SAS
paths are unaffected.
Before this change, uploading to an existing bucket on Ceph (radosgw)
could fail with:
BucketAlreadyExists: 409 Conflict
when rclone attempted to create the destination bucket (which it does
by default unless --s3-no-check-bucket is set).
The Ceph rgw S3 implementation never returns BucketAlreadyOwnedByYou;
it returns BucketAlreadyExists for every CreateBucket on an existing
bucket, even one the caller owns. With the use_already_exists quirk
set to true, rclone wraps BucketAlreadyExists as a non-retriable error
and aborts the transfer.
The Ceph provider used to set useAlreadyExists = true explicitly. When
the s3 providers were converted to YAML files in f28c83c, Ceph did not
set use_already_exists so it picked up the default of true (via
set(&opt.UseAlreadyExists, true, provider.Quirks.UseAlreadyExists)),
which matched the previous behaviour but is the wrong setting for
Ceph.
This sets use_already_exists: false for the Ceph provider so rclone
ignores BucketAlreadyExists on CreateBucket and continues with the
upload.
Side effect: this partially reverts #7871 for the Ceph provider. If a
user tries to create a bucket on Ceph that is actually owned by
someone else, rclone will no longer fail fast at CreateBucket time;
the subsequent object PUT will fail instead. This is unavoidable on
Ceph since the server does not distinguish "already owned by you" from
"owned by someone else".
When Mega.LoginWithKeys() fails to make the API request, it leaves the
object in a state where Mega.FS.root is nil because it could never query
any information about the filesystem tree. An easy way for this to
happen is if the device is not connected to the internet.
Previously, these failures would be ignored, but Fs.findRoot() on the
rclone side is written in a way that assumes the go-meta filesystem will
have a non-nil root. This leads to an immediate nil pointer dereference
when NewFs() calls Fs.findRoot().
This commit fixes the problem by making LoginWithKeys() failures hard
failures, similar to the MultiFactorLogin() path.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Replace AuthRequired bool with NoAuth bool on the rc.Call struct and
flip the auth check logic. Previously endpoints were unauthenticated
by default and had to opt in with AuthRequired: true, which led to
security vulnerabilities when developers forgot to set the flag.
Now all endpoints require authentication by default. Only explicitly
safe read-only endpoints are marked with NoAuth: true:
- rc/noop
- rc/error
- rc/list
- core/version
- core/stats
- core/group-list
- core/transferred
- core/du
- cache/stats
- vfs/list
- vfs/stats
- vfs/queue
- job/status
- job/list
See GHSA-25qr-6mpr-f7qx, GHSA-jfwf-28xr-xw6q
Some S3-compatible servers (e.g. Archiware P5) reject requests with an
empty `?delimiter=` query parameter. For recursive listings, pass `nil`
instead of a pointer to an empty string so the parameter is omitted
entirely from the request.
Fixes#9342
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download.
If --azureblob-decompress is set, this change erases the length and hash on
compressed objects so they can be downloaded successfully, at the cost
of not being able to check the length or the hash of the downloaded
object.
If --azureblob-decompress is not set the compressed files will be downloaded
as-is providing compressed objects with intact size and hash
information.
Fixes#9337
Set the User-Agent to include the APN prefix for Azure backends
(azureblob, azurefiles, onelake) to identify rclone as a Microsoft
Partner. The User-Agent is now:
APN/1.0 rclone/1.0 rclone/<version>
Add a ctx parameter to vfs.New() so callers can pass in context
carrying ConfigInfo and FilterInfo. The context is stripped of
cancellation but config and filter values are preserved into a fresh
background context.
`json:"entry_permissions"` is known to be either empty [] or of
structure {string: boolean}. This may have been a breaking API change on
Drime's side. Because EntryPermissions is not used, the type was changed
to `any` to capture both cases, otherwise we could implement custom
unmarshalling for that type.
The Global Acceleration Endpoint (cos.accelerate.myqcloud.com) of
Tencent COS does not seem to support "CreateBucket" (maybe also other
bucket management operations). Since the acceleration functionality must
be enabled per-bucket in the Tencent Cloud console, the bucket will
always exist before this endpoint is used, so this check can be safely
skipped.
Now, "no_check_bucket = true" will be auto set when using this endpoint.
Why "NewFs()": on-the-fly remotes (connection string remotes), for
example, ":s3,provider=TencentCOS,...:..." will also be fixed.
Why no unit test: I can't find a good way to test "NewFs()" without
leveraging live endpoints. I think we can extract all existing mutations
for different providers (e.g., AWS, Fastly, and Rabata) from "NewFs()"
to a new function in the future.
Some Tencent docs about this CDN endpoint:
- English: Global Acceleration Endpoint | https://www.tencentcloud.com/pt/document/product/436/40700
- Chinese: 对象存储 全球加速概述_腾讯云 | https://cloud.tencent.com/document/product/436/38866
Assisted-By: OpenCode
