mirror of
https://github.com/rclone/rclone.git
synced 2026-10-10 15:01:37 -04:00
Version v1.75.1
This commit is contained in:
Nick Craig-Wood committed
1 parent
4158f63d2f
commit
687d264b68
22 files changed
+7762
-4145
No files matched your search
@@ -1059,11 +1059,12 @@ The following backends have known issues that need more investigation:
|
||||
- [`TestBisyncRemoteLocal/normalization`](https://pub.rclone.org/integration-tests/current/huaweidrive-cmd.bisync-TestHuaweiDrive-1.txt)
|
||||
- [`TestBisyncLocalRemote/ext_paths`](https://pub.rclone.org/integration-tests/current/huaweidrive-cmd.bisync-TestHuaweiDrive-1.txt)
|
||||
- [`TestBisyncLocalRemote/extended_filenames`](https://pub.rclone.org/integration-tests/current/huaweidrive-cmd.bisync-TestHuaweiDrive-1.txt)
|
||||
- [4 more](https://pub.rclone.org/integration-tests/current/)
|
||||
- [3 more](https://pub.rclone.org/integration-tests/current/)
|
||||
- `TestPcloud` (`pcloud`)
|
||||
- [`TestBisyncRemoteRemote/check_access`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
|
||||
- [`TestBisyncRemoteRemote/rmdirs`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
|
||||
- Updated: 2026-07-31-010017
|
||||
- [`TestBisyncRemoteLocal/createemptysrcdirs`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
|
||||
- [`TestBisyncLocalRemote/resolve`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
|
||||
- [`TestBisyncRemoteRemote/createemptysrcdirs`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
|
||||
- Updated: 2026-09-04-010006
|
||||
<!--- end list_failures - DO NOT EDIT THIS SECTION - use make commanddocs --->
|
||||
|
||||
The following backends either have not been tested recently or have known issues
|
||||
|
||||
@@ -6,6 +6,149 @@ description: "Rclone Changelog"
|
||||
|
||||
# Changelog
|
||||
|
||||
## v1.75.1 - 2026-09-04
|
||||
|
||||
[See commits](https://github.com/rclone/rclone/compare/v1.75.0...v1.75.1)
|
||||
|
||||
- Security
|
||||
- archive
|
||||
- Fix zip slip path traversal in untrusted zip files GHSA-66hp-wgxq-6f5q CVE-PENDING (Nick Craig-Wood)
|
||||
- Hide any archive entry which escapes the directory being listed GHSA-66hp-wgxq-6f5q (Nick Craig-Wood)
|
||||
- Reject unsafe entry names when mounting squashfs images GHSA-66hp-wgxq-6f5q (Nick Craig-Wood)
|
||||
- Fix zip subdirectory root matching sibling directories GHSA-66hp-wgxq-6f5q (Nick Craig-Wood)
|
||||
- Fix zip entry named "." hiding every other file GHSA-66hp-wgxq-6f5q (Nick Craig-Wood)
|
||||
- Fix "directory not found" for archive paths containing "./" or "//" GHSA-66hp-wgxq-6f5q (Nick Craig-Wood)
|
||||
- build
|
||||
- Fix multiple CVEs by upgrading to go1.26.6 (Nick Craig-Wood)
|
||||
- CVE-2026-56860: net/url: quadratic complexity in resolvePath
|
||||
- CVE-2026-56858: html/template: JavaScript regexp context tracking
|
||||
- CVE-2026-56862: crypto/tls: limit handshake messages accepted post-handshake
|
||||
- CVE-2026-56853: net/http: apply ReadHeaderTimeout to unencrypted HTTP/2 check
|
||||
- CVE-2026-56859: encoding/xml: recursion depth guard during decode
|
||||
- CVE-2026-33818: encoding/asn1: enforce maximum recursion depth
|
||||
- CVE-2026-46600: net: panic parsing an invalid SVCB or HTTPS RR in dnsmessage
|
||||
- CVE-2026-39821: net/http: reject ASCII-only Punycode-encoded labels in idna
|
||||
- Update golang.org/x/crypto to v0.56.0 to fix multiple CVEs (Nick Craig-Wood)
|
||||
- CVE-2026-56854: ssh: source-address critical option not enforced for non-public-key auth callbacks
|
||||
- CVE-2026-78662: ssh: a malicious peer could flood an undecided channel's incoming requests, deadlocking the connection
|
||||
- CVE-2026-56855: ssh: a malicious peer could send crafted messages on an established channel, deadlocking the connection
|
||||
- Update golang.org/x/image to v0.45.0 to fix CVE-2026-46603 (Nick Craig-Wood)
|
||||
- CVE-2026-46603: excessive memory allocation during VP8L decoding
|
||||
- fs: Confine directory listing entries that escape the root GHSA-3vxh-3pcx-9m8q GHSA-38xv-hf3p-h7mq CVE-PENDING (Nick Craig-Wood)
|
||||
- fshttp: Don't send `--header` values to other hosts on redirect GHSA-486v-q2wf-fp2r CVE-PENDING (Nick Craig-Wood)
|
||||
- http: Don't leak configured headers to other hosts or over plaintext on redirect GHSA-486v-q2wf-fp2r CVE-PENDING (Nick Craig-Wood)
|
||||
- lib/rest: Check HTTPS downgrades against the original request on redirect GHSA-486v-q2wf-fp2r CVE-PENDING (Nick Craig-Wood)
|
||||
- local
|
||||
- Fix dir metadata escaping the root through a planted symlink GHSA-f8g7-2xjc-7mfh CVE-PENDING (Nick Craig-Wood)
|
||||
- Fix btime escaping the root via a planted symlink GHSA-f8g7-2xjc-7mfh CVE-PENDING (Nick Craig-Wood)
|
||||
- Fix panic on Range request past the end of a symlink GHSA-p6m2-r3w9-mpxw CVE-PENDING (Nick Craig-Wood)
|
||||
- serve docker
|
||||
- Reject volume names that escape the base directory GHSA-p6vx-hf7p-98j6 (Nick Craig-Wood)
|
||||
- Reject volume names resolving to the base directory itself GHSA-p6vx-hf7p-98j6 (Nick Craig-Wood)
|
||||
- Re-derive volume mountpoint from name when restoring state GHSA-p6vx-hf7p-98j6 (Nick Craig-Wood)
|
||||
- serve ftp: Fix auth-proxy sessions sharing credentials by username GHSA-c476-6w5q-jw77 CVE-PENDING (Nick Craig-Wood)
|
||||
- serve s3
|
||||
- Fix memory exhaustion from client-declared multipart part size GHSA-2p48-j3qc-rx9f CVE-PENDING (Nick Craig-Wood)
|
||||
- Reject bogus multipart part sizes in the reorder buffer GHSA-2p48-j3qc-rx9f (Nick Craig-Wood)
|
||||
- Fix auth proxy accepting any request signed with an empty secret GHSA-xwwr-4h3p-r22c CVE-PENDING (Nick Craig-Wood)
|
||||
- **NB** the auth proxy protocol for `serve s3` has changed - the proxy program is now given the access key ID as `user` and must return the secret as `_secret_access_key`
|
||||
- Fix each server accepting the `--auth-key` credentials of all the others (Nick Craig-Wood)
|
||||
- Fix misleading anonymous access log when using an auth proxy via rc GHSA-p569-5gjg-9cmj CVE-PENDING (Nick Craig-Wood)
|
||||
- serve sftp: Fix auth proxy configured via rc being silently ignored GHSA-p569-5gjg-9cmj CVE-PENDING (Nick Craig-Wood)
|
||||
- Bug Fixes
|
||||
- accounting
|
||||
- Fix memory leak on long-running rcd (nielash)
|
||||
- Fix memory leak from stats groups on long-running rcd (nielash)
|
||||
- Fix bwlimit burst overflow (Rayan Salhab)
|
||||
- bisync
|
||||
- Fix memory leak when running via the rc (nielash)
|
||||
- Fix failed transfers of empty files being recorded as synced (Nick Craig-Wood)
|
||||
- build: Make go1.26 the minimum required version as needed by golang.org/x/crypto v0.56.0 (Nick Craig-Wood)
|
||||
- config: Redact env var config values in logs (Pastalikek65)
|
||||
- doc fixes (Anton Karpov, CAOShurong, Dean Chen, Nick Craig-Wood, Recoordinate, Rodrigo Rodrigues, Shantanav Mukherjee, shaurya)
|
||||
- lib/batcher: Prevent commits racing shutdown (Loi Nguyen)
|
||||
- lib/transform: Fix panic in `truncate_keep_extension` (VXNCXNX)
|
||||
- multipart: Fix chunked uploads storing truncated objects when the source ends early (Nick Craig-Wood)
|
||||
- operations: Fix silent truncation of streaming uploads whose source ends early (Nick Craig-Wood)
|
||||
- serve
|
||||
- Fix VFS instance leaks on server startup failures and shutdown (Hakan İSMAİL)
|
||||
- Pass the client IP address to the auth proxy (am-at-enrollvb)
|
||||
- serve http: Prevent scrolling to the top on page reload (Sune Mølgaard)
|
||||
- serve nfs: Fix EIO when creating symlinks with `--vfs-links` (SillyZir)
|
||||
- serve s3
|
||||
- Fix failed uploads deleting or corrupting the object at the key (Nick Craig-Wood)
|
||||
- Fix crash when a multipart upload is aborted while a part is uploading (Nick Craig-Wood)
|
||||
- Fix modtime not being set when only mtime metadata is supplied on PUT (Nick Craig-Wood)
|
||||
- Upload all multipart uploads via the VFS so they obey `--bwlimit` and show in stats (Nick Craig-Wood)
|
||||
- Reserve the `.rclone_temp_` prefix for temporary objects (Nick Craig-Wood)
|
||||
- Clean up abandoned multipart uploads after `--multipart-expiry` (Nick Craig-Wood)
|
||||
- vfscache
|
||||
- Fix reader deadlock when the item size drops below the read offset (Dave)
|
||||
- Fix log message growing without bound on repeated write errors (Vijay Misal)
|
||||
- walk: Stop directory traversal when the context is cancelled (Rahman Yilmaz)
|
||||
- VFS
|
||||
- Synchronize poll updates with shutdown (Loi Nguyen)
|
||||
- Make poll shutdown lifecycle deterministic (Loi Nguyen)
|
||||
- Crypt
|
||||
- Fix hash mismatches with `no_data_encryption` on backends which check upload hashes (Nick Craig-Wood)
|
||||
- Fix directory names which look like versioned file names (TowyTowy)
|
||||
- Warn about directories with legacy version-like encrypted names (Nick Craig-Wood)
|
||||
- Azure Blob
|
||||
- Fix Entra ID server-side copy source authentication (Edward Klesel)
|
||||
- Fix spurious vfs cache corruption errors during chunked reads (Nick Craig-Wood)
|
||||
- Azurefiles
|
||||
- Fix zero padded files being created when the source ends early (Nick Craig-Wood)
|
||||
- Box
|
||||
- Fix truncated files being uploaded successfully when the source ends early (Rohit Behera)
|
||||
- Compress
|
||||
- Fix corrupted objects being created when the source ends early (Nick Craig-Wood)
|
||||
- Drive
|
||||
- Don't list trashed files when removing a directory into the trash (alliasgher)
|
||||
- Dropbox
|
||||
- Preserve Paper export paths on lookup (Loi Nguyen)
|
||||
- Fix context cancellation (e.g. `--max-duration` limit) not stopping in-flight requests (debaditya)
|
||||
- Fix chunked uploads of truncated files never finishing (Nick Craig-Wood)
|
||||
- Don't retry chunked upload requests when the upload has been cancelled (Nick Craig-Wood)
|
||||
- Decode received shared-file names (Sanjay Kanth A)
|
||||
- Fix ChangeNotify when the root's case differs from Dropbox's (Loi Nguyen)
|
||||
- Filelu
|
||||
- Fix truncated files being uploaded successfully when the source ends early (Nick Craig-Wood)
|
||||
- Fix duplicate root path during multipart folder creation (kingston125)
|
||||
- Huaweidrive
|
||||
- Fix truncated files being uploaded successfully when the source ends early (Rohit Behera)
|
||||
- Iclouddrive
|
||||
- Fix uploads into an app container failing with 412 (Christian De Santis)
|
||||
- Internetarchive
|
||||
- Fix corrupted files being created when the source ends early (Nick Craig-Wood)
|
||||
- Internxt
|
||||
- Persist rotated token returned by the user info call (0rangeSeaW0lf)
|
||||
- Onedrive
|
||||
- Fix 403 Forbidden for configuration personal onedrive (machsix)
|
||||
- Fall back to manual drive ID entry when drive listing fails (SillyZir)
|
||||
- Don't retry multipart upload chunk on 404 (upload session not found) (water)
|
||||
- Overview
|
||||
- Fix "internal error: no overview data found" on 32 bit architectures (Nick Craig-Wood)
|
||||
- Pikpak
|
||||
- Fix truncated files being created when the source ends early (Nick Craig-Wood)
|
||||
- Fix truncated single part uploads reported as ok when source ends early (Nick Craig-Wood)
|
||||
- Protondrive
|
||||
- Fix files uploaded with v1.75.0 not being readable in the Proton apps (Nick Craig-Wood)
|
||||
- Fix corrupted uploads after a retried upload error (Nick Craig-Wood)
|
||||
- Quatrix
|
||||
- Fix chunk upload retries and fix memory leak (Nick Craig-Wood)
|
||||
- S3
|
||||
- Update Mega endpoints (Nick Craig-Wood)
|
||||
- Treat UploadPart success without ETag as retryable error (CAOShurong)
|
||||
- Fix server side copy failing with `--s3-no-head-object` (Anatoly Tarnavsky)
|
||||
- Sia
|
||||
- Fix corrupted files being created when the source ends early (Nick Craig-Wood)
|
||||
- Smb
|
||||
- Reuse the upload connection for SetModTime (alliasgher)
|
||||
- WebDAV
|
||||
- Fix SetModTime failing and hashes missing on Nextcloud (Nick Craig-Wood)
|
||||
- Yandex
|
||||
- Fix truncated files being uploaded successfully when the source ends early (Rohit Behera)
|
||||
|
||||
## v1.75.0 - 2026-07-31
|
||||
|
||||
[See commits](https://github.com/rclone/rclone/compare/v1.74.0...v1.75.0)
|
||||
|
||||
@@ -1107,7 +1107,7 @@ rclone [flags]
|
||||
--use-json-log Use json log format
|
||||
--use-mmap Use mmap allocator (see docs)
|
||||
--use-server-modtime Use server modified time instead of object metadata
|
||||
--user-agent string Set the user-agent to a specified string (default "rclone/v1.75.0")
|
||||
--user-agent string Set the user-agent to a specified string (default "rclone/v1.75.1")
|
||||
-v, --verbose count Print lots more stuff (repeat for more)
|
||||
-V, --version Print the version number
|
||||
--webdav-auth-redirect Preserve authentication on redirect
|
||||
|
||||
@@ -231,12 +231,12 @@ rclone convmv "stories/The Quick Brown Fox!.txt" --name-transform "all,command=e
|
||||
|
||||
```console
|
||||
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}"
|
||||
// Output: stories/The Quick Brown Fox!-20260731
|
||||
// Output: stories/The Quick Brown Fox!-20260904
|
||||
```
|
||||
|
||||
```console
|
||||
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}"
|
||||
// Output: stories/The Quick Brown Fox!-2026-07-31 0340PM
|
||||
// Output: stories/The Quick Brown Fox!-2026-09-04 0450PM
|
||||
```
|
||||
|
||||
```console
|
||||
|
||||
@@ -292,6 +292,11 @@ for all `mount` and `serve` commands on macOS. For details, see [vfs-case-sensit
|
||||
|
||||
### NFS mount
|
||||
|
||||
For macOS (and other platforms where this path is supported), prefer the dedicated
|
||||
[rclone nfsmount](/commands/rclone_nfsmount/) command. It starts the NFS server and
|
||||
performs the mount for you. `rclone mount` itself still uses FUSE (macFUSE/FUSE-T)
|
||||
and does not switch to NFS via a flag.
|
||||
|
||||
This method spins up an NFS server using [serve nfs](/commands/rclone_serve_nfs/)
|
||||
command and mounts it to the specified mountpoint. If you run this in background
|
||||
mode using |--daemon|, you will need to send SIGTERM signal to the rclone process
|
||||
@@ -629,7 +634,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -684,7 +689,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
|
||||
@@ -293,6 +293,11 @@ for all `mount` and `serve` commands on macOS. For details, see [vfs-case-sensit
|
||||
|
||||
### NFS mount
|
||||
|
||||
For macOS (and other platforms where this path is supported), prefer the dedicated
|
||||
[rclone nfsmount](/commands/rclone_nfsmount/) command. It starts the NFS server and
|
||||
performs the mount for you. `rclone mount` itself still uses FUSE (macFUSE/FUSE-T)
|
||||
and does not switch to NFS via a flag.
|
||||
|
||||
This method spins up an NFS server using [serve nfs](/commands/rclone_serve_nfs/)
|
||||
command and mounts it to the specified mountpoint. If you run this in background
|
||||
mode using |--daemon|, you will need to send SIGTERM signal to the rclone process
|
||||
@@ -630,7 +635,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -685,7 +690,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
|
||||
@@ -100,7 +100,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -155,7 +155,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
|
||||
@@ -162,7 +162,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -217,7 +217,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
|
||||
@@ -93,7 +93,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -148,7 +148,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
@@ -545,9 +546,10 @@ This config generated must have this extra parameter
|
||||
|
||||
- `_root` - root to use for the backend
|
||||
|
||||
And it may have this parameter
|
||||
And it may have these parameters
|
||||
|
||||
- `_obscure` - comma separated strings for parameters to obscure
|
||||
- `_secret_access_key` - the secret for S3 access key auth (see below)
|
||||
|
||||
If password authentication was used by the client, input to the proxy
|
||||
process (on STDIN) would look similar to this:
|
||||
@@ -555,7 +557,8 @@ process (on STDIN) would look similar to this:
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"pass": "mypassword"
|
||||
"pass": "mypassword",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -565,10 +568,44 @@ proxy process (on STDIN) would look similar to this:
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
|
||||
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
If the client authenticated with an S3 access key (`rclone serve s3`),
|
||||
the client never sends its secret, only a signature made with it, so
|
||||
the input contains just the access key ID as the `user` with no `pass`
|
||||
or `public_key`:
|
||||
|
||||
```json
|
||||
{
|
||||
"user": "AKIAIOSFODNN7EXAMPLE",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
In this case the program must look up the secret access key for that
|
||||
access key ID and return it in the `_secret_access_key` field of the
|
||||
output. Rclone then uses that secret to verify the signature on the
|
||||
request, refusing the request if it does not match. This means the
|
||||
proxy program is the source of truth for both the credentials and the
|
||||
backend they map to. If the program does not return
|
||||
`_secret_access_key` or returns it empty the request is refused.
|
||||
|
||||
The program's answer for an access key ID is cached (see below) but
|
||||
is checked with the program again after 5 minutes even if the access
|
||||
key ID is in constant use, so revoking an access key ID in the
|
||||
program takes effect within 5 minutes. A rotated secret takes effect
|
||||
on the first request signed with it.
|
||||
|
||||
The `client_ip` key holds the IP address the client connected from,
|
||||
without a port number. It can be used to restrict logins to certain
|
||||
networks, or to log authentication attempts centrally. It is omitted if
|
||||
the client has no IP address, for example when connecting over a unix
|
||||
socket. Note that if rclone is behind a reverse proxy this will be the
|
||||
address of the reverse proxy and not the original client.
|
||||
|
||||
And as an example return this on STDOUT
|
||||
|
||||
```json
|
||||
@@ -594,11 +631,12 @@ to make proxy to many different sftp backends, you could make the
|
||||
in the output and the user to `user`. For security you'd probably want
|
||||
to restrict the `host` to a limited list.
|
||||
|
||||
An internal cache of backends is keyed on the `user` and a hash of the
|
||||
`pass` or `public_key`. This means that if a user's password or
|
||||
public-key changes, or the proxy returns different config parameters
|
||||
(eg a rotated `api_key`), a fresh backend will be created on the next
|
||||
request rather than the cached one being reused.
|
||||
An internal cache of backends is keyed on the `user`, a hash of the
|
||||
`pass` or `public_key`, and the `client_ip`. This means that if a
|
||||
user's password or public-key changes, the client connects from a new IP
|
||||
address, or the proxy returns different config parameters (eg a rotated
|
||||
`api_key`), a fresh backend will be created on the next request rather
|
||||
than the cached one being reused.
|
||||
|
||||
This can be used to build general purpose proxies to any kind of
|
||||
backend that rclone supports.
|
||||
|
||||
@@ -236,7 +236,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -291,7 +291,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
@@ -688,9 +689,10 @@ This config generated must have this extra parameter
|
||||
|
||||
- `_root` - root to use for the backend
|
||||
|
||||
And it may have this parameter
|
||||
And it may have these parameters
|
||||
|
||||
- `_obscure` - comma separated strings for parameters to obscure
|
||||
- `_secret_access_key` - the secret for S3 access key auth (see below)
|
||||
|
||||
If password authentication was used by the client, input to the proxy
|
||||
process (on STDIN) would look similar to this:
|
||||
@@ -698,7 +700,8 @@ process (on STDIN) would look similar to this:
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"pass": "mypassword"
|
||||
"pass": "mypassword",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -708,10 +711,44 @@ proxy process (on STDIN) would look similar to this:
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
|
||||
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
If the client authenticated with an S3 access key (`rclone serve s3`),
|
||||
the client never sends its secret, only a signature made with it, so
|
||||
the input contains just the access key ID as the `user` with no `pass`
|
||||
or `public_key`:
|
||||
|
||||
```json
|
||||
{
|
||||
"user": "AKIAIOSFODNN7EXAMPLE",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
In this case the program must look up the secret access key for that
|
||||
access key ID and return it in the `_secret_access_key` field of the
|
||||
output. Rclone then uses that secret to verify the signature on the
|
||||
request, refusing the request if it does not match. This means the
|
||||
proxy program is the source of truth for both the credentials and the
|
||||
backend they map to. If the program does not return
|
||||
`_secret_access_key` or returns it empty the request is refused.
|
||||
|
||||
The program's answer for an access key ID is cached (see below) but
|
||||
is checked with the program again after 5 minutes even if the access
|
||||
key ID is in constant use, so revoking an access key ID in the
|
||||
program takes effect within 5 minutes. A rotated secret takes effect
|
||||
on the first request signed with it.
|
||||
|
||||
The `client_ip` key holds the IP address the client connected from,
|
||||
without a port number. It can be used to restrict logins to certain
|
||||
networks, or to log authentication attempts centrally. It is omitted if
|
||||
the client has no IP address, for example when connecting over a unix
|
||||
socket. Note that if rclone is behind a reverse proxy this will be the
|
||||
address of the reverse proxy and not the original client.
|
||||
|
||||
And as an example return this on STDOUT
|
||||
|
||||
```json
|
||||
@@ -737,11 +774,12 @@ to make proxy to many different sftp backends, you could make the
|
||||
in the output and the user to `user`. For security you'd probably want
|
||||
to restrict the `host` to a limited list.
|
||||
|
||||
An internal cache of backends is keyed on the `user` and a hash of the
|
||||
`pass` or `public_key`. This means that if a user's password or
|
||||
public-key changes, or the proxy returns different config parameters
|
||||
(eg a rotated `api_key`), a fresh backend will be created on the next
|
||||
request rather than the cached one being reused.
|
||||
An internal cache of backends is keyed on the `user`, a hash of the
|
||||
`pass` or `public_key`, and the `client_ip`. This means that if a
|
||||
user's password or public-key changes, the client connects from a new IP
|
||||
address, or the proxy returns different config parameters (eg a rotated
|
||||
`api_key`), a fresh backend will be created on the next request rather
|
||||
than the cached one being reused.
|
||||
|
||||
This can be used to build general purpose proxies to any kind of
|
||||
backend that rclone supports.
|
||||
|
||||
@@ -167,7 +167,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -222,7 +222,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
|
||||
@@ -26,6 +26,12 @@ docs](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html)).
|
||||
`--auth-key` is not provided then `serve s3` will allow anonymous
|
||||
access.
|
||||
|
||||
Alternatively `--auth-proxy` can be used to look up the secret for each
|
||||
access key ID and choose the backend it maps to (see [Auth
|
||||
Proxy](#auth-proxy) below). When an auth proxy is in use `--auth-key`
|
||||
is ignored and every request must be signed with the secret the proxy
|
||||
returns for its access key ID.
|
||||
|
||||
Like all rclone flags `--auth-key` can be set via environment
|
||||
variables, in this case `RCLONE_AUTH_KEY`. Since this flag can be
|
||||
repeated, the input to `RCLONE_AUTH_KEY` is CSV encoded. Because the
|
||||
@@ -101,24 +107,73 @@ access_key_id = ACCESS_KEY_ID
|
||||
secret_access_key = SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
## Object uploads (PUT)
|
||||
|
||||
A `PutObject` upload only ever changes the object at its key atomically, on
|
||||
success, a failed or interrupted PUT neither removes nor overwrites the
|
||||
object already stored at the key, and never leaves a partial object visible
|
||||
at it.
|
||||
|
||||
Remotes that upload atomically (e.g. object stores such as `s3`) are streamed
|
||||
straight to the destination. On remotes where a partial upload would
|
||||
otherwise be visible (e.g. `local`), and whenever `--vfs-cache-mode` is
|
||||
`writes` or above, the upload is written to a temporary object that is
|
||||
renamed into place on success; these remotes need to support a server-side
|
||||
move or copy for this (nearly all do - without move or copy the upload is
|
||||
written directly and a failed PUT may leave a partial object at the key). If
|
||||
`serve s3` is killed part-way through an upload the temporary object (named
|
||||
with a leading `.rclone_temp_put_`) may be left behind; it is hidden from
|
||||
S3 listings but must be removed manually.
|
||||
|
||||
## Multipart uploads
|
||||
|
||||
By default `serve s3` **streams** each multipart upload, in part-number
|
||||
order, into a single `PutStream` upload to the underlying remote, so the
|
||||
whole file is never buffered in memory - memory use stays bounded by the
|
||||
parts in flight. The remote then performs its own internal upload (for
|
||||
example its own multipart upload, still with bounded memory). This works
|
||||
for any remote that supports `PutStream`, which is nearly all of them,
|
||||
including through `crypt`.
|
||||
|
||||
The upload is atomic so the destination object only ever changes on a
|
||||
Multipart uploads are written, in part-number order, to a temporary
|
||||
object which is renamed into place, server-side, on completion, so the
|
||||
upload is atomic. The object at the key only ever changes on a
|
||||
successful completion. A failed or aborted upload never affects any
|
||||
object already stored under that name. Remotes that upload atomically
|
||||
already (object stores such as `s3`) are streamed straight to the
|
||||
destination. On remotes where a partial upload would otherwise be visible
|
||||
(such as `local`), the parts are streamed to a temporary object that is
|
||||
moved into place, server-side, on completion; these remotes therefore
|
||||
also need to support a server-side move or copy.
|
||||
object already stored under that name and a partly-uploaded object
|
||||
never becomes visible under it.
|
||||
|
||||
With the default `--vfs-cache-mode off` `serve s3` **streams** each
|
||||
multipart upload, in part-number order, into a single streaming upload
|
||||
to the underlying remote, so the whole file is never buffered in
|
||||
memory. Memory use stays bounded by the parts in flight. The remote
|
||||
then performs its own internal upload (for example its own multipart
|
||||
upload, still with bounded memory). Remotes that don't support
|
||||
streaming uploads (those that must know the file size before the
|
||||
upload starts, such as `onedrive`, `pcloud`, `jottacloud`, `mailru`,
|
||||
`opendrive`, `putio`, `protondrive` and `zoho`) have the parts spooled
|
||||
to a temporary file on **local disk** instead, and uploaded with the
|
||||
size then known on completion, so they need local disk space for the
|
||||
largest objects in flight rather than memory.
|
||||
|
||||
With `--vfs-cache-mode writes` (or `full`) the parts are written to a
|
||||
temporary file in the VFS cache and uploaded by the VFS write-back -
|
||||
see [Multipart uploads and the VFS
|
||||
cache](#multipart-uploads-and-the-vfs-cache) below.
|
||||
|
||||
The rename into place needs the remote to support a server-side move
|
||||
or copy, which nearly all do. It is a cheap rename on most remotes,
|
||||
but on object stores without a real rename (such as `s3` itself) the
|
||||
move is performed as a server-side copy and delete of the whole
|
||||
object, which can take time and API calls for large objects.
|
||||
Concurrent multipart uploads of the same key (which S3 permits) are
|
||||
safe. Each writes its own temporary object and the last to complete
|
||||
wins.
|
||||
|
||||
On the few remotes that support neither server side move nor copy, the
|
||||
parts are written straight to the destination object instead and never
|
||||
buffered in memory. This is at some cost in atomicity - the incomplete
|
||||
object is visible under its final name while the upload is in flight,
|
||||
as it also is for a plain object PUT on such remotes, and concurrent
|
||||
multipart uploads of the same key write to the same object and can
|
||||
interleave. A failed or aborted upload still leaves any pre-existing
|
||||
object untouched provided the remote uploads atomically and the VFS
|
||||
cache is off; on a remote where partial uploads are visible it may
|
||||
leave partial data at the key (like a plain PUT there), and with
|
||||
`--vfs-cache-mode writes` (or `full`) a write to the cache cannot be
|
||||
abandoned, so an aborted upload's partial data is written back to the
|
||||
remote as if it had completed.
|
||||
|
||||
**Features**
|
||||
|
||||
@@ -133,10 +188,14 @@ also need to support a server-side move or copy.
|
||||
as one continuous stream.
|
||||
- The destination object only ever changes atomically, on completion: an
|
||||
aborted or failed upload leaves any pre-existing object of the same
|
||||
name untouched, and a partly-uploaded object never becomes visible.
|
||||
- Backend-agnostic - it only needs the remote to support `PutStream`
|
||||
(plus a server-side move or copy on remotes that don't upload
|
||||
atomically).
|
||||
name untouched, and a partly-uploaded object never becomes visible
|
||||
(except on the few remotes with no server-side move or copy, as
|
||||
above).
|
||||
- Multipart uploads go through the VFS like any other upload, so they
|
||||
show in rclone's transfer stats and obey `--bwlimit`.
|
||||
- Backend-agnostic - it only needs the remote to support a server-side
|
||||
move or copy for the rename into place, which nearly all do; a remote
|
||||
without streaming upload support spools to local disk as above.
|
||||
|
||||
**Limitations**
|
||||
|
||||
@@ -162,31 +221,121 @@ also need to support a server-side move or copy.
|
||||
upload and the client must start it again. (The remote's own upload
|
||||
still retries its internal chunks.)
|
||||
- Parts are serialised into one stream, so ingest from the client is
|
||||
effectively single-threaded, although the remote's own upload still
|
||||
runs concurrently.
|
||||
- On remotes that don't upload atomically (such as `local`), the
|
||||
completed object is moved into place with a server-side operation.
|
||||
This is a cheap rename on most such remotes. On these remotes, if
|
||||
`serve s3` is killed part-way through an upload the temporary object
|
||||
(named with a leading `.rclone_multipart_upload_`) may be left behind;
|
||||
it is hidden from S3 listings but must be removed manually.
|
||||
effectively single-threaded. When streaming, the remote's own upload
|
||||
runs concurrently with the parts arriving; with the local disk spool
|
||||
or the VFS cache the upload to the remote only starts on completion.
|
||||
- If `serve s3` is killed part-way through an upload the temporary
|
||||
object (named with a leading `.rclone_temp_multipart_`) may be left
|
||||
behind; it is hidden from S3 listings but must be removed manually.
|
||||
|
||||
### Multipart uploads and the VFS cache
|
||||
|
||||
With `--vfs-cache-mode writes` (or `full`) multipart uploads do not
|
||||
stream to the remote at all. The parts are written, in part-number
|
||||
order, to a temporary file in the VFS cache. On completion the file is
|
||||
renamed into place and uploaded by the VFS write-back, exactly like a
|
||||
plain object PUT. This needs no streaming upload support from the
|
||||
remote. The rename normally happens in the cache before the upload has
|
||||
started, but the VFS requires the remote to support a server-side move
|
||||
or copy to rename files at all (and uses one if the temporary file has
|
||||
already been written back, e.g. with `--vfs-write-back 0`). On remotes
|
||||
without either, the parts are written to the cache directly under the
|
||||
final key instead: the upload still never touches memory, but it loses
|
||||
its atomicity - the in-flight upload is visible at the key, and an
|
||||
aborted upload cannot be abandoned once in the cache, so its partial
|
||||
data is written back to the remote as if it were a completed object.
|
||||
|
||||
Remotes that benefit from `--vfs-cache-mode writes`:
|
||||
|
||||
- **Remotes over slow or unreliable links.** A failure in a streamed
|
||||
upload aborts the whole multipart upload and the client must start
|
||||
again from the first part; a failed write-back upload is retried by
|
||||
the VFS (see `--vfs-cache-max-age` and friends) without the client
|
||||
being involved. Ingest from the client also runs at local disk speed
|
||||
rather than being throttled to the remote's pace.
|
||||
- **Workloads that read back or overwrite what they just wrote.** The
|
||||
completed object stays in the cache, so subsequent `GET`/`HEAD`
|
||||
requests are served locally, and plain PUTs and multipart uploads to
|
||||
the same key go through the same cache entry so the last write wins
|
||||
regardless of upload style.
|
||||
|
||||
The trade-offs of the VFS cache:
|
||||
|
||||
- The whole object lands on local disk, so the cache (`--cache-dir`)
|
||||
needs space for the largest objects in flight; `--vfs-cache-max-size`
|
||||
cannot evict files which are still being uploaded.
|
||||
- The `200 OK` for `CompleteMultipartUpload` means the data is safely
|
||||
in the **local cache**, not yet on the remote - the same durability
|
||||
the cache gives plain PUTs. If an acknowledgement must mean the data
|
||||
has reached the remote (for example WAL archiving), use the default
|
||||
`--vfs-cache-mode off`.
|
||||
- The upload to the remote only starts on completion, rather than
|
||||
overlapping with the parts arriving, so the data reaches the remote
|
||||
later than with streaming.
|
||||
- If `serve s3` is killed part-way through an upload, the temporary
|
||||
file survives in the cache and the VFS cache recovery uploads it to
|
||||
the remote on restart as a temporary object (named with a leading
|
||||
`.rclone_temp_multipart_`); as with the streaming path, it is
|
||||
hidden from S3 listings but must be removed manually.
|
||||
|
||||
### Cleaning up temporary objects
|
||||
|
||||
If `serve s3` is killed part-way through an upload it can leave a
|
||||
temporary object behind, named with a leading `.rclone_temp_`. This
|
||||
whole prefix is reserved: any object whose name (the last
|
||||
`/`-separated segment of its key) starts with `.rclone_temp_` is
|
||||
hidden from S3 listings, so don't give real objects such names - an
|
||||
existing object with such a name disappears from listings (though it
|
||||
stays accessible directly by its key: only listings hide reserved
|
||||
names, `GET`, `HEAD` and `DELETE` of the exact key still work). A
|
||||
temporary object never holds acknowledged data - uploads whose
|
||||
temporary object survived were never confirmed to the client - so old
|
||||
ones are safe to delete:
|
||||
|
||||
rclone delete --min-age 24h --include ".rclone_temp_*" remote:path
|
||||
|
||||
The `--min-age` protects uploads which are still in progress: make sure
|
||||
it is longer than your longest upload, especially if several `serve s3`
|
||||
instances share the same remote.
|
||||
|
||||
rclone v1.75 named its temporary multipart objects
|
||||
`.rclone_multipart_upload_*`; leftovers from an older server are also
|
||||
hidden from listings and can be cleaned up the same way.
|
||||
|
||||
### Abandoned uploads
|
||||
|
||||
A client which starts a multipart upload and vanishes without either
|
||||
completing or aborting it would otherwise hold on to its resources
|
||||
forever.
|
||||
|
||||
An incomplete multipart upload which has had no activity for
|
||||
`--multipart-expiry` (default `24h`) is therefore aborted and cleaned
|
||||
up, exactly as if the client had called `AbortMultipartUpload`, and a
|
||||
`NOTICE` is logged.
|
||||
|
||||
An upload with a part still being received is never expired, however
|
||||
slowly the part is arriving, and each completed part restarts the
|
||||
clock, so the expiry only needs to outlast the client's pauses
|
||||
*between* parts, not the whole upload.
|
||||
|
||||
Late operations on an expired upload fail with `NoSuchUpload`, as they
|
||||
do on real S3 when a lifecycle rule has aborted the upload. Set
|
||||
`--multipart-expiry 0` to keep incomplete uploads forever.
|
||||
|
||||
### Disabling streaming
|
||||
|
||||
If you pass `--disable-multipart-streaming`, or the remote doesn't
|
||||
support `PutStream` (or doesn't upload atomically and can't move or copy
|
||||
server-side), multipart uploads are instead **buffered in memory**
|
||||
by the underlying S3 library: every part is held in memory and the whole
|
||||
object is written out in one go when the upload completes (the previous
|
||||
behaviour). This removes the in-order/contiguous-part restriction above,
|
||||
so parts can be uploaded in any order, but **memory use grows with the
|
||||
size of the upload**, so it is only suitable for small objects. A one-off
|
||||
`NOTICE` is logged the first time this happens.
|
||||
|
||||
Alternatively, if the client is an rclone `s3` remote (like the
|
||||
`[serves3]` example above), you can set `use_multipart_uploads = false`
|
||||
on it so it uploads each object as a single stream and skips multipart
|
||||
uploads altogether.
|
||||
If you pass `--disable-multipart-streaming`, multipart uploads are
|
||||
instead **buffered in memory** by the underlying S3 library: every
|
||||
part is held in memory and the whole object is written out in one go
|
||||
when the upload completes. This removes the in-order/contiguous-part
|
||||
restriction above, so parts can be uploaded in any order, but **memory
|
||||
use grows with the size of the upload**, so it is only suitable for
|
||||
small objects. A one-off `NOTICE` is logged the first time this
|
||||
happens. This flag is the only thing that makes multipart uploads
|
||||
buffer in memory - it is never done because of missing remote
|
||||
capabilities. Consider `--vfs-cache-mode writes` instead, which
|
||||
buffers the upload in the VFS cache on disk and takes precedence over
|
||||
`--disable-multipart-streaming`.
|
||||
|
||||
## Bugs
|
||||
|
||||
@@ -436,7 +585,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -491,7 +640,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
@@ -862,6 +1012,127 @@ If the file has no metadata it will be returned as `{}` and if there
|
||||
is an error reading the metadata the error will be returned as
|
||||
`{"error":"error string"}`.
|
||||
|
||||
## Auth Proxy
|
||||
|
||||
If you supply the parameter `--auth-proxy /path/to/program` then
|
||||
rclone will use that program to generate backends on the fly which
|
||||
then are used to authenticate incoming requests. This uses a simple
|
||||
JSON based protocol with input on STDIN and output on STDOUT.
|
||||
|
||||
**PLEASE NOTE:** `--auth-proxy` and `--authorized-keys` cannot be used
|
||||
together, if `--auth-proxy` is set the authorized keys option will be
|
||||
ignored.
|
||||
|
||||
There is an example program
|
||||
[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/bin/test_proxy.py)
|
||||
in the rclone source code.
|
||||
|
||||
The program's job is to take a `user` and `pass` on the input and turn
|
||||
those into the config for a backend on STDOUT in JSON format. This
|
||||
config will have any default parameters for the backend added, but it
|
||||
won't use configuration from environment variables or command line
|
||||
options - it is the job of the proxy program to make a complete
|
||||
config.
|
||||
|
||||
This config generated must have this extra parameter
|
||||
|
||||
- `_root` - root to use for the backend
|
||||
|
||||
And it may have these parameters
|
||||
|
||||
- `_obscure` - comma separated strings for parameters to obscure
|
||||
- `_secret_access_key` - the secret for S3 access key auth (see below)
|
||||
|
||||
If password authentication was used by the client, input to the proxy
|
||||
process (on STDIN) would look similar to this:
|
||||
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"pass": "mypassword",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
If public-key authentication was used by the client, input to the
|
||||
proxy process (on STDIN) would look similar to this:
|
||||
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
If the client authenticated with an S3 access key (`rclone serve s3`),
|
||||
the client never sends its secret, only a signature made with it, so
|
||||
the input contains just the access key ID as the `user` with no `pass`
|
||||
or `public_key`:
|
||||
|
||||
```json
|
||||
{
|
||||
"user": "AKIAIOSFODNN7EXAMPLE",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
In this case the program must look up the secret access key for that
|
||||
access key ID and return it in the `_secret_access_key` field of the
|
||||
output. Rclone then uses that secret to verify the signature on the
|
||||
request, refusing the request if it does not match. This means the
|
||||
proxy program is the source of truth for both the credentials and the
|
||||
backend they map to. If the program does not return
|
||||
`_secret_access_key` or returns it empty the request is refused.
|
||||
|
||||
The program's answer for an access key ID is cached (see below) but
|
||||
is checked with the program again after 5 minutes even if the access
|
||||
key ID is in constant use, so revoking an access key ID in the
|
||||
program takes effect within 5 minutes. A rotated secret takes effect
|
||||
on the first request signed with it.
|
||||
|
||||
The `client_ip` key holds the IP address the client connected from,
|
||||
without a port number. It can be used to restrict logins to certain
|
||||
networks, or to log authentication attempts centrally. It is omitted if
|
||||
the client has no IP address, for example when connecting over a unix
|
||||
socket. Note that if rclone is behind a reverse proxy this will be the
|
||||
address of the reverse proxy and not the original client.
|
||||
|
||||
And as an example return this on STDOUT
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "sftp",
|
||||
"_root": "",
|
||||
"_obscure": "pass",
|
||||
"user": "me",
|
||||
"pass": "mypassword",
|
||||
"host": "sftp.example.com"
|
||||
}
|
||||
```
|
||||
|
||||
This would mean that an SFTP backend would be created on the fly for
|
||||
the `user` and `pass`/`public_key` returned in the output to the host given. Note
|
||||
that since `_obscure` is set to `pass`, rclone will obscure the `pass`
|
||||
parameter before creating the backend (which is required for sftp
|
||||
backends).
|
||||
|
||||
The program can manipulate the supplied `user` in any way, for example
|
||||
to make proxy to many different sftp backends, you could make the
|
||||
`user` be `user@example.com` and then set the `host` to `example.com`
|
||||
in the output and the user to `user`. For security you'd probably want
|
||||
to restrict the `host` to a limited list.
|
||||
|
||||
An internal cache of backends is keyed on the `user`, a hash of the
|
||||
`pass` or `public_key`, and the `client_ip`. This means that if a
|
||||
user's password or public-key changes, the client connects from a new IP
|
||||
address, or the proxy returns different config parameters (eg a rotated
|
||||
`api_key`), a fresh backend will be created on the next request rather
|
||||
than the cached one being reused.
|
||||
|
||||
This can be used to build general purpose proxies to any kind of
|
||||
backend that rclone supports.
|
||||
|
||||
```
|
||||
rclone serve s3 remote:path [flags]
|
||||
```
|
||||
@@ -878,7 +1149,7 @@ rclone serve s3 remote:path [flags]
|
||||
--client-ca string Client certificate authority to verify clients with
|
||||
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
|
||||
--dir-perms FileMode Directory permissions (default 777)
|
||||
--disable-multipart-streaming Buffer multipart uploads in memory instead of streaming them to the backend (see the Multipart uploads docs section)
|
||||
--disable-multipart-streaming Buffer multipart uploads in memory instead of streaming them to the backend
|
||||
--etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5")
|
||||
--file-perms FileMode File permissions (default 666)
|
||||
--force-path-style If true use path style access if false use virtual hosted style (default true)
|
||||
@@ -889,7 +1160,8 @@ rclone serve s3 remote:path [flags]
|
||||
--link-perms FileMode Link permissions (default 666)
|
||||
--max-header-bytes int Maximum size of request header (default 4096)
|
||||
--min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
|
||||
--multipart-streaming-buffer-limit SizeSuffix Maximum memory buffered per streamed multipart upload for parts arriving out of order, 0 for unlimited (see the Multipart uploads docs section) (default 256Mi)
|
||||
--multipart-expiry Duration Abort incomplete multipart uploads idle for longer than this, 0 to keep forever (default 1d)
|
||||
--multipart-streaming-buffer-limit SizeSuffix Maximum memory buffered per streamed multipart upload for parts arriving out of order, 0 for unlimited (default 256Mi)
|
||||
--no-checksum Don't compare checksums on up/download
|
||||
--no-cleanup Not to cleanup empty folder after object is deleted
|
||||
--no-modtime Don't read/write the modification time (can speed things up)
|
||||
|
||||
@@ -140,7 +140,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -195,7 +195,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
@@ -592,9 +593,10 @@ This config generated must have this extra parameter
|
||||
|
||||
- `_root` - root to use for the backend
|
||||
|
||||
And it may have this parameter
|
||||
And it may have these parameters
|
||||
|
||||
- `_obscure` - comma separated strings for parameters to obscure
|
||||
- `_secret_access_key` - the secret for S3 access key auth (see below)
|
||||
|
||||
If password authentication was used by the client, input to the proxy
|
||||
process (on STDIN) would look similar to this:
|
||||
@@ -602,7 +604,8 @@ process (on STDIN) would look similar to this:
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"pass": "mypassword"
|
||||
"pass": "mypassword",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -612,10 +615,44 @@ proxy process (on STDIN) would look similar to this:
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
|
||||
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
If the client authenticated with an S3 access key (`rclone serve s3`),
|
||||
the client never sends its secret, only a signature made with it, so
|
||||
the input contains just the access key ID as the `user` with no `pass`
|
||||
or `public_key`:
|
||||
|
||||
```json
|
||||
{
|
||||
"user": "AKIAIOSFODNN7EXAMPLE",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
In this case the program must look up the secret access key for that
|
||||
access key ID and return it in the `_secret_access_key` field of the
|
||||
output. Rclone then uses that secret to verify the signature on the
|
||||
request, refusing the request if it does not match. This means the
|
||||
proxy program is the source of truth for both the credentials and the
|
||||
backend they map to. If the program does not return
|
||||
`_secret_access_key` or returns it empty the request is refused.
|
||||
|
||||
The program's answer for an access key ID is cached (see below) but
|
||||
is checked with the program again after 5 minutes even if the access
|
||||
key ID is in constant use, so revoking an access key ID in the
|
||||
program takes effect within 5 minutes. A rotated secret takes effect
|
||||
on the first request signed with it.
|
||||
|
||||
The `client_ip` key holds the IP address the client connected from,
|
||||
without a port number. It can be used to restrict logins to certain
|
||||
networks, or to log authentication attempts centrally. It is omitted if
|
||||
the client has no IP address, for example when connecting over a unix
|
||||
socket. Note that if rclone is behind a reverse proxy this will be the
|
||||
address of the reverse proxy and not the original client.
|
||||
|
||||
And as an example return this on STDOUT
|
||||
|
||||
```json
|
||||
@@ -641,11 +678,12 @@ to make proxy to many different sftp backends, you could make the
|
||||
in the output and the user to `user`. For security you'd probably want
|
||||
to restrict the `host` to a limited list.
|
||||
|
||||
An internal cache of backends is keyed on the `user` and a hash of the
|
||||
`pass` or `public_key`. This means that if a user's password or
|
||||
public-key changes, or the proxy returns different config parameters
|
||||
(eg a rotated `api_key`), a fresh backend will be created on the next
|
||||
request rather than the cached one being reused.
|
||||
An internal cache of backends is keyed on the `user`, a hash of the
|
||||
`pass` or `public_key`, and the `client_ip`. This means that if a
|
||||
user's password or public-key changes, the client connects from a new IP
|
||||
address, or the proxy returns different config parameters (eg a rotated
|
||||
`api_key`), a fresh backend will be created on the next request rather
|
||||
than the cached one being reused.
|
||||
|
||||
This can be used to build general purpose proxies to any kind of
|
||||
backend that rclone supports.
|
||||
|
||||
@@ -313,7 +313,7 @@ at all times. The buffered data is bound to one open file and won't be
|
||||
shared.
|
||||
|
||||
This flag is a upper limit for the used memory per open file. The
|
||||
buffer will only use memory for data that is downloaded but not not
|
||||
buffer will only use memory for data that is downloaded but not
|
||||
yet read. If the buffer is empty, only a small amount of memory will
|
||||
be used.
|
||||
|
||||
@@ -368,7 +368,8 @@ longest. This cache flushing strategy is efficient and more relevant
|
||||
files are likely to remain cached.
|
||||
|
||||
The `--vfs-cache-max-age` will evict files from the cache
|
||||
after the set time since last access has passed. The default value of
|
||||
after the set time since last access has passed; it is based on access time,
|
||||
not on when the file was first added to the cache. The default value of
|
||||
1 hour will start evicting files from cache that haven't been accessed
|
||||
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
|
||||
and will wait for 1 more hour before evicting. Specify the time with
|
||||
@@ -765,9 +766,10 @@ This config generated must have this extra parameter
|
||||
|
||||
- `_root` - root to use for the backend
|
||||
|
||||
And it may have this parameter
|
||||
And it may have these parameters
|
||||
|
||||
- `_obscure` - comma separated strings for parameters to obscure
|
||||
- `_secret_access_key` - the secret for S3 access key auth (see below)
|
||||
|
||||
If password authentication was used by the client, input to the proxy
|
||||
process (on STDIN) would look similar to this:
|
||||
@@ -775,7 +777,8 @@ process (on STDIN) would look similar to this:
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"pass": "mypassword"
|
||||
"pass": "mypassword",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -785,10 +788,44 @@ proxy process (on STDIN) would look similar to this:
|
||||
```json
|
||||
{
|
||||
"user": "me",
|
||||
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
|
||||
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
If the client authenticated with an S3 access key (`rclone serve s3`),
|
||||
the client never sends its secret, only a signature made with it, so
|
||||
the input contains just the access key ID as the `user` with no `pass`
|
||||
or `public_key`:
|
||||
|
||||
```json
|
||||
{
|
||||
"user": "AKIAIOSFODNN7EXAMPLE",
|
||||
"client_ip": "192.168.1.1"
|
||||
}
|
||||
```
|
||||
|
||||
In this case the program must look up the secret access key for that
|
||||
access key ID and return it in the `_secret_access_key` field of the
|
||||
output. Rclone then uses that secret to verify the signature on the
|
||||
request, refusing the request if it does not match. This means the
|
||||
proxy program is the source of truth for both the credentials and the
|
||||
backend they map to. If the program does not return
|
||||
`_secret_access_key` or returns it empty the request is refused.
|
||||
|
||||
The program's answer for an access key ID is cached (see below) but
|
||||
is checked with the program again after 5 minutes even if the access
|
||||
key ID is in constant use, so revoking an access key ID in the
|
||||
program takes effect within 5 minutes. A rotated secret takes effect
|
||||
on the first request signed with it.
|
||||
|
||||
The `client_ip` key holds the IP address the client connected from,
|
||||
without a port number. It can be used to restrict logins to certain
|
||||
networks, or to log authentication attempts centrally. It is omitted if
|
||||
the client has no IP address, for example when connecting over a unix
|
||||
socket. Note that if rclone is behind a reverse proxy this will be the
|
||||
address of the reverse proxy and not the original client.
|
||||
|
||||
And as an example return this on STDOUT
|
||||
|
||||
```json
|
||||
@@ -814,11 +851,12 @@ to make proxy to many different sftp backends, you could make the
|
||||
in the output and the user to `user`. For security you'd probably want
|
||||
to restrict the `host` to a limited list.
|
||||
|
||||
An internal cache of backends is keyed on the `user` and a hash of the
|
||||
`pass` or `public_key`. This means that if a user's password or
|
||||
public-key changes, or the proxy returns different config parameters
|
||||
(eg a rotated `api_key`), a fresh backend will be created on the next
|
||||
request rather than the cached one being reused.
|
||||
An internal cache of backends is keyed on the `user`, a hash of the
|
||||
`pass` or `public_key`, and the `client_ip`. This means that if a
|
||||
user's password or public-key changes, the client connects from a new IP
|
||||
address, or the proxy returns different config parameters (eg a rotated
|
||||
`api_key`), a fresh backend will be created on the next request rather
|
||||
than the cached one being reused.
|
||||
|
||||
This can be used to build general purpose proxies to any kind of
|
||||
backend that rclone supports.
|
||||
|
||||
@@ -121,7 +121,7 @@ Flags for general networking and HTTP stuff.
|
||||
--tpslimit float Limit HTTP transactions per second to this
|
||||
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
|
||||
--use-cookies Enable session cookiejar
|
||||
--user-agent string Set the user-agent to a specified string (default "rclone/v1.75.0")
|
||||
--user-agent string Set the user-agent to a specified string (default "rclone/v1.75.1")
|
||||
```
|
||||
|
||||
|
||||
|
||||
@@ -210,6 +210,12 @@ For example, to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'
|
||||
|
||||
You can set multiple headers, e.g. '"Cookie","name=value","Authorization","xxx"'.
|
||||
|
||||
The headers are only sent to the host in the configured URL. If the
|
||||
server redirects to another host (including a subdomain or a different
|
||||
port) the headers are not sent to it, or to any further hop in that
|
||||
redirect chain. When headers are set, a redirect from https to http is
|
||||
refused as it would send them in cleartext.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: headers
|
||||
|
||||
+31
-22
@@ -2291,38 +2291,47 @@ Properties:
|
||||
- "br-ne1.magaluobjects.com"
|
||||
- Fortaleza, CE (BR), br-ne1
|
||||
- Provider: Magalu
|
||||
- "s3.eu-amsterdam.megas4.com"
|
||||
- Mega S4 Amsterdam
|
||||
- "s3.eu-luxembourg-1.megas4.com"
|
||||
- Mega S4 Luxembourg 1
|
||||
- Provider: Mega
|
||||
- "s3.eu-luxembourg.megas4.com"
|
||||
- Mega S4 Luxembourg
|
||||
- "s3.eu-luxembourg-2.megas4.com"
|
||||
- Mega S4 Luxembourg 2
|
||||
- Provider: Mega
|
||||
- "s3.eu-paris.megas4.com"
|
||||
- Mega S4 Paris
|
||||
- "s3.eu-amsterdam-1.megas4.com"
|
||||
- Mega S4 Amsterdam 1
|
||||
- Provider: Mega
|
||||
- "s3.eu-barcelona.megas4.com"
|
||||
- Mega S4 Barcelona
|
||||
- "s3.eu-amsterdam-2.megas4.com"
|
||||
- Mega S4 Amsterdam 2
|
||||
- Provider: Mega
|
||||
- "s3.ca-montreal.megas4.com"
|
||||
- Mega S4 Montreal
|
||||
- "s3.eu-paris-1.megas4.com"
|
||||
- Mega S4 Paris 1
|
||||
- Provider: Mega
|
||||
- "s3.ca-vancouver.megas4.com"
|
||||
- Mega S4 Vancouver
|
||||
- "s3.eu-paris-2.megas4.com"
|
||||
- Mega S4 Paris 2
|
||||
- Provider: Mega
|
||||
- "s3.ap-tokyo.megas4.com"
|
||||
- Mega S4 Tokyo
|
||||
- "s3.eu-barcelona-1.megas4.com"
|
||||
- Mega S4 Barcelona 1
|
||||
- Provider: Mega
|
||||
- "s3.eu-central-1.s4.mega.io"
|
||||
- Mega S4 eu-central-1 (Amsterdam, legacy)
|
||||
- "s3.eu-barcelona-2.megas4.com"
|
||||
- Mega S4 Barcelona 2
|
||||
- Provider: Mega
|
||||
- "s3.eu-central-2.s4.mega.io"
|
||||
- Mega S4 eu-central-2 (Bettembourg, legacy)
|
||||
- "s3.ca-montreal-1.megas4.com"
|
||||
- Mega S4 Montreal 1
|
||||
- Provider: Mega
|
||||
- "s3.ca-central-1.s4.mega.io"
|
||||
- Mega S4 ca-central-1 (Montreal, legacy)
|
||||
- "s3.ca-montreal-2.megas4.com"
|
||||
- Mega S4 Montreal 2
|
||||
- Provider: Mega
|
||||
- "s3.ca-west-1.s4.mega.io"
|
||||
- Mega S4 ca-west-1 (Vancouver, legacy)
|
||||
- "s3.ca-vancouver-1.megas4.com"
|
||||
- Mega S4 Vancouver 1
|
||||
- Provider: Mega
|
||||
- "s3.ca-vancouver-2.megas4.com"
|
||||
- Mega S4 Vancouver 2
|
||||
- Provider: Mega
|
||||
- "s3.ap-tokyo-1.megas4.com"
|
||||
- Mega S4 Tokyo 1
|
||||
- Provider: Mega
|
||||
- "s3.ap-tokyo-2.megas4.com"
|
||||
- Mega S4 Tokyo 2
|
||||
- Provider: Mega
|
||||
- "oos.eu-west-2.outscale.com"
|
||||
- Outscale EU West 2 (Paris)
|
||||
|
||||
Reference in new issue
Block a user