NewStatsGroup started the averageLoop goroutine unconditionally at
group creation. In an rcd daemon driven by many short rc sync/move
calls (a common pattern for scheduled spool flushes), each call gets
a fresh job/N stats group. When such a job transferred zero files
the loop was never stopped, because _stopAverageLoop is only reached
via DoneTransferring once transferring and checking both go from
non-empty back to empty, which never happens if nothing was ever
transferring in the first place. The result was one leaked goroutine
per rc call, growing unbounded until the daemon was OOM-killed
(reported: ~61k goroutines and ~640 MB RSS after ~7 days from a
per-minute timer over 6 mappings).
This is the same class of leak as #8571, which fixed the equivalent
auto-start in NewStats. Fix it the same way: do not start the average
loop at group creation. NewTransfer and NewTransferRemoteSize already
call startAverageLoop when real transfer activity begins, and
DoneTransferring already stops it when the last transfer completes,
so on-demand behaviour is unchanged for groups that actually do work.
Groups that never transfer anything now cost zero goroutines.
Adds a regression test that fails without the fix.
Fixes#9567
Mknod is now implemented by mount, mount2 and cmount, so exercise it from
the shared vfstest suite: create a regular file (S_IFREG) through the
mounted path and check it reads back as a 0-byte regular file. This is the
path the kernel NFS server drives when a client creates a file over an
exported mount.
Before this there was no shared coverage for Mknod; it now runs against
each backend via RunTests.
Suggested in #9548.
Signed-off-by: Sandy Luppino <s.luppino@opendrives.com>
Seekdir handled only a rewind to offset 0 and returned ENOTSUP otherwise.
The stateless kernel NFS server opens a fresh directory handle and seeks to
the last returned cookie on every readdir continuation, so any listing
spanning more than one readdir batch failed over NFS. dirStream is a
snapshot taken at Readdir time and go-fuse assigns each entry a sequential
offset, so seeking to off positions the stream at index off; off == 0 still
resets to the start, preserving the rewind/re-read behaviour.
Before: ls of a directory that spans more than one readdir batch failed
over NFS with "Unknown error 524".
After: it lists correctly.
Adds TestDirStreamSeekdir covering rewind, mid-stream resume and the EOF
clamp. The full NFS path was validated against a real Linux
nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2.
Fixes#9547
setAttr left attr.Ino unset (0) and the NewInode sites left StableAttr.Ino
unset, so the kernel saw inode 0 while the node identity differed, which
breaks NFS file-handle validation. Set both to the stable VFS inode. The
bazil cmd/mount backend does not hit this because its framework assigns
stable inodes automatically; go-fuse needs them set explicitly.
Before: chmod/chown/truncate on a just-written file through an
NFS-exported mount2 mount failed with ESTALE.
After: they succeed.
Exercising the NFS handle-validation path needs a kernel NFS server, so it
isn't covered by the local vfstest harness; validated against a real Linux
nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2.
#9547
The kernel NFS server creates regular files with MKNOD (it
creates-then-opens, so vfs_create routes through fuse_create -> FUSE_MKNOD
when there is no open intent), but mount2 only implemented Create
(FUSE_CREATE, used by local and SMB clients). Without Mknod every NFS file
creation failed with ENOTSUPP. This mirrors the cmd/mount mknod handler from
#2115.
Before: touch through an NFS-exported mount2 mount failed with ENOTSUPP.
After: files create normally.
Exercising this needs a kernel NFS server, so it isn't covered by the
local vfstest harness; validated against a real Linux nfs-kernel-server
export over NFSv3, NFSv4.0 and NFSv4.2.
#9547
The mega backend keeps the whole account as an in-memory tree. go-mega's
Delete with destroy=true (hard_delete) removed the node from its lookup
table but left it in its parent's children, so a hard deleted file kept
showing up in directory listings until the tree was reloaded. In a
long-running mount or serve the file reappeared once the VFS directory
cache expired and re-read the backend.
Update go-mega to pick up the fix, and let the ghost test exercise
whichever delete mode the remote is configured with.
The mega backend keeps the whole account as an in-memory tree which
go-mega reconciles asynchronously from the server's event stream. After
an upload, delete or move the optimistic local update could be undone
moments later when go-mega replayed the corresponding server event,
re-adding a node to its parent. In a long-running process such as mount
or serve this could briefly show a just-deleted file in a listing, or
make a moved file reappear shortly afterwards.
Wait for the server to confirm uploads, deletes and moves so the
in-memory tree is settled before returning, matching what Rmdir and
Purge already do. The move wait was also previously started after the
server-side move so only the rename was covered.
Add an internal test reproducing the lingering listing via a tight
put/remove/list loop.
When a cached file was closed, --vfs-handle-caching kept its handle and
downloaders alive for a grace period and closed them later from a timer.
The deferred close drops the item lock while tearing down the
downloaders, leaving the file handle open. A reopen landing in that
window saw no grace timer and a live handle, failed to create the cache
file with "internal error: didn't Close file" and removed the cache
file, which hung the application reopening the file.
Reopens now wait for an in-progress grace-period close to finish so they
start from a fully closed item.
See: https://forum.rclone.org/t/opening-a-recently-closed-and-cached-file-hangs-rclone/53986/
This commit allows global config options to be set as flat parameters on all rc
commands. This makes the rc commands much more like command line parameters and
will aid understanding of how the rc is used.
It is backwards compatible with the old method using _config and _filter.
A backend flag set on the command line to a value that happened to
equal its default was silently ignored, letting the config file win
instead.
For example --sftp-user defaults to the current user, so connecting as
that same user with --sftp-user=USER had no effect and caused rclone
to use the value from the config file.
See: https://forum.rclone.org/t/sftp-user-parsing-as-cli-argument-broken/53955
DeleteFile always passes a nil backupDir to DeleteFileWithBackupDir, so
it never honours --backup-dir. The previous comment said it would move
the file into the backup dir when --backup-dir was in effect, which does
not match the code. Update the comment to state that DeleteFile always
deletes and that callers should use DeleteFileWithBackupDir when
--backup-dir support is required.
Also document on DeleteFileWithBackupDir that the backupDir is found with
BackupDir, which is relatively expensive, so it should be looked up once
outside any delete loop rather than per object.
#7566
govulncheck reported four vulnerabilities in golang.org/x/image v0.41.0,
all reachable via the internxt backend's thumbnail/image upload path:
- CVE-2026-46604 (GO-2026-5066): panic decoding a TIFF image with an
out-of-bounds strip offset
- CVE-2026-46602 (GO-2026-5062): unbounded memory use from lack of a
limit on TIFF tile sizes
- CVE-2026-46601 (GO-2026-5061): panic on a WEBP VP8 alpha channel size
mismatch
- CVE-2026-33813 (GO-2026-4961): panic decoding a large WEBP image on
32-bit platforms
Updating to v0.43.0 fixes all four.
After an upload (notably multipart) to a slow backing remote, the file
lives in the VFS and is returned by ListBucket, but node.DirEntry() stays
nil until the --vfs-write-back writeback completes. HeadObject and
GetObject returned gofakes3.KeyNotFound while it was nil, so a HEAD/GET in
that window 404'd even though the object existed.
getFileHashByte already falls back to hashing the VFS cache when the
backing object isn't available yet. Drop the early nil return, pass the
node (not the fs.Object) to getFileHashByte, and take the Content-Type
from fs.MimeTypeFromName when the backing object isn't there yet.
The HeadObject path translates a 404 (Not Found) and a 405 (Method Not
Allowed) response into fs.ErrorObjectNotFound, but the GetObject path
returns the raw AWS SDK error instead. With --s3-no-head-object set,
Object.Open() is the first request to hit the source for a GET
operation, so callers that rely on fs.ErrorObjectNotFound see an
opaque "operation error S3: GetObject, ... StatusCode: 404" string
when --s3-no-head-object is set.
Mirror the error handling of HeadObject. The 405 (Method Not Allowed)
case applies to GetObject for the same reason it applies to
HeadObject: A request that specifies the versionId of a delete marker
returns 405 instead of 404 (commit 8470bdf810, AWS S3 delete-marker
docs).
Co-authored-by: Jan Schlien <rclone-github@jan-o-sch.net>
This adds two new Dropbox backend flags:
--dropbox-skip-shared-folders skips all shared folder mount points
regardless of ownership.
--dropbox-skip-unowned-folders only skips shared folders that are
not owned by the current user.
These help avoid backing up the same shared folder multiple times when
backing up multiple Dropbox accounts.
Fixes#9514
Before this change, with --files-from and --no-traverse, a single file
that could not be read (for example permission denied) stopped all the
other files in the list being copied.
This happened because the error was returned from the listing, which
caused the whole source listing to be discarded.
This change counts and logs such per-file errors and carries on, so the
readable files are still copied and rclone exits with a non-zero error
code.
Fixes#9115
Option help strings are also used to generate the website documentation,
so some contain markdown links with root-relative targets such as
[encoding section in the overview](/overview/#encoding). These render
correctly on rclone.org but are confusing in the interactive config
prompt, where the user sees the raw markdown and the link has no
reachable root.
Rewrite such links to text (https://rclone.org/path) when showing an
option's help in the interactive config. The raw help is left unchanged
so documentation generation is unaffected.
Add a .github/CODEOWNERS file mapping backends and core subsystems to
their maintainers so GitHub auto-requests reviews. Move the per-area
responsibility mapping out of MAINTAINERS.md into CODEOWNERS, leaving
MAINTAINERS.md as the maintainer list with a pointer to CODEOWNERS.
Since v1.74.0, ncdu doubled every keystroke on Windows: pressing Enter
entered a directory and immediately descended a level deeper, "n" sorted
by name and then reversed the order, and so on. Arrow keys were
unaffected.
The cause is an upstream regression in github.com/gdamore/tcell/v2
pulled in by the v2.9.0 -> v2.13.8 bump in commit 23917555. The same
regression affects lazygit and micro.
There is no fixed tcell release, so pin tcell back to v2.9.0, the last
version before that commit, until the input handling is fixed upstream.
See: https://github.com/gdamore/tcell/issues/1124
See: https://github.com/jesseduffield/lazygit/issues/5344Fixes#9539
The Cobra generated PowerShell completion script captures rclone's output
through a pipeline with Invoke-Expression. PowerShell decodes that output
using [Console]::OutputEncoding, which on non-UTF-8 hosts (for example
PowerShell 5.1 on a Windows install with an OEM code page such as CP852)
misinterprets the UTF-8 bytes rclone emits and corrupts remote and path
names containing non-ASCII characters, so tab completion produces a path
that does not exist.
Inject "[Console]::OutputEncoding = [System.Text.Encoding]::UTF8" into the
generated script immediately before the Invoke-Expression call. This is safe
on PowerShell 7+, where UTF-8 is already the default. If the expected line is
not present (for example after a Cobra template change) the script is emitted
unmodified so we never produce a corrupted completion script.
Convert the bug report and feature request templates to GitHub issue forms.
Rewrite the pull request template to require that non-trivial changes are
discussed in an issue first, and to spell out that backend changes need a
clean test_all run and a test account for the integration tester before
they can be merged. Document the latter requirement in CONTRIBUTING.md.
Remove the legacy ISSUE_TEMPLATE.md which is superseded by the forms.
The new "trace" dump flag attaches a net/http/httptrace ClientTrace to
each HTTP transaction and logs the connection level events - DNS
resolution, TCP connect, TLS handshake (including the negotiated TLS
version, cipher, ALPN protocol and server certificate), connection
reuse, request write and time to first response byte. Each line is
tagged with the time elapsed since the start of the transaction and the
request pointer so it can be correlated with the other dumps.
This is complementary to the existing dump flags: it shows how the
connection behaved rather than what was sent, which is useful for
debugging connectivity, DNS, TLS, proxy and keep-alive problems.
The new "errors" dump flag makes the HTTP dump conditional on the
transaction failing with a retryable error (a transport error, HTTP 429
or HTTP 5xx), so first-failure diagnostics can be captured without the
noise of dumping every transaction. The existing dump flags continue to
control what is dumped, for example --dump errors,bodies, and on its own
--dump errors dumps the headers.
TestS3Minio brings up a minio container via the fstest/testserver
framework, which exec's bash init.d scripts that shell out to docker.
This is not available on all platforms - Windows has no POSIX shell to
run the scripts, and macOS CI runners have no docker daemon - which
caused the build to fail there.
Add testy.SkipUnlessDocker to detect whether the framework can run and
skip the test when it cannot.
Previously serve s3 buffered every part of a multipart upload in memory
(in the gofakes3 S3 library) and concatenated them when the upload
completed, so memory use grew with the size of the upload.
serve s3 now streams the parts, in part-number order, into a single
PutStream upload to the underlying remote, which performs its own upload
with bounded memory. The whole file is never held in memory - memory use
is bounded by the parts in flight. This works for any remote that
supports PutStream (nearly all, including crypt) and for any part size,
so clients that don't produce uniform-sized parts (for example
PostgreSQL backup tools such as pgBarman and pgBackRest) work too.
Parts must arrive in ascending, contiguous part-number order; parts
uploaded out of order are buffered until their turn, and there is no
per-part retry (a failure aborts the whole upload). These trade-offs are
documented.
Passing --disable-multipart-streaming, or using a remote without
PutStream, reverts to buffering the parts in memory (the previous
behaviour); a one-off NOTICE is logged the first time this happens.
Fixes#7453
Run still uses a fresh local directory as the backing Fs that the
server wraps. RunWithBackend takes an extra remote name (e.g.
"TestS3Minio:") and uses a random subdirectory of that remote instead,
starting the matching fstest/testserver/init.d script on the way in
and tearing it down on the way out.
AuthProxy is only run for the local backend.
