When extsort.Strings() cannot create temporary files (e.g. due to
apparmor restrictions or permission denied), it returns a nil sorter
with the error on errChan. The code then called Sort() on the nil
sorter, causing a panic.
Check for nil sorter and return the error instead of panicking.
Fixes#9244
The format field was read in Handle() without synchronization while
setFormat() could write it concurrently from InitLogging(). This
caused a data race detected by the race detector, failing
TestListBucketsAuthProxy in cmd/serve/s3.
Fix by protecting all access to format with the existing mutex.
This reduces the number of go routines which can get out of hand when
using large --transfers and --multi-thread-streams from potentially
--multi-thread-streams * --transfers Go routines to --max-memory /
--multi-thread-chunk-size
It serializes the memory allocator in each transfer which should be
good for performance and reduce lock contention.
Before this change server side copies would show at 0% until they were
done then show at 100%.
With support from the backend, server side copies can now be accounted
in real time. This will only work for backends which have been
modified and themselves get feedback about how copies are going.
If the transfer fails, the bytes accounted will be reversed.
Before this change when doing a sync with `--no-traverse` and
`--files-from` we could call `NewObject` a total of `--checkers` *
`--checkers` times simultaneously.
With `--checkers 128` this can exceed the 10,000 thread limit and
fails when run on a local to local transfer because `NewObject` calls
`lstat` which is a syscall which needs an OS thread of its own.
This patch uses a weighted semaphore to limit the number of
simultaneous calls to `NewObject` to `--checkers` instead which won't
blow the 10,000 thread limit and is far more sensible use of OS
resources.
Fixes#9073
These stats weren't being updated in the global stats read by rc
core/stats:
- transferQueue
- deletesSize
- serverSideCopies
- serverSideCopyBytes
- serverSideMoves
- serverSideMoveBytes
Before the log re-organisation in:
8d353039a6 log: add log rotation to --log-file
rclone would write any backtraces to the --log-file which was very
convenient for users.
This got accidentally disabled due to a typo which meant backtraces
started going to stderr even if --log-file was supplied.
This fixes the problem.
We accidentally added a non `camelCase` parameter to the rc
(`config_password`)- this fixes it (to `configPassword`) but accepts
the old name too as it has been in a release.
Before this change, if any code called fs.Fatal(f) then it would stop
rclone as designed. However this is not appropriate when using the RC
API - we want the error returned to the user.
This change turns the fs.Fatal(f) call into a panic which is caught by
the RC API handler and returned to the user as a 500 error.
* Adds "aix/ppc64" to the cross-compile target list.
* Including AIX in the build tag of "metadata_other.go".
* Excluding AIX from the main ncdu build tags.
* Marking AIX as an unsupported platform for ncdu.
* Excluding AIX from the fallback redirect implementation.
* Excluding AIX from unix build tags to avoid undefined unix.WNOHANG.
We were seeing a (non-fatal) error in our logs:
```
Failed to seek log file to end: seek /proc/1/fd/1: illegal seek
```
Because we open the log file with O_APPEND,
we don't need to manually seek to the end.
As https://pkg.go.dev/os#File.Seek also confirms
that the behavior of `Seek` is not specified
if the file has been opened with O_APPEND,
remove the `Seek` call.
Before this change, you had to modify a fragile data-structure
containing all providers. This often led to things being out of order,
duplicates and conflicts whilst merging. As well as the changes for
one provider being in different places across the file.
After this change, new providers are defined in an easy to edit YAML file,
one per provider.
The config output has been tested before and after for all providers
and any changes are cosmetic only.
Before this change --no-traverse was calling NewObject on directories
(where it would always fail) as well as files. This was very
noticeable when doing syncs with --max-age which were only
transferring a small number of objects. This should have been very
quick, but the NewObject calls for each directory slowed the sync down
a lot.
This changes replaces the check to see if the source entry is an
Object that got missed out from this commit:
88e30eecbf march: fix deadlock when using --no-traverse - fixes#8656
