Testing against OneDrive personal (free) and OneDrive for Business shows
- personal accounts now create versions on setting the modification
time and can delete them, so --onedrive-no-versions and rclone
cleanup work there
- --onedrive-link-password works on OneDrive for Business
- personal free accounts can't set a link password or --expire
- --onedrive-link-type embed only works on OneDrive personal
- personal accounts store times with 1s precision, not mS
See #9917
With --vfs-metadata-extension set, looking up the metadata file of a
file which was open for write and not yet uploaded caused a nil pointer
panic. Such a file has no object to read the modification time from.
Use the modification time of the VFS node instead, which is valid
whether or not the file has been uploaded.
TestDeleteFatalError set --max-delete before writing its files, so on
chunker, which deletes while uploading, the setup failed.
TestDirMoveMoveError and TestDirMoveContext test the core DirMove logic
with a wrapping Fs. This fails on remotes without Move (eg s3, memory)
and on those whose objects don't belong to the wrapped Fs (eg archive),
so they now only run on local.
findItem read the response status code when the lookup failed, so a
failure with no HTTP response panicked.
Thanks to @manus-pi for finding this problem.
DirMove discarded the error from the destination check and read the
response status code, so a failure with no HTTP response panicked.
Other errors were reported as the destination existing; they are now
returned.
Thanks to @manus-pi for finding this problem.
Rmdir and Purge read the response status code before checking for an
error, so a failed DeleteFolder call with no HTTP response (a network
error, or purging the root which fails validation) panicked.
Thanks to @manus-pi for finding this problem.
The request log printed the escaped URL, so non-ASCII file names showed
up as long runs of %XX escapes. Log the unescaped URL path instead, as
the other serve commands do.
VFS.AddVirtual always added a file entry to the directory cache, even
when called with isDir set, so a virtual directory showed up as a file
and nothing could be added inside it.
See #9310
With -L/--copy-links a symlink pointing to one of its parent
directories was followed until the OS gave up with "too many levels
of symbolic links" 40 levels deep. As every looping symlink multiplies
the listing, a small tree with three such symlinks listed millions of
entries.
When a followed symlink points to a directory, compare it with the
directory being listed and each of its parents using os.SameFile and
if it matches report an error and skip it, the same way as a circular
symlink. A symlink to a sibling directory is still followed. Loops
excluded by the directory filters are skipped quietly as before.
The default for --smb-user was the name of the user running rclone
config, and rclone does not write defaults to the config file, so
entering your own user name left it out. A remote made that way then
logged in as whoever ran it later, e.g. root under systemd.
Make the default blank and look up the current user when the remote is
used, as the ftp backend does. Existing configs work as before.
When the source matched a file in --copy-dest, copyDest server-side
copied it over an existing destination which differed, so sync, copy
and copyto with --copy-dest could modify a file --immutable should have
protected.
Leave such a destination for the caller, which rejects it with
ErrorImmutableModified as for any other modified file.
--immutable was only enforced by sync, so copyto, moveto, copy and move
of a single file, and the operations/copyfile and operations/movefile rc
calls overwrote a destination which differed from the source.
Fail these with the same ErrorImmutableModified as sync, and reject
--no-check-dest with --immutable as sync does, since without checking
the destination it cannot be protected.
The box backend logged an int64 with %q which printed
%!q(int64=123) instead of the sequence ID.
serve s3 passed the message from gofakes3 as the format string, so
any % in it was interpreted as a formatting directive and the message
was mangled.
job/status returns the complete output of the job so it is as
sensitive as the call which started the job, e.g. config/dump started
with _async returns the config file including any secrets. It was
possible to read this without authentication where an authenticated
and an unauthenticated rc server share the same process, e.g. rclone
gui --rc.
An unauthenticated client can't start jobs on a server which needs
authentication, other than with the calls which don't need it, so
this shouldn't affect existing users.
Handle read the list of extra outputs without holding the mutex, so
calling AddOutput while logging caused a data race.
This reads the extra outputs once under the mutex.
The wait to obey a Retry-After error from the server used
time.Sleep() so it carried on sleeping when the context was
cancelled, e.g. by job/stop or --max-duration, and it also slept
after the last try when there was nothing left to retry.
Copying a file also logged the low level retry number counting from
0 rather than 1 as everything else does.
When the backend has no DirMove method, operations.DirMove moves the
objects one by one. If one of the moves failed the movers stopped
without receiving the rest of the objects, so DirMove blocked forever
once the channel filled up. This could be seen renaming a directory on
a mount of a backend without DirMove, such as s3.
The moves were also done with context.Background() instead of the
caller's context, so they didn't have the caller's config, couldn't
be cancelled and weren't counted in the caller's stats.
When a fatal error such as --max-delete being reached stopped the
deletions, the deleters returned without receiving the rest of the
objects, so whatever was sending them blocked forever once the
channel filled up.
For example "rclone delete --max-delete 1" on a directory with more
files than --checkers hung instead of returning the error.
The deleters now keep receiving objects without deleting them after a
fatal error.
Params.GetInt64 parsed string parameters with strconv.ParseInt(x, 10, 0).
A bitSize of 0 means int, which is 32 bits on 386, arm, mips and mipsle,
so any value outside the int32 range was rejected as out of range even
though it fits in the int64 the method returns.
Parameters reach the rc API as strings on the paths that matter here: the
rc server puts every URL query and form value into Params as a string, and
rclone rc turns each key=value argument into a string. So on a 32 bit build
operations/getfile could not be given an offset, count, head or tail beyond
2 GiB, and debug/set-soft-memory-limit could not be given a limit beyond
2 GiB, while the same commands worked on a 64 bit build.
Parse with a bitSize of 64 to match the declared return type. This is the
string half of the range checking that was added to the float64 branch of
the same function in 976d05e1.
Return the first non-EOF input error encountered while Rcat probes whether an
upload is small. Previously that error was ignored and the full probe buffer,
including bytes that were never read, could be passed to Put or PutStream.
- Reject uploads when object lookup fails for reasons other than "not found".
- Serialize uploads to the same object to prevent races between the
existence check and the write.
Gzip metadata is read from the wrapped remote and could contain an
invalid block size or incomplete block index. A range read could then
panic in the seekable gzip reader.
Validate the gzip sidecar invariants before constructing a reader so
malformed remote metadata returns an error instead of crashing rclone.
The start function was run inside an if condition, where bash ignores
errexit, so a failed docker run was not noticed. The script then
printed its connection details anyway and the test spent 100 seconds
trying to connect before failing with a message that hid the real
error.
Run start in a subshell with errexit on and check its status
explicitly so the failure is reported immediately with docker's
error message.
The minio/minio repository has been removed from Docker Hub so the
TestS3Minio test server could not be started and every Linux CI run
failed. quay.io/minio/minio still serves the final release so use
that instead.
OCI rejects a multipart completion request with no parts. Probe unknown-size
streams through a buffered reader and use a regular upload when the stream is
empty. Peek preserves a non-empty stream for the selected upload path.
Internxt stores a file as a (plainName, type) pair and never derives the
split itself, so it is a convention shared between clients. This backend
split at the final dot, storing and looking up ".bashrc" as an empty name
of type "bashrc", where the web, desktop, Linux and macOS clients all keep
the leading dot in plainName. List rebuilt the full name so such files
appeared, but NewObject looked them up by the split and missed them.
- Replaced the preUploadCheck function with findFile for better clarity and efficiency in checking file existence.
- Introduced splitNameExt to encapsulate name and extension parsing logic.
- Updated NewObject and Update methods to utilize findFile for improved file metadata handling.
- Enhanced error handling and reduced redundant code in file checks.
Use OCI BatchDeleteObjects to remove up to 1,000 objects per request when
purging. Include directory markers so purging a bucket leaves it empty for
deletion.
OCI Object Storage rejects NUL, carriage return and line feed in
object names. NUL is already encoded by rclone, while carriage returns
and line feeds need EncodeCrLf so rclone can represent those names
without issuing invalid OCI requests.
https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/managingobjects.htm
When rclone archive create is run without a destination the archive
should be written to stdout, but ArchiveCreate called
CheckValidDestination on the nil dst and panicked with a nil pointer
dereference. Skip the destination check when there is no destination.
Also remove a leftover debug Printf that wrote to stdout before the
archive data.
Signed-off-by: Vladimir Babin <vovababin@gmail.com>
Nick Craig-Wood