When the HTTP server failed to initialise, for example because the
listen address was already in use, rclone panicked with a nil pointer
dereference instead of reporting the error.
The deferred cleanup in the constructor read the provider from the
named return value, but `return nil, err` sets that to nil before the
deferred function runs. Use a local variable for the server instead.
(cherry picked from commit f2a390b2d4)
The auth proxy was only given the user and their password or public
key, so a proxy program had no way to restrict logins to particular
networks, or to record where an authentication attempt came from.
The JSON sent to the program now has a client_ip key holding the bare
IP the client connected from, with the port stripped so IPv6 arrives
as 2001:db8::1 rather than [2001:db8::1]:52344. An IPv4-mapped IPv6
address is reported as plain IPv4 so that a client arriving over a
dual-stack listener still matches IPv4 networks. The key is omitted
when the client has no IP address.
The IP is also mixed into the backend cache key. That is needed as the
program is only run on a cache miss, so a client from a
non-allowlisted address presenting valid credentials within the 5
minute cache lifetime would get a cache hit and be let in without the
program being consulted at all.
(cherry picked from commit 5dd34275dc)
Previously, GET requests for a directory URL always returned an HTML
directory listing. There was no way to suppress this, unlike
`serve webdav` which has supported --disable-dir-list since #4191.
This adds the same flag to `serve http`. When set, GET requests for
directory URLs return 404 instead of a listing, while file downloads
continue to work normally.
Based on the approach suggested in #6306.
Fixes#4000
This refactors all 8 serve protocols to use `rc.ParseOptions` for VFS
and protocol options decoding. It also implements parameter validation in the
main runner using `rc.CheckParamsUsed`.
- Update dlna, ftp, http, nfs, restic, s3, sftp, and webdav to call
rc.ParseOptions, enabling nested option block support.
- Remove unused configstruct imports.
- Update startRc in cmd/serve/rc.go to copy input parameters and
call rc.CheckParamsUsed to reject unknown parameters.
- Add TestRcStartFlatNestedAndUnknownRejection to cmd/serve/rc_test.go.
- Update inline documentation in serve/start command help to include
nested blocks information (vfsOpt, proxyOpt, opt) and a new WebDAV example.
Go 1.27 changes compress/flate output, which also changes archive/zip
byte output. The HTTP zip download tests currently compare raw zip bytes
against golden files, so they fail even though the generated zip archives
contain the expected files.
Compare zip entries and decompressed contents instead of the exact
compressed byte stream.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
The --disable-zip flag was registered manually and was missing from
OptionsInfo, so it could not be set over the rc interface. Move it
into OptionsInfo like serve webdav does, which keeps the command line
flag and also makes it settable via rc.
Add a ctx parameter to vfs.New() so callers can pass in context
carrying ConfigInfo and FilterInfo. The context is stripped of
cancellation but config and filter values are preserved into a fresh
background context.
Add gzip compression for directory listings and text assets served over HTTP.
This reduces the rclone repository file listing from 40 kB to 8 kB and reduces
the rclone MANUAL.txt from 2.7 MB to 700 kB.
This makes listings and assets served across the network load faster.
The compression level of 5 should be a good balance between size and speed.
Browsers make a request to /favicon.ico when visiting pages generated
by the HTTP server.
Previously, if remotes did not have a /favicon.ico then the server
responded with a 404, causing browsers to show a default icon.
This adds a tiny fallback embedded PNG rclone favicon to help users
identify the rclone browser tab.
This is so that they can import cmd/serve without causing an import
loop.
The active servers can now be configured by commenting lines out in
cmd/all/all.go like all the other commands.
This commit modernizes Go usage. This was done with:
go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./...
Then files needed to be `go fmt`ed and a few comments needed to be
restored.
The modernizations include replacing
- if/else conditional assignment by a call to the built-in min or max functions added in go1.21
- sort.Slice(x, func(i, j int) bool) { return s[i] < s[j] } by a call to slices.Sort(s), added in go1.21
- interface{} by the 'any' type added in go1.18
- append([]T(nil), s...) by slices.Clone(s) or slices.Concat(s), added in go1.21
- loop around an m[k]=v map update by a call to one of the Collect, Copy, Clone, or Insert functions from the maps package, added in go1.21
- []byte(fmt.Sprintf...) by fmt.Appendf(nil, ...), added in go1.19
- append(s[:i], s[i+1]...) by slices.Delete(s, i, i+1), added in go1.21
- a 3-clause for i := 0; i < n; i++ {} loop by for i := range n {}, added in go1.22
fs.CountError is called when an error is encountered. The method was
calling GlobalStats().Error(err) which incremented the error at the
global stats level. This led to calls to core/stats with group= filter
returning an error count of 0 even if errors actually occured.
This change requires the context to be provided when calling
fs.CountError. Doing so, we can retrieve the correct StatsInfo to
increment the errors from.
Fixes#5865
This changes log statements from log to fs package, which is required for --use-json-log
to properly make log output in JSON format. The recently added custom linting rule,
handled by ruleguard via gocritic via golangci-lint, warns about these and suggests
the alternative. Fixing was therefore basically running "golangci-lint run --fix",
although some manual fixup of mainly imports are necessary following that.
This also
- move in use options (Opt) from vfsflags to vfscommon
- change os.FileMode to vfscommon.FileMode in parameters
- rework vfscommon.FileMode and add tests
Before this change it wasn't possible to see where transfers were
going from and to in core/stats and core/transferred.
When use in rclone mount in particular this made interpreting the
stats very hard.
This adds an additional parameter to the creation of each flag. This
specifies one or more flag groups. This **must** be set for global
flags and **must not** be set for local flags.
This causes flags.md to be built with sections to aid comprehension
and it causes the documentation pages for each command (and the
`--help`) to be built showing the flags groups as specified in the
`groups` annotation on the command.
See: https://forum.rclone.org/t/make-docs-for-mortals-not-only-rclone-gurus/39476/
This change addresses two issues with commands that re-used
flags from common packages:
1) cobra.Command definitions did not include the command specific
prefix in doc strings.
2) Command specific flag prefixes were added after generating
command doc strings.
Before this change only serve http was Shutting down its server which
was causing other servers such as serve restic to leave behind their
unix sockets.
This change moves the finalisation to lib/http so all servers have it
and removes it from serve http.
Fixes#6648
- add support for unix sockets (which skip the auth).
- add support for multiple listeners
- collapse unnecessary internal structure of lib/http so it can all be
imported together
- moves files in sub directories of lib/http into the main lib/http
directory and reworks the code that uses them.
See: https://forum.rclone.org/t/wip-rc-rcd-over-unix-socket/33619Fixes: #6605
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.
This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
Nick Craig-Wood