Log entries which rclone can attribute to the rc job which made them
now have jobid and group entries in the JSON log. core/log can filter
on these with the new group and jobid parameters.
_logs and job/status now return the logs of the job itself plus the
logs made while it was running which aren't attributed to any job,
leaving out the logs of other jobs running at the same time. Note
that calls sharing a stats group with _group are attributed
separately as they are different jobs.
If _logs is set to true or a log level on an rc call then the logs
made while the call was running are returned as _logs in the result,
or in the error response if the call failed. For _async calls the logs
so far are returned as logs in job/status.
This needs the log buffer enabling with --log-buffer-size.
Logs aren't currently attributed to the call which made them so this
returns everything logged while the call was running. Because of this
_logs is refused on an rc server without authentication so the logs
can't be read through the calls which don't need authentication.
This adds the --log-buffer-size flag which keeps the most recent logs
in memory in JSON format, dropping the oldest when it is full.
These can then be read with the new rc call core/log which returns
the log entries from a given sequence number so it can be polled to
follow the log.
The log buffer is disabled by default and can be enabled at runtime
with options/set.
Several documentation links pointed at anchors or paths that no longer
resolve, and the S3 directory buckets section named the config option
and flag in the plural, which does not match the backend.
Co-authored-by: shaurya <19599684+no-hup@users.noreply.github.com>
Co-authored-by: no-hup <shauryaj.finance@gmail.com>
Allow --order-by to rank files using comma-separated rclone path globs. Patterns are evaluated in order, unmatched files are placed last, and path ordering makes ties deterministic.
Fixes#3975
Previously running rclone config (or driving it via the rc API or web
GUI) with -vv would write secrets to the debug log.
This was dangerous as users debugging a failing config flow often
paste their -vv logs into the forum or GitHub issues.
These values are now redacted from the log as "XXX". Values whose
option is known are only redacted if the option is marked IsPassword
or Sensitive, so normal answers remain visible.
Use --dump auth to see the unredacted values when debugging a config
flow - rclone prints a warning that secrets will appear in the log
when this is in effect.
This was discovered by CodeQL: https://github.com/rclone/rclone/security/code-scanning/182
The new "trace" dump flag attaches a net/http/httptrace ClientTrace to
each HTTP transaction and logs the connection level events - DNS
resolution, TCP connect, TLS handshake (including the negotiated TLS
version, cipher, ALPN protocol and server certificate), connection
reuse, request write and time to first response byte. Each line is
tagged with the time elapsed since the start of the transaction and the
request pointer so it can be correlated with the other dumps.
This is complementary to the existing dump flags: it shows how the
connection behaved rather than what was sent, which is useful for
debugging connectivity, DNS, TLS, proxy and keep-alive problems.
The new "errors" dump flag makes the HTTP dump conditional on the
transaction failing with a retryable error (a transport error, HTTP 429
or HTTP 5xx), so first-failure diagnostics can be captured without the
noise of dumping every transaction. The existing dump flags continue to
control what is dumped, for example --dump errors,bodies, and on its own
--dump errors dumps the headers.
This allows backend config to contain
- `override.var` - set var during remote creation only
- `global.var` - set var in the global config permanently
Fixes#8563
Nick Craig-Wood