Objective: to facilitate capturing metrics for relatively short-lived kopia
executions.
Motivation: Currently, kopia allows a. exposing metrics via a Prometheus
exporter; and b. pushing metrics to a Prometheus gateway. In certain
scenarios, such as short lived executions, it is not possible to reliably
scrape the metrics from the exporter endpoint. And, while the pusher
approach would work, it requires starting a separate push gateway.
This change allows saving the metrics to a local file before kopia exists
by specifying the metrics output directory. For example,
`kopia --metrics-directory /tmp/kopia-metrics/ snapshot create ....`
In this case, after kopia exists, the `/tmp/kopia-metrics/` directory will
contain a file with a named of the form:
`<date>-<time>-<command_subcommand>.prom`
Motivation: reduce the function complexity for the linter, so additional
functionality can be added to `startMetrics()`
No functional changes.
Changes and transformations:
- Break down observability.startMetrics() into: maybeStartListener().
maybeStartMetricsPusher() and maybeStartTraceExporter().
- Inline observabilityFlags.getExporter in maybeStartTraceExporter.
- Only create resource when exporter is non-nil.
- Simplify maybeStartListener: reduce indentation by returning early.
- Simplify maybeStartMetricsPusher: reduce indentation by returning early.
Lack of generics support is blocking various dependency upgrades,
so this unblocks that.
Temporarily disabled `checklocks` linter until it is fixed upstream.
* feat(infra): improved support for in-process testing
* support for killing of a running server using simulated Ctrl-C
* support for overriding os.Stdin
* migrated many tests from the exe runner to in-process runner
* added required indirection when defining Envar() so we can later override it in tests
* refactored CLI runners by moving environment overrides to CLITestEnv
New flag `--enable-jaeger-collector` and the corresponding
`KOPIA_ENABLE_JAEGER_COLLECTOR` environment variable enables Jaeger
exporter, which by default sends OTEL traces to Jaeger collector on
http://localhost:14268/api/traces
To change this, use environment variables:
* `OTEL_EXPORTER_JAEGER_ENDPOINT`
* `OTEL_EXPORTER_JAEGER_USER`
* `OTEL_EXPORTER_JAEGER_PASSWORD`
When tracing is disabled, the impact on performance is negligible.
To see this in action:
1. Download latest Jaeger all-in-one from https://www.jaegertracing.io/download/
2. Run `jaeger-all-in-one` binary without any parameters.
3. Run `kopia --enable-jaeger-collector snapshot create ...`
4. Go to http://localhost:16686/search and search for traces
When enabled, metrics are pushed to the provided Prometheus Push
Gateway at the start and end of each command and periodically every
few seconds.
```
--metrics-push-addr=http://address:port
--metrics-push-interval=5s
--metrics-push-job=kopia
--metrics-push-grouping=a:b --metrics-push-grouping=c:d
--metrics-push-username=user
--metrics-push-password=pass
```
