This is a breaking change to users who might be using Kopia as a library.
### Log Format
```json
{"t":"<timestamp-rfc-3389-microseconds>", "span:T1":"V1", "span:T2":"V2", "n":"<source>", "m":"<message>", /*parameters*/}
```
Where each record is associated with one or more spans that describe its scope:
* `"span:client": "<hash-of-username@hostname>"`
* `"span:repo": "<random>"` - random identifier of a repository connection (from `repo.Open`)
* `"span:maintenance": "<random>"` - random identifier of a maintenance session
* `"span:upload": "<hash-of-username@host:/path>"` - uniquely identifies upload session of a given directory
* `"span:checkpoint": "<random>"` - encapsulates each checkpoint operation during Upload
* `"span:server-session": "<random>"` -single client connection to the server
* `"span:flush": "<random>"` - encapsulates each Flush session
* `"span:maintenance": "<random>"` - encapsulates each maintenance operation
* `"span:loadIndex" : "<random>"` - encapsulates index loading operation
* `"span:emr" : "<random>"` - encapsulates epoch manager refresh
* `"span:writePack": "<pack-blob-ID>"` - encapsulates pack blob preparation and writing
(plus additional minor spans for various phases of the maintenance).
Notable points:
- Used internal zero allocation JSON writer for reduced memory usage.
- renamed `--disable-internal-log` to `--disable-repository-log` (controls saving blobs to repository)
- added `--disable-content-log` (controls writing of `content-log` files)
- all storage operations are also logged in a structural way and associated with the corresponding spans.
- all content IDs are logged in a truncated format (since first N bytes that are usually enough to be unique) to improve compressibility of logs (blob IDs are frequently repeated but content IDs usually appear just once).
This format should make it possible to recreate the journey of any single content throughout pack blobs, indexes and compaction events.
The kopia server was not uploading any logs to the repository,
because "repodiag" blob uploads would always fail.
The cause was the following: when the (log) repodiag blob
PUT operation was initiated, the `Context` used for this
operation was already canceled.
The context used for blob uploads is passed to
`repodiag.NewLogManager` when opening the repository.
In the case of the kopia server, the repository is asynchronously
opened in `server.Server.InitReposotoryAsync`. The context
passed to `repo.Open` is canceled after the "open repository"
server task completes.
This issue was introduced in #1691
Change:
Use `context.WithoutCancel()` instead of the context passed
when the repo diagnoser is created.
New tests are included to reproduce this failure and verify
the fix.
- test: ensure server logs are uploaded to the repo
- test: honor cancellation in map storage
- test: repodiag context cancellation
Ref:
- #1691
* refactor(general): clarify comments
* refactor(general): rename `repodiag.logWriteSyncer`
This name reflects:
- the function of the struct; and
- the interface it implements (`zapcore.WriteSyncer`)
This name reflects the function of the struct / type.
The previous name (`Writer`) conveyed that this type
implemented the `Writer` interface / `Write` function,
which it does not.
Cleanups:
- use non-format variants of Log/Print with no additional args;
- fold in Fprintf call with no args into the following one;
- add missing arg placeholder in format strings;
- use require.Positive instead of Greater(..., 0);
- rename function to fillWithZeros to avoid collision with builtin clear;
- define type for context key to avoid collisions.
Use non-formatting logging functions for message without formatting.
For example, `log.Info("message")` instead of `log.Infof("message")`
Configure linter for printf-like functions
