Make purpose parameter a string.
Refactor TestDeriveKeyFromMasterKey
- make variables local to the test and rename accordingly
- make the purpose a `const (untyped) string`
Objective: make the tests more robust and reduce random failures.
Preliminary refactoring:
- Accept testing.TB in testenv helpers. This is needed to
use `require.EventuallyWithT` in socket activation tests.
- Rename parameters for clarity
Tests refactoring:
- use t.Cleanup instead of defer where appropriate
- create file handlers in test routine instead of go routines
- remove unnecessary var declaration
- increased wait time to 30 seconds.
- allow running socket activation test on Darwin
Ref:
- #3283
- #3313
- #3318
- upgrade to golangci-lint 2.6.1
- updates for gosec
- updates for govet
- updates for perfsprint
- updates modernize
Leaves out modernize:omitempty due to conflicts with tests
Fixes field names in CleanupMarkersStats persistent struct.
Also renames variables for consistency.
- Followup fix for: #4900 add stats to maintenance run - CleanupMarkers
- Ref: #4848
This is a breaking change to users who might be using Kopia as a library.
### Log Format
```json
{"t":"<timestamp-rfc-3389-microseconds>", "span:T1":"V1", "span:T2":"V2", "n":"<source>", "m":"<message>", /*parameters*/}
```
Where each record is associated with one or more spans that describe its scope:
* `"span:client": "<hash-of-username@hostname>"`
* `"span:repo": "<random>"` - random identifier of a repository connection (from `repo.Open`)
* `"span:maintenance": "<random>"` - random identifier of a maintenance session
* `"span:upload": "<hash-of-username@host:/path>"` - uniquely identifies upload session of a given directory
* `"span:checkpoint": "<random>"` - encapsulates each checkpoint operation during Upload
* `"span:server-session": "<random>"` -single client connection to the server
* `"span:flush": "<random>"` - encapsulates each Flush session
* `"span:maintenance": "<random>"` - encapsulates each maintenance operation
* `"span:loadIndex" : "<random>"` - encapsulates index loading operation
* `"span:emr" : "<random>"` - encapsulates epoch manager refresh
* `"span:writePack": "<pack-blob-ID>"` - encapsulates pack blob preparation and writing
(plus additional minor spans for various phases of the maintenance).
Notable points:
- Used internal zero allocation JSON writer for reduced memory usage.
- renamed `--disable-internal-log` to `--disable-repository-log` (controls saving blobs to repository)
- added `--disable-content-log` (controls writing of `content-log` files)
- all storage operations are also logged in a structural way and associated with the corresponding spans.
- all content IDs are logged in a truncated format (since first N bytes that are usually enough to be unique) to improve compressibility of logs (blob IDs are frequently repeated but content IDs usually appear just once).
This format should make it possible to recreate the journey of any single content throughout pack blobs, indexes and compaction events.
Track and report errors separately according to the type of error:
- missing pack
- truncated pack
- unreadable content.
Add a counter stat for the contents that are
read and fully verified (via `GetContent`).
Count errors grouped by pack ID using a `CountersMap`.
This allows determining the number of referenced contents
that were missing in a particular pack.
Report the counter stats via structured logging.
---
Sample output:
$ kopia content verify --progress-interval=0.5s --download-percent=100
Listing blobs...
Listed 102 blobs.
Verifying contents...
Verified 1 contents, 0 errors, estimating...
Verified 279 contents, 0 errors, estimating...
Verified 512 of 624 contents (82.1%), 0 errors, remaining 0s, ETA 2025-09-17 23:03:38 PDT
Finished verifying contents
verifyCounters: {"verifiedContents":624,"totalErrorCount":0,"contentsInMissingPacks":0,\
"contentsInTruncatedPacks":0,"unreadableContents":0,"readContents":624,\
"missingPacks":0,"truncatedPacks":0,"corruptedPacks":0}
- use `slices.Clone`
- remove stale `.gometalinter.json`
- unexport `maintenance.dropDeletedContents`
- rename `fetchIndexBlob`
- use `require` in `TestTimeFuncWiring`
Functional changes:
- Key derivation no longer panics, instead it returns an
error on failed condition checks.
- Migration to crypto/hkdf required changes in various
function signatures, adding an error return value for
propagating key derivation errors up the calling chain.
Tests were refactored to account for the signature change as well.
Non-functional changes:
- migrate to crypto/pbkdf2
- migrate to crypto/sha3
- migrate to crypto/hkdf
- add deriveHMACSecret function to simplify caller function
- simplify deriveHMACSecret
- fix test: specify non-empty HMACSecret
* remove unnecessary type argument
* modernize with max
* unexport getPartial and update comment
* unexport getFull
* verifyNotCached helper
* use require helpers
* leverage verify[Not]Cached
* use windowsOSName const
* fix comment wrapping
* require in stat_test
* use 512 as the write size and log allocated size
* rename const to expectedMinAllocSize
* write a single byte to test file
* add TestGetBlockSizeFromCurrentFS
* require Positive
* log before invariant check
