Fixes field names in CleanupMarkersStats persistent struct.
Also renames variables for consistency.
- Followup fix for: #4900 add stats to maintenance run - CleanupMarkers
- Ref: #4848
This is a breaking change to users who might be using Kopia as a library.
### Log Format
```json
{"t":"<timestamp-rfc-3389-microseconds>", "span:T1":"V1", "span:T2":"V2", "n":"<source>", "m":"<message>", /*parameters*/}
```
Where each record is associated with one or more spans that describe its scope:
* `"span:client": "<hash-of-username@hostname>"`
* `"span:repo": "<random>"` - random identifier of a repository connection (from `repo.Open`)
* `"span:maintenance": "<random>"` - random identifier of a maintenance session
* `"span:upload": "<hash-of-username@host:/path>"` - uniquely identifies upload session of a given directory
* `"span:checkpoint": "<random>"` - encapsulates each checkpoint operation during Upload
* `"span:server-session": "<random>"` -single client connection to the server
* `"span:flush": "<random>"` - encapsulates each Flush session
* `"span:maintenance": "<random>"` - encapsulates each maintenance operation
* `"span:loadIndex" : "<random>"` - encapsulates index loading operation
* `"span:emr" : "<random>"` - encapsulates epoch manager refresh
* `"span:writePack": "<pack-blob-ID>"` - encapsulates pack blob preparation and writing
(plus additional minor spans for various phases of the maintenance).
Notable points:
- Used internal zero allocation JSON writer for reduced memory usage.
- renamed `--disable-internal-log` to `--disable-repository-log` (controls saving blobs to repository)
- added `--disable-content-log` (controls writing of `content-log` files)
- all storage operations are also logged in a structural way and associated with the corresponding spans.
- all content IDs are logged in a truncated format (since first N bytes that are usually enough to be unique) to improve compressibility of logs (blob IDs are frequently repeated but content IDs usually appear just once).
This format should make it possible to recreate the journey of any single content throughout pack blobs, indexes and compaction events.
Track and report errors separately according to the type of error:
- missing pack
- truncated pack
- unreadable content.
Add a counter stat for the contents that are
read and fully verified (via `GetContent`).
Count errors grouped by pack ID using a `CountersMap`.
This allows determining the number of referenced contents
that were missing in a particular pack.
Report the counter stats via structured logging.
---
Sample output:
$ kopia content verify --progress-interval=0.5s --download-percent=100
Listing blobs...
Listed 102 blobs.
Verifying contents...
Verified 1 contents, 0 errors, estimating...
Verified 279 contents, 0 errors, estimating...
Verified 512 of 624 contents (82.1%), 0 errors, remaining 0s, ETA 2025-09-17 23:03:38 PDT
Finished verifying contents
verifyCounters: {"verifiedContents":624,"totalErrorCount":0,"contentsInMissingPacks":0,\
"contentsInTruncatedPacks":0,"unreadableContents":0,"readContents":624,\
"missingPacks":0,"truncatedPacks":0,"corruptedPacks":0}
- use `slices.Clone`
- remove stale `.gometalinter.json`
- unexport `maintenance.dropDeletedContents`
- rename `fetchIndexBlob`
- use `require` in `TestTimeFuncWiring`
Functional changes:
- Key derivation no longer panics, instead it returns an
error on failed condition checks.
- Migration to crypto/hkdf required changes in various
function signatures, adding an error return value for
propagating key derivation errors up the calling chain.
Tests were refactored to account for the signature change as well.
Non-functional changes:
- migrate to crypto/pbkdf2
- migrate to crypto/sha3
- migrate to crypto/hkdf
- add deriveHMACSecret function to simplify caller function
- simplify deriveHMACSecret
- fix test: specify non-empty HMACSecret
* remove unnecessary type argument
* modernize with max
* unexport getPartial and update comment
* unexport getFull
* verifyNotCached helper
* use require helpers
* leverage verify[Not]Cached
* use windowsOSName const
* fix comment wrapping
* require in stat_test
* use 512 as the write size and log allocated size
* rename const to expectedMinAllocSize
* write a single byte to test file
* add TestGetBlockSizeFromCurrentFS
* require Positive
* log before invariant check
While sorting by modification time is possible with OS tools,
such as `ls`, including the directory creation time in the
name facilitates cleanup in local dev environments where
tests fail often.
- enable `forcetypeassert` linter in non-test files
- add `//nolint` annotations
- add `testutil.EnsureType` helper for type assertions
- enable `forcetypeassert` linter in test files
- nit: rename var to packCountByPrefix
- leverage impossible package
- use maps.Clone
- unexport indirectObjectID
- unexport compressed
- rename function to flushBufferLocked
- add checklocks annotations to functions that must be called under w.mu
* fix(server): fixed server-based notifications
Used TypedEventArgs instead of `any` to ensure all notification data
carries type information, allowing the server to property deserialize it.
* fix
Minor cleanups in diff tests: helper to generate (direct) object ids.
- nit: rename variables for consistency
- nit: Use `k` prefix in oids where appropriate
- nit: remove unnecessary intermediate vars
- nit: specify known map length
Adds a `--stats-only` flag to the `diff` command to reduce
output verbosity.
When `kopia diff` is invoked with the `--stats-only` flag it
only outputs the aggregate statistics of changes.
As jsonencoding is used to pass some config parameters, move it to a
package that callers external to kopia can name. This ensures they can
actually configure those parameters if needed.
* use `os.CreateTemp` in `tempfile.CreateAutoDelete`
* refactor `TestTempFile`: add `VerifyTempfile` test helper
* add test for `createUnixFallback`
* rename function to `tempfile.CreateAutoDelete`
* remove the `dir` parameter to `tempfile.CreateAutoDelete`,
only an empty string was passed, apart from test cases.
* guard against panic in TestShadowCopy
* use uint8 for clarity
* unexport writeContentAsyncAndVerify
* fix typo in test function name
* remove commented interface functions
* use atomic.Int32
* cleanups in socket server activation test
* leverage stdlib's maps and slices packages
replace uses of `golang.org/x/exp/maps`
* nit: leverage `maps.Values`
