Cleanup nits:
- get error handling policy upfront and improve readability in uploader
- update error message
- update field documentation and update flag description
- remove unused function
- const `isWindows` and remove redundant condition check
- add `getEnvVarBool` helper
- refactor common helper for mockfs.AddError* functions, and
add mockfs.AddErrorEntry<Type> wrappers for clarity.
- removed list of skipped tests from gotestsum summary
Revert "feat(snapshots): localfs support for passing options (#5044)"
commit c8c4615595.
Fix: return `ErrorEntry` for permission denied instead of aborting
When iterating a directory, if `lstat` fails with permission denied on
an entry, return an `ErrorEntry` instead of an error that stops the
entire directory iteration.
Previously, a single inaccessible entry, such as, a FUSE/sshfs mount
owned by another user, would cause the entire containing directory to
fail and be omitted from the snapshot, resulting in data loss.
Now, the inaccessible entry is returned as an ErrorEntry which is
handled according to the configured error handling policy, allowing
iteration to continue and the rest of the directory to be backed up.
- Fixes: #5045
Differentiate entry type when ignoring failed entries
Fix tests for new behavior, including handling timing-dependent
behavior when snapshots --fail-fast
---------
Co-authored-by: Geoffrey D. Bennett <g@netcraft.com.au>
- upgrade to golangci-lint 2.6.1
- updates for gosec
- updates for govet
- updates for perfsprint
- updates modernize
Leaves out modernize:omitempty due to conflicts with tests
This is a breaking change to users who might be using Kopia as a library.
### Log Format
```json
{"t":"<timestamp-rfc-3389-microseconds>", "span:T1":"V1", "span:T2":"V2", "n":"<source>", "m":"<message>", /*parameters*/}
```
Where each record is associated with one or more spans that describe its scope:
* `"span:client": "<hash-of-username@hostname>"`
* `"span:repo": "<random>"` - random identifier of a repository connection (from `repo.Open`)
* `"span:maintenance": "<random>"` - random identifier of a maintenance session
* `"span:upload": "<hash-of-username@host:/path>"` - uniquely identifies upload session of a given directory
* `"span:checkpoint": "<random>"` - encapsulates each checkpoint operation during Upload
* `"span:server-session": "<random>"` -single client connection to the server
* `"span:flush": "<random>"` - encapsulates each Flush session
* `"span:maintenance": "<random>"` - encapsulates each maintenance operation
* `"span:loadIndex" : "<random>"` - encapsulates index loading operation
* `"span:emr" : "<random>"` - encapsulates epoch manager refresh
* `"span:writePack": "<pack-blob-ID>"` - encapsulates pack blob preparation and writing
(plus additional minor spans for various phases of the maintenance).
Notable points:
- Used internal zero allocation JSON writer for reduced memory usage.
- renamed `--disable-internal-log` to `--disable-repository-log` (controls saving blobs to repository)
- added `--disable-content-log` (controls writing of `content-log` files)
- all storage operations are also logged in a structural way and associated with the corresponding spans.
- all content IDs are logged in a truncated format (since first N bytes that are usually enough to be unique) to improve compressibility of logs (blob IDs are frequently repeated but content IDs usually appear just once).
This format should make it possible to recreate the journey of any single content throughout pack blobs, indexes and compaction events.
- use snapshot end time as tie breaker for equal start time
- pass prototype snapshot manifest in uploader
- remove -1 ns hack for checkpoints and fix corresponding test
- test snapshot.SortByTime
- enable `forcetypeassert` linter in non-test files
- add `//nolint` annotations
- add `testutil.EnsureType` helper for type assertions
- enable `forcetypeassert` linter in test files
- use 'require/assert'
- refactor TestUploadMetadataCompression as a table test
- allow tests to run in parallel
- use t.Cleanup and add a missing cleanup
- use maps.Clone
* fix(snapshots): Remove checkpoints after a complete snapshot
... by setting start time of checkpoints one nanosecond earlier than
that of the snapshot.
* test(snapshots): Test for leftover checkpoints
* fix linter issues
* removed stray curly brace
---------
Co-authored-by: Jarek Kowalski <jaak@jkowalski.net>
