Cleanups:
- use non-format variants of Log/Print with no additional args;
- fold in Fprintf call with no args into the following one;
- add missing arg placeholder in format strings;
- use require.Positive instead of Greater(..., 0);
- rename function to fillWithZeros to avoid collision with builtin clear;
- define type for context key to avoid collisions.
Addresses failure introduced in #3963
Change: allow `delete-random-snapID` action in the function `tryDeleteAction`.
Test: Manual local run was successful.
The robustness tests set soft limits for content cache size
and metadata cache size. This PR adds the hard limits for
both the caches, thus reducing the memory required to
run the robustness tests for long term.
The soft limits are set for content cache and metadata cache.
This may cause the cache to bloat beyond expectations.
Setting hard limits for the size would keep the cache bloat in
check, thus reducing the memory needed to required to run
the robustness tests over a long time.
---------
Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
Adds the maintenance action to multi-client robustness jobs.
The robustness jobs do not run a maintenance action until now.
---------
Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
The robustness tests perform delete actions on random subdirectories and
files. If the actions encounter `directory not found` error, the framework
returns a `no-op` error.
This change ignores the `no-op` errors specifically for "delete" actions.
This change will reduce the failure frequency when a delete actions on the
source directory result in a no-op.
The robustness test `TestRandomizedSmall` generate file action gets
picked up four to eight times more frequently than the other actions.
This PR reduces the frequency of the file generation action so that the
test is more representative of the user workflow.
The robustness framework runs do not cover delete actions at the moment.
This change adds the following actions to the robustness jobs
- delete random subdirectories
- delete contents of directories
Almost all were easy to replace, except ones exposed via JSON which
have been left as-is.
The linter has a cool behavior where it flags attempts to pass
`atomic.Int32` for example by value , which is always a mistake,
say as an argument to `fmt.Sprintf()`
Lack of generics support is blocking various dependency upgrades,
so this unblocks that.
Temporarily disabled `checklocks` linter until it is fixed upstream.
* Move upgrade repository test function
* Delete test for deleted function
* Move upgrade repository test function
* Resolve conflicts
* Check for error in status function
* Add unit test
- nit: re-group struct fields
- nit: use consts
- nit: remove unnecessary fmt.Errorf(...) usage
- nit: avoid unnecessarily calling defaultActionControls when there is already a value
- robustness: increase readability in actions map declaration
- Prefer named functions over closures for the actions.
- nit fix typo
- nit: simplify robustness Log.StringThisRun
Iterates only over the tail of the slice, which avoids iterating over the entire slice
From https://github.com/google/gvisor/tree/master/tools/checklocks
This will perform static verification that we're using
`sync.Mutex`, `sync.RWMutex` and `atomic` correctly to guard access
to certain fields.
This was mostly just a matter of adding annotations to indicate which
fields are guarded by which mutex.
In a handful of places the code had to be refactored to allow static
analyzer to do its job better or to not be confused by some
constructs.
In one place this actually uncovered a bug where a function was not
releasing a lock properly in an error case.
The check is part of `make lint` but can also be invoked by
`make check-locks`.
* fixed new gocritic violations
* fixed new 'contextcheck' violations
* fixed 'gosec' warnings
* suppressed ireturn and varnamelen linters
* fixed tenv violations, enabled building robustness tests on arm64
* fixed remaining linux failures
* makefile: fixed 'lint-all' target when running on arm64
* linter: increase deadline
* disable nilnil linter - to be enabled in separate PR
The dual time measurement is described in
https://go.googlesource.com/proposal/+/master/design/12914-monotonic.md
The fix is to discard hidden monotonic time component of time.Time
by converting to unix time and back.
Reviewed usage of clock.Now() and replaced with timetrack.StartTimer()
when measuring time.
The problem in #1402 was that passage of time was measured using
the monotonic time and not wall clock time. When the computer goes
to sleep, monotonic time is still monotonic while wall clock time makes
a leap when the computer wakes up. This is the behavior that
epoch manager (and most other compontents in Kopia) rely upon.
Fixes#1402
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
* refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* chore: remove //nolint:gosec for os.ReadFile
At the time of this commit, the G304 rule of gosec does not include the
`os.ReadFile` function. We remove `//nolint:gosec` temporarily until
https://github.com/securego/gosec/pull/706 is merged.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Detailed changes:
* Implement Kopia API client
* Implement kopia client
* Implement new persister
* Separate storeLoad and Delete concurrency tests
* Update Store interface to return error for Delete
* Return errors from os.RemoveAll
* Robustness test updates
* Push robustness metadata without writing to fs
* Fix testenv.AssertNoError references
* Update tests to use new kopia persister
* Minor updates to multiclient test cases
* Use require.NoError instead of assertNoError
* Add context to store interface
* Update logging to be less verbose
* Use io instead of ioutil package
* Simplify restore by using object ID
* Accommodate repository.NewWriter signature change
* Improve tests to increase code coverage
* Spelling and error string fixes
* Address lint errors
Co-authored-by: Nick <nick@kasten.io>
* content: fixed data race in IterateUnreferencedBlobs
* upload: fixed data race between uploader and estimator
* testing: fixed data race in repo/blob/logging test
* makefile: run tests on CI/linux/amd64 with -race
* robustness: fixed test race
* content: fixed data race getContentDataUnlocked that triggers TestParallelWrites - looks scary but in practice very hard to trigger in real life and does not cause data corruption
* testing: reduce test complexity under race detector
* server: fixed minor race in refreshStatus()
* testing: reduced depth of sharedTestDataDir2
* ci: run race detector in separate job
* ci: run unit test race detector in parallel to integration tests
Protect filesystem subtrees from concurrent manipulation during critical sections
if engine actions are called asynchronously. This change provides coordination
between the `Snapshotter` and the `FileWriter`. For example, the `FileWriter`
should be blocked from perturbing the same directory tree if a
Gather-Snapshot is taking place along that tree simultaneously.
This will ensure the fingerprint data accumulated during the `Gather` phase
will correspond unambiguously to the data included in the snapshot.
Extend build flags to kopia snapshotter
This package now imports fswalker which can only be built for
darwin,amd64 or linux,amd64
Add metadata R/W locking for asynchronous accesses to robustness engine metadata.
Remove the index from the Store interface and maintain it only in Checker, where it's used.
* Path lock utility for coordination between robustness engine actions
Add a utility to ensure path-based synchronization between goroutines. If a path is locked, a subsequent Lock will block if the requested path is the same, or a child/parent (recursive), of the locked path.
This assists with coordination between asynchronous robustness engine actions that may rely on the underlying data directory remaining unchanged. For example:
- between gathering a filesystem fingerprint and taking a snapshot.
- when one WriteFilesAtDepth command has traversed into a directory that another goroutine has picked for deletion.
* Fix linter
