Use non-formatting logging functions for message without formatting.
For example, `log.Info("message")` instead of `log.Infof("message")`
Configure linter for printf-like functions
Followup cleanup for #3919
All the changes are in the `epoch` package
Summary:
- Removal of now unused code and tests.
- Refactoring to simplify test and remove a linter annotation.
- Fix a typo
While the current implementation should not produce non-contiguous
ranges, this invariant check may be too stringent, in particular for
repositories created and mutated by older kopia (client) processes.
For backwards compatibility the check is relaxed to allow operations to
proceed in older repositories.
Change default 'allowWritesOnIndexLoad' to false when env var is unset
Add TestNoEpochAdvanceOnIndexRead
Ref:
- Followup to #3645
- Avoid index (epoch) cleanup and compaction during index reads #3638
- Make "read" commands/operations really read-only. #3639
Add:
- epoch.Manager.MaybeCompactSingleEpoch
- getCompactedEpochRange helper
- oldestUncompactedEpoch helper
- TestOldestUncompactedEpoch
- Tests for MaybeCompactSingleEpoch
Ref:
- Subset and dependency of #3651
- Depends on #3735
- Avoid index (epoch) cleanup and compaction during index reads #3638
- Make "read" commands/operations really read-only. #3639
Refactoring for the original implementation with intRange and
getKeyRange from closed-open ranges [lo, hi) to closed ranges: [lo, hi].
The primary motivation is for consistency with the implementation
of epoch.RangeMetadata in the same package, and thus avoid
confusion and reduce cognitive load.
Changes:
- adds a getContiguousKeyRange wrapper that checks for contiguity.
- getKeyRange simply returns a range with minimum and maximum
values for the keys in the map.
- changes the range implementation from closed-open ranges [lo, hi)
to closed ranges: [lo, hi] where both lo and hi are included in the range.
- Additional unit tests are included.
- renames intRange to closedIntRange to reflect new functionality.
Ref:
- Follow up refactor(general): add epoch.getKeyRange helper #3721
- Needed for refactor(general): add epoch.Manager.MaybeCompactSingleEpoch #3728
- Avoid index (epoch) cleanup and compaction during index reads #3638
Conditionally disables epoch index maintenance operations when loading indexes.
This prevents (potentially expensive) cleanup write operations on the index
read path.
The behavior is controlled via the `epoch.Manager.allowCleanupWritesOnIndexLoad`
field, which can be temporarily overridden via an environment variable.
This override mechanism will be removed in the near future.
Refs:
- #3174
- #3224
- #3225
- #3638
- #3639
Refactor: move index compaction and cleanup out of refreshAttemptLocked
Introduces an `allowWritesOnLoadHelper` to check whether or not writes
can be performed when loading the indexes. Currently this is only
a function of whether the storage is in read-only mode. In the near
future, an explicit flag will be added to control this behavior.
Fix epoch manager: avoid single-epoch compaction when writes are disallowed.
Functional change: prevents compacting single epochs when writes are
disallowed, that is when using read-only storage. Currently, the epoch
manager will attempt to perform single-epoch compactions for all
eligible epochs, even on read-only storage.
Ref:
- #3224
- #3225
- #3638
- #3639
Rename faketime.AutoAdvance parameter to start for clarity
Clarify faketime.ClockTimeWithOffset.Advance documentation
Refactor `faketime.TimeAdvance`:
Add `faketime.NewAutoAdvance` to create auto-advancing
`TimeAdvance` clocks.
Refactor faketime.NewTimeAdvance to return non-auto-advancing clocks,
which can still be explicitly advanced.
Removes the now unused `autoDelta` parameter, since it is always 0.
Rename function to `sm.setupCachesAndIndexManagers`
It matches the current implementation better.
Unexport `indexblob.ManagerV1.epochMgr`
Rename function for clarity to `advanceEpochMarker`
Cleanup logs after cleaning up epoch manager.
This is consistent with quick maintenance as well.
Make 'def' a test-local variable instead of a package-level
variable, it is not used outside this test.
Cleanup epoch test newTestEnv()
Misc:
* Reword comment and fix typo
* Fix log message
* `log.Debug` for non-formatted output
Unexport epoch.Manager.forceAdvanceEpoch. It is only used in tests.
Moved implementation to the `epoch_manager_test.go` file.
Remove `RetentionMode` and `RetentionPeriod` from `content.ManagerOptions` struct
* Set epoch manager not to compact if readonly
Thread read-only parameter through the stack and make use of it when
loading epochs.
* Add test for manager read-only mode
Uses a bit of a contrived setup, but the state used in the test is
possible to reach given the right set of conditions and can cause the
manager to get stuck trying to refresh.
* Pull read-only status from Storage provider
* chore(ci): upgraded linter to 1.53.3
This flagged a bunch of unused parameters, so the PR is larger than
usual, but 99% mechanical.
* separate lint CI task
* run Lint in separate CI
Lint fixes in preparation for moving to Go 1.20
Remove deprecated calls to `rand.Seed`
In Go 1.20 the default generator is seeded randomly at program startup,
which is the desired behavior for these tests.
Remove uses of deprecated rand.Read: replace with calls to rand.Uint64()
Remove deprecated uses of rand.Read in content manager tests and
S3 versioned tests.
Adds a concurrency-safe helpers to provide functionality similar to that
provided by `rand.Read(b []byte) (int, error)`
Lack of generics support is blocking various dependency upgrades,
so this unblocks that.
Temporarily disabled `checklocks` linter until it is fixed upstream.
Instead of passing static content.FormattingOptions (and caching it)
we now introduce an interface to provide its values.
This will allow the values to dynamically change at runtime in the
future to support cases like live migration.
* sharded: plumbed through blob.PutOptions
* blob: removed blob.Storage.SetTime() method
This was only used for `kopia repo sync-to` and got replaced with
an equivalent blob.PutOptions.SetTime, which wehn set to non-zero time
will attempt to set the modification time on a file.
Since some providers don't support changing modification time, we
are able to emulate it using per-blob metadata (on B2, Azure and GCS),
sadly S3 is still unsupported, because it does not support returning
metadata in list results.
Also added PutOptions.GetTime, which when set to not nil, will
populate the provided variable with actual time that got assigned
to the blob.
Added tests that verify that each provider supports GetTime
and SetTime according to this spec.
* blob: additional test coverage for filesystem storage
* blob: added PutBlobAndGetMetadata() helper and used where appropriate
* fixed test failures
* pr feedback
* Update repo/blob/azure/azure_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* blobtesting: fixed object_locking_map.go
* blobtesting: removed SetTime from ObjectLockingMap
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
Previously epoch manager would discard (on read) any index blobs
associated with epoch N written after epoch N+2 has already started.
There was no signal to the writer that the write is actually bad. This
change reports slow writes as a failure and disables read-time filtering.
This is important for repositories copied using sync-to because it
may not necessarily preserve the modification time leading to problems
with restore.
