Avoid minio allocations by implementing `ReadAt` function.
This change causes Minio libraries to reduce memory allocations.
Specifically, buffer allocations for S3 upload.
Add tests for new functionality.
---------
Co-authored-by: Shikhar Mall <shikhar.mall@veeam.com>
Co-authored-by: Nick <nick@kasten.io>
Change default 'allowWritesOnIndexLoad' to false when env var is unset
Add TestNoEpochAdvanceOnIndexRead
Ref:
- Followup to #3645
- Avoid index (epoch) cleanup and compaction during index reads #3638
- Make "read" commands/operations really read-only. #3639
Add:
- epoch.Manager.MaybeCompactSingleEpoch
- getCompactedEpochRange helper
- oldestUncompactedEpoch helper
- TestOldestUncompactedEpoch
- Tests for MaybeCompactSingleEpoch
Ref:
- Subset and dependency of #3651
- Depends on #3735
- Avoid index (epoch) cleanup and compaction during index reads #3638
- Make "read" commands/operations really read-only. #3639
Refactoring for the original implementation with intRange and
getKeyRange from closed-open ranges [lo, hi) to closed ranges: [lo, hi].
The primary motivation is for consistency with the implementation
of epoch.RangeMetadata in the same package, and thus avoid
confusion and reduce cognitive load.
Changes:
- adds a getContiguousKeyRange wrapper that checks for contiguity.
- getKeyRange simply returns a range with minimum and maximum
values for the keys in the map.
- changes the range implementation from closed-open ranges [lo, hi)
to closed ranges: [lo, hi] where both lo and hi are included in the range.
- Additional unit tests are included.
- renames intRange to closedIntRange to reflect new functionality.
Ref:
- Follow up refactor(general): add epoch.getKeyRange helper #3721
- Needed for refactor(general): add epoch.Manager.MaybeCompactSingleEpoch #3728
- Avoid index (epoch) cleanup and compaction during index reads #3638
* feat(repository): added metrics timeseries computation
This code is not used anywhere yet, will be used for reporting over
repository metrics.
* fixed checklocks
* Update internal/metrics/metrics_timeseries.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
---------
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
Conditionally disables epoch index maintenance operations when loading indexes.
This prevents (potentially expensive) cleanup write operations on the index
read path.
The behavior is controlled via the `epoch.Manager.allowCleanupWritesOnIndexLoad`
field, which can be temporarily overridden via an environment variable.
This override mechanism will be removed in the near future.
Refs:
- #3174
- #3224
- #3225
- #3638
- #3639
Refactor: move index compaction and cleanup out of refreshAttemptLocked
Introduces an `allowWritesOnLoadHelper` to check whether or not writes
can be performed when loading the indexes. Currently this is only
a function of whether the storage is in read-only mode. In the near
future, an explicit flag will be added to control this behavior.
Fix epoch manager: avoid single-epoch compaction when writes are disallowed.
Functional change: prevents compacting single epochs when writes are
disallowed, that is when using read-only storage. Currently, the epoch
manager will attempt to perform single-epoch compactions for all
eligible epochs, even on read-only storage.
Ref:
- #3224
- #3225
- #3638
- #3639
* Implement volume shadow copy support on Windows
* Update go-vss version
* Fix unused variables
* Rename upload_actions*.go files
* Move vss settings to a separate policy section
* Handle existing shadow copy root
* Fix tests
* Fix lint issues
* Add cli policy test
* Add OS snapshot integration test
* Add GitHub Actions VSS test
* Fix "Incorrect function" error for root VSS snapshots
* Rename err to finalErr in createOSSnapshot
* Add OSSnapshotMode test
* Do not modify paths starting with \\?\ on Windows
* Allow warning messages in logfile tests
* Fix ignorefs not wrapping OS snapshot directory
* Retry VSS creation if another op was in progress
---------
Co-authored-by: Jarek Kowalski <jaak@jkowalski.net>
Rename faketime.AutoAdvance parameter to start for clarity
Clarify faketime.ClockTimeWithOffset.Advance documentation
Refactor `faketime.TimeAdvance`:
Add `faketime.NewAutoAdvance` to create auto-advancing
`TimeAdvance` clocks.
Refactor faketime.NewTimeAdvance to return non-auto-advancing clocks,
which can still be explicitly advanced.
Removes the now unused `autoDelta` parameter, since it is always 0.
Rename function to `sm.setupCachesAndIndexManagers`
It matches the current implementation better.
Unexport `indexblob.ManagerV1.epochMgr`
Rename function for clarity to `advanceEpochMarker`
Cleanup logs after cleaning up epoch manager.
This is consistent with quick maintenance as well.
Make 'def' a test-local variable instead of a package-level
variable, it is not used outside this test.
Cleanup epoch test newTestEnv()
Misc:
* Reword comment and fix typo
* Fix log message
* `log.Debug` for non-formatted output
Unexport epoch.Manager.forceAdvanceEpoch. It is only used in tests.
Moved implementation to the `epoch_manager_test.go` file.
Remove `RetentionMode` and `RetentionPeriod` from `content.ManagerOptions` struct
* fix(repository): fixed handling of content.Info
Previously content.Info was an interface which was implemented by:
* index.InfoStruct
* index.indexEntryInfoV1
* index.indexEntryInfoV2
The last 2 implementations were relying on memory-mapped files
which in rare cases could be closed while Kopia was still processing
them leading to #2599.
This changes fixes the bug and strictly separates content.Info (which
is now always a struct) from the other two (which were renamed as
index.InfoReader and only used inside repo/content/...).
In addition to being safer, this _should_ reduce memory allocations.
* reduce the size of content.Info with proper alignment.
* pr feedback
* renamed index.InfoStruct to index.Info
* feat(server): reduce server refreshes of the repository
Previously each source would refresh itself from the repository
very frequently to determine the upcoming snapshot time. This change
refactors source manager so it does not own the repository connection
on its own but instead delegates all policy reads through the server.
Also introduces a new server scheduler that is responsible for
centrally managing the snapshot schedule and triggering snapshots
when they are due.
* Update cli/command_server_start.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update internal/server/server.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update internal/server/server_maintenance.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* pr feedback
---------
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
This is a generic scheduler that executes a loop consisting of:
- determining upcoming set of events
- waiting until the appropriate time
- triggering the events
Added improved providervalidation logic which tests for read-after-write
property between connections. The new test was failing before the change
and is now passing for Google Drive, OneDrive and DropBox.
* feat(repository): apply retention policies server-side
This allows append-only snapshots where the client can never delete
arbitrary manifests and policies are maintained on the server.
The client only needs permissions to create snapshots in a given, which
automatically gives them permission to invoke the server-side method
for their own snapshots only.
* Update cli/command_acl_add.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/api_manifest.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/api_manifest.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/grpc_session.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
---------
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
Previously some logs from a running server were only kept in memory
(including storage activity logs) which was confusing to many folks.
This changes the behavior so that logs are sent to their regular
(console/file) file locations in addition to the UI tasks.
Old behavior can be restored by adding `--no-persistent-logs` to
server.
