* feat(repository): added metrics timeseries computation
This code is not used anywhere yet, will be used for reporting over
repository metrics.
* fixed checklocks
* Update internal/metrics/metrics_timeseries.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
---------
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
Conditionally disables epoch index maintenance operations when loading indexes.
This prevents (potentially expensive) cleanup write operations on the index
read path.
The behavior is controlled via the `epoch.Manager.allowCleanupWritesOnIndexLoad`
field, which can be temporarily overridden via an environment variable.
This override mechanism will be removed in the near future.
Refs:
- #3174
- #3224
- #3225
- #3638
- #3639
Refactor: move index compaction and cleanup out of refreshAttemptLocked
Introduces an `allowWritesOnLoadHelper` to check whether or not writes
can be performed when loading the indexes. Currently this is only
a function of whether the storage is in read-only mode. In the near
future, an explicit flag will be added to control this behavior.
Fix epoch manager: avoid single-epoch compaction when writes are disallowed.
Functional change: prevents compacting single epochs when writes are
disallowed, that is when using read-only storage. Currently, the epoch
manager will attempt to perform single-epoch compactions for all
eligible epochs, even on read-only storage.
Ref:
- #3224
- #3225
- #3638
- #3639
* Implement volume shadow copy support on Windows
* Update go-vss version
* Fix unused variables
* Rename upload_actions*.go files
* Move vss settings to a separate policy section
* Handle existing shadow copy root
* Fix tests
* Fix lint issues
* Add cli policy test
* Add OS snapshot integration test
* Add GitHub Actions VSS test
* Fix "Incorrect function" error for root VSS snapshots
* Rename err to finalErr in createOSSnapshot
* Add OSSnapshotMode test
* Do not modify paths starting with \\?\ on Windows
* Allow warning messages in logfile tests
* Fix ignorefs not wrapping OS snapshot directory
* Retry VSS creation if another op was in progress
---------
Co-authored-by: Jarek Kowalski <jaak@jkowalski.net>
Rename faketime.AutoAdvance parameter to start for clarity
Clarify faketime.ClockTimeWithOffset.Advance documentation
Refactor `faketime.TimeAdvance`:
Add `faketime.NewAutoAdvance` to create auto-advancing
`TimeAdvance` clocks.
Refactor faketime.NewTimeAdvance to return non-auto-advancing clocks,
which can still be explicitly advanced.
Removes the now unused `autoDelta` parameter, since it is always 0.
Rename function to `sm.setupCachesAndIndexManagers`
It matches the current implementation better.
Unexport `indexblob.ManagerV1.epochMgr`
Rename function for clarity to `advanceEpochMarker`
Cleanup logs after cleaning up epoch manager.
This is consistent with quick maintenance as well.
Make 'def' a test-local variable instead of a package-level
variable, it is not used outside this test.
Cleanup epoch test newTestEnv()
Misc:
* Reword comment and fix typo
* Fix log message
* `log.Debug` for non-formatted output
Unexport epoch.Manager.forceAdvanceEpoch. It is only used in tests.
Moved implementation to the `epoch_manager_test.go` file.
Remove `RetentionMode` and `RetentionPeriod` from `content.ManagerOptions` struct
* fix(repository): fixed handling of content.Info
Previously content.Info was an interface which was implemented by:
* index.InfoStruct
* index.indexEntryInfoV1
* index.indexEntryInfoV2
The last 2 implementations were relying on memory-mapped files
which in rare cases could be closed while Kopia was still processing
them leading to #2599.
This changes fixes the bug and strictly separates content.Info (which
is now always a struct) from the other two (which were renamed as
index.InfoReader and only used inside repo/content/...).
In addition to being safer, this _should_ reduce memory allocations.
* reduce the size of content.Info with proper alignment.
* pr feedback
* renamed index.InfoStruct to index.Info
* feat(server): reduce server refreshes of the repository
Previously each source would refresh itself from the repository
very frequently to determine the upcoming snapshot time. This change
refactors source manager so it does not own the repository connection
on its own but instead delegates all policy reads through the server.
Also introduces a new server scheduler that is responsible for
centrally managing the snapshot schedule and triggering snapshots
when they are due.
* Update cli/command_server_start.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update internal/server/server.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update internal/server/server_maintenance.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* pr feedback
---------
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
This is a generic scheduler that executes a loop consisting of:
- determining upcoming set of events
- waiting until the appropriate time
- triggering the events
Added improved providervalidation logic which tests for read-after-write
property between connections. The new test was failing before the change
and is now passing for Google Drive, OneDrive and DropBox.
* feat(repository): apply retention policies server-side
This allows append-only snapshots where the client can never delete
arbitrary manifests and policies are maintained on the server.
The client only needs permissions to create snapshots in a given, which
automatically gives them permission to invoke the server-side method
for their own snapshots only.
* Update cli/command_acl_add.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/api_manifest.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/api_manifest.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/grpc_session.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
---------
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
Previously some logs from a running server were only kept in memory
(including storage activity logs) which was confusing to many folks.
This changes the behavior so that logs are sent to their regular
(console/file) file locations in addition to the UI tasks.
Old behavior can be restored by adding `--no-persistent-logs` to
server.
* Set epoch manager not to compact if readonly
Thread read-only parameter through the stack and make use of it when
loading epochs.
* Add test for manager read-only mode
Uses a bit of a contrived setup, but the state used in the test is
possible to reach given the right set of conditions and can cause the
manager to get stuck trying to refresh.
* Pull read-only status from Storage provider
* Rename UnsupportedBlobRetention struct
Rename this struct to DefaultProviderImplementation in preparation for
adding other simple "default" functionality to it.
* Add other functions to default provider
Add other simple function implementations to the default provider so
that other providers can just embed this to get basic behavior.
* Cleanup existing users of default provider
* Add default provider to remaining storage types
Add the default provider to remaining storage providers and remove
functions that are now implemented by the default provider.
* Add new blob.Storage call to see if it's readonly
Return whether the storage is readonly so higher layers in the stack can
selectively disable some functionality if needed, like compaction.
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
This improves the performance of gather writes (p and q blobs) in
particular on high-latency links by reducing round-trips.
Tested by uploading Kopia working directory
(3 GB, files:32157 dirs:6051) to SFTP server on LAN (TrueNAS) over WiFi:
Before: 2m4s (~24 MB/s)
After: 1m13s (~41 MB/s)
* Store and return retention info in test storage
Add a new interface and function that allows getting retention
information during testing. This allows for more exact comparisons about
retention duration and mode in tests.
* Fixup how blobtesting retention extension works
Use the clock instead of the object's mod time so that extensions are
from the "current time." This aligns with how the S3 blob storage
functions.
* Update retention tests to use more precise checks
Where possible, use the information returned by GetRetention in tests
that deal with retention information. This allows for more precise
comparions of retention duration and mode instead of indirectly testing
duration by advancing the clock and attempting to modify blobs.
fix(general): change cache full logging frequency to once every 10 minutes
Addresses concerns about too many messages in the logs (#3085)
fix(general): avoid logging cache key.
Instead log the cache description, which provides information about
the types of contents being cached.
Followups to #3085
- fixed directory iteration order
- reduced providervalidation memory usage
- disabled one test case of TestSnapshotSparseRestore
(filed https://github.com/kopia/kopia/issues/3178 to fix)
* More robust error comparisons in retention tests
Update tests for retention to use `ErrorIs` checks instead of comparing
error messages.
* Use `require.NoError` in retention tests
Minor cleanup to reduce branches in code by using `require.NoError`
instead of if-blocks and `t.Fatal`.
* feat(snapshots): added ability to use cron expressions to schedule snapshots
We use `github.com/hashicorp/cronexpr` to parse and evaluate expressions,
as documented in https://github.com/hashicorp/cronexpr#implementation
* upgrade ui
* pr feedback
* Implement ability to extend retention time on S3 buckets using Object Locks
* Move object-lock extension to maintenance.Params.
* Use a default function for unsupported extensions instead of duplicating code
* Fix potential lockup during object-lock extension
* Fix race condition. Add more code coverage
* rebase to V3
* Add checks to prevent user from setting Retention Period < Full Maintenance Interval
---------
Co-authored-by: Ashlie Martinez <ashmrtnz@alcion.ai>
* chore(ci): upgraded linter to 1.53.3
This flagged a bunch of unused parameters, so the PR is larger than
usual, but 99% mechanical.
* separate lint CI task
* run Lint in separate CI
