* Add rough estimation mechanism
* Extract data size estimation scan to standalone function
* Introduce estimation type flag
* Change signature of EstimatedDataSize
* Extract estimation logic and test it
* Adjust err handling in GetVolumeSizeInfo
* Make GetVolumeSizeInfo reusable
* Fix type issue for windows platform
* Use adaptive estimation with threshold
* Configure compressor for k and x prefixed content
Adds metadata compression setting to policy
Add support to configure compressor for k and x prefixed content
Set zstd-fastest as the default compressor for metadata in the policy
Adds support to set and show metadata compression to kopia policy commands
Adds metadata compression config to dir writer
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Pass concatenate options with ConcatenateOptions struct
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move content compression handling to caller
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move handling manifests to manifest pkg
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Correct const in server_test
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Remove unnecessary whitespace
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Disable metadata compression for < V2 format
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
---------
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
Followups to #3655
* wrap fs.Reader
* nit: remove unnecessary intermediate variable
* nit: rename local variable
* cleanup: move restore.Progress interface to cli pkg
* move cliRestoreProgress to a separate file
* refactor(general): replace switch with if/else for clarity
Removes a tautology for `err == nil`, which was guaranteed
to be true in the second case statement for the switch.
Replacing the switch statement with and if/else block is clearer.
* initialize restoreProgress in restore command
* fix: use error.Wrapf with format string and args
Simplify SetCounters signature:
Pass arguments in a `restore.Stats` struct.
`SetCounters(s restore.Stats)`
Simplifies call sites and implementation.
In this case it makes sense to pass all the values
using the restore.Stats struct as it simplifies
the calls.
However, this pattern should be avoided in general
as it essentially makes all the arguments "optional".
This makes it easy to miss setting a value and simply
passing 0 (the default value), thus it becomes error
prone.
In this particular case, the struct is being passed
through verbatim, thus eliminating the risk of
missing a value, at least in the current state of
the code.
Cleanups:
- use non-format variants of Log/Print with no additional args;
- fold in Fprintf call with no args into the following one;
- add missing arg placeholder in format strings;
- use require.Positive instead of Greater(..., 0);
- rename function to fillWithZeros to avoid collision with builtin clear;
- define type for context key to avoid collisions.
Use non-formatting logging functions for message without formatting.
For example, `log.Info("message")` instead of `log.Infof("message")`
Configure linter for printf-like functions
* Implement volume shadow copy support on Windows
* Update go-vss version
* Fix unused variables
* Rename upload_actions*.go files
* Move vss settings to a separate policy section
* Handle existing shadow copy root
* Fix tests
* Fix lint issues
* Add cli policy test
* Add OS snapshot integration test
* Add GitHub Actions VSS test
* Fix "Incorrect function" error for root VSS snapshots
* Rename err to finalErr in createOSSnapshot
* Add OSSnapshotMode test
* Do not modify paths starting with \\?\ on Windows
* Allow warning messages in logfile tests
* Fix ignorefs not wrapping OS snapshot directory
* Retry VSS creation if another op was in progress
---------
Co-authored-by: Jarek Kowalski <jaak@jkowalski.net>
Rename faketime.AutoAdvance parameter to start for clarity
Clarify faketime.ClockTimeWithOffset.Advance documentation
Refactor `faketime.TimeAdvance`:
Add `faketime.NewAutoAdvance` to create auto-advancing
`TimeAdvance` clocks.
Refactor faketime.NewTimeAdvance to return non-auto-advancing clocks,
which can still be explicitly advanced.
Removes the now unused `autoDelta` parameter, since it is always 0.
Rename function to `sm.setupCachesAndIndexManagers`
It matches the current implementation better.
Unexport `indexblob.ManagerV1.epochMgr`
Rename function for clarity to `advanceEpochMarker`
Cleanup logs after cleaning up epoch manager.
This is consistent with quick maintenance as well.
Make 'def' a test-local variable instead of a package-level
variable, it is not used outside this test.
Cleanup epoch test newTestEnv()
Misc:
* Reword comment and fix typo
* Fix log message
* `log.Debug` for non-formatted output
Reorder streaming file upload defer functions slightly so the reader is
closed prior to calling FinishedFile. This allows folks who hook into
those callbacks to make stronger assumptions about the ordering/state of
things if needed.
* chore(ci): upgraded linter to 1.53.3
This flagged a bunch of unused parameters, so the PR is larger than
usual, but 99% mechanical.
* separate lint CI task
* run Lint in separate CI
* Enable compression for StreamingFile entries
Also take fs.Entry for helper that gets compression policy instead of
fs.File.
* Add simple test for compression on StreamingFiles
Just checks that the repo size is smaller than the size it would be if
compression wasn't enabled.
* Return ReadCloser from StreamingFile
Allow better resource management by returning something that can be closed
when dealing with StreamingFiles.
* Close StreamingFile Reader during upload
* Use NopCloser on inputs that don't implement Close
Fixup callers of the StreamingFile API by wrapping regular Readers with
NopCloser calls where necessary.
* Properly populate file size for cached entries
StreamingFiles don't know their size until they are read. This leads to
entries marked with size 0 because size is not compared when determining
if something is cached or not as the size is sourced from the current
(unread) entry.
Instead, create the dir entry from the previous entry. As StreamingFile
do not allow setting file mode, permissions, owner, etc using the old
entry suffices.
* Use cached entry size for stats
Also use the cached entry size when calculating stats since
StreamingFile has 0 size.
* Update tests for cached files to check size
Check at least the total size of the snapshot is updated when handling
StreamingFiles.
Also modified an end-to-end test to also check that these extra mode flags work when snapshotting+restoring.
Manually tested fuse-mount.
Co-authored-by: Luca Citi <lciti@ieee.org>
* Apply labels to checkpoint snapshot manifests
This allows looking up checkpoint snapshot manifests by label if
desired.
* Fixup other calls to Upload function
* Fix missed Upload call
* Revert code to pass checkpoint labels as arg
This reverts commits
* bf844ff068
* 76438c9bc1
* d748b6361b
* Add checkpoint labels to checkpoint manifests
Allow finer-grained lookups on checkpoints by adding labels to them.
Labels are specified in the Uploader struct and apply to all checkpoints
for a snapshot.
* Test checkpoint labels.
Be strict about testing and have a separate copy of the labels.
Almost all were easy to replace, except ones exposed via JSON which
have been left as-is.
The linter has a cool behavior where it flags attempts to pass
`atomic.Int32` for example by value , which is always a mistake,
say as an argument to `fmt.Sprintf()`
This removes tons of boilerplate code around:
- retry loop
- connection management
- storage registration
* used generics in runInParallel
* introduced generics in freepool
* introduced strong typing for workshare.Pool and workshare.AsyncGroup
* fixed linter error on openbsd
Lack of generics support is blocking various dependency upgrades,
so this unblocks that.
Temporarily disabled `checklocks` linter until it is fixed upstream.
* Update display on repository summary
* Apply throughout app
* Situate units_test
* Update Command Line documentation
* Envar cleanup
* Rename to BytesString
* Restore envar string available for test
* Remove extraneous empty check and restore UIPreferences field for frontend
* PR: config bool cleanup and missed `BaseEnv`s
* Fix lint and test
* feat(cli): Allow restore from snapshoted path
* Find files in multiple snapshots
* Added --snapshot-time to restore
* Added restore by path test
* More timespec formats
* Test for snapshot list with a file in multiple snapshots
* Handle restore without target path
* Fix for tests
* Made changes requested in PR and rebased
* Make callback for upload file completion
Callback does not indicate that a file will be reachable immediately in
the resulting snapshot, but does indicate that the uploader is done
processing the file in some way (either via uploading data or finding a
previous version in the repo) and whether there was an error processing
the file.
* Tests for new FinishedFile callback
Ensure hadErr is properly populated and FinishedFile is called even if
the file was considered cached.
* Refine comment on interface function slightly
* Give callback error instead of bool about error
* Add locks around concurrent accesses in test
* Allow setting mod time on StreamingFiles
Only set during struct creation. Default the old constructor to using
the current time as the mod time.
* Change how mod time is handled for StreamingFiles
Don't set StreamingFile mod time in the uploader, instead use the value
in the file's metadata. Also allows StreamingFiles to be recognized as
cached files (previously uploaded). StreamingFiles don't know their file
size until they've been completely uploaded so leaving that out makes
them eligible for being marked as "cached".
This commit combined with the previous commit slightly changes how
timestamps on StreamingFiles are handled. It will result in them having
slightly earlier timestamps because they are now set on struct creation
instead of when the file was uploaded.
As timestamps are fairly fine-grained and the default is to use the
current time as the mod time it seems unlikely this patch will result in
incorrectly thinking a StreamingFile is cached even though it has
changed size.
* Uploader test for StreamingFile caching
Also hide the flag, since it's not recommended to be tweaked anyway.
The value of <=45m is very important for safety of the garbage collection algorithms - too long an interval between checkpoints could mean that GC treats contents in the middle of being uploaded as unused, because they are not reachable from any snapshots or checkpoints.
Fixes#2193
