* feat(general): various notifications improvements
* added API to test notification profiles
* added --http-header to webhook notification configuration
* refactored configuration to always apply defaults before persisting options in the repository
* added 'notification profile show --profile-name=X' command
* more tests
* more test coverage
* report notification code coverage
This was caused by the client using key derivation algorithm
from a config file (which did not have it when it was generated
using old version of Kopia).
Fixes#4254
* feat(cli): send error notifications and snapshot reports
Notifications will be sent to all configured notification profiles
according to their severity levels.
The following events will trigger notifications:
- Snapshot is created (CLI only, severity >= report)
- Server Maintenance error occurs (CLI, server and UI, severity >= error)
- Any other CLI error occurs (CLI only, severity >= error).
A flag `--no-error-notifications` can be used to disable error notifications.
* added template tests
* improved time formatting in templates
* plumb through notifytemplate.Options
* more testing for formatting options
* fixed default date format to RFC1123
* Add rough estimation mechanism
* Extract data size estimation scan to standalone function
* Introduce estimation type flag
* Change signature of EstimatedDataSize
* Extract estimation logic and test it
* Adjust err handling in GetVolumeSizeInfo
* Make GetVolumeSizeInfo reusable
* Fix type issue for windows platform
* Use adaptive estimation with threshold
* Configure compressor for k and x prefixed content
Adds metadata compression setting to policy
Add support to configure compressor for k and x prefixed content
Set zstd-fastest as the default compressor for metadata in the policy
Adds support to set and show metadata compression to kopia policy commands
Adds metadata compression config to dir writer
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Pass concatenate options with ConcatenateOptions struct
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move content compression handling to caller
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move handling manifests to manifest pkg
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Correct const in server_test
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Remove unnecessary whitespace
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Disable metadata compression for < V2 format
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
---------
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
Checks whether any of the `httpServer.Serve*()` calls returns
`ErrServerClosed`.
Handles `ErrServerClosed` inside the `startServerWithOptionalTLS` function
instead of propagating it up. This means that `startServerWithOptionalTLS`
returns a nil error when the HTTP server is closed, so the caller does not need
to check for `ErrServerClosed`
No functional changes otherwise.
Snapshot verify can now accept a list of one or more snapshot IDs (manifest IDs) to be used as the subject(s) of the snapshot verify operation.
Added a unit test that ensures only the requested snapshots are verified when their IDs are provided.
Now in addition to the existing --source, --file-id, and --directory-id flags, which work the same, the user can provide snapshot IDs as args:
kopia snapshot verify <snapID> <snapID> <snapID>
Followups to #3655
* wrap fs.Reader
* nit: remove unnecessary intermediate variable
* nit: rename local variable
* cleanup: move restore.Progress interface to cli pkg
* move cliRestoreProgress to a separate file
* refactor(general): replace switch with if/else for clarity
Removes a tautology for `err == nil`, which was guaranteed
to be true in the second case statement for the switch.
Replacing the switch statement with and if/else block is clearer.
* initialize restoreProgress in restore command
* fix: use error.Wrapf with format string and args
Simplify SetCounters signature:
Pass arguments in a `restore.Stats` struct.
`SetCounters(s restore.Stats)`
Simplifies call sites and implementation.
In this case it makes sense to pass all the values
using the restore.Stats struct as it simplifies
the calls.
However, this pattern should be avoided in general
as it essentially makes all the arguments "optional".
This makes it easy to miss setting a value and simply
passing 0 (the default value), thus it becomes error
prone.
In this particular case, the struct is being passed
through verbatim, thus eliminating the risk of
missing a value, at least in the current state of
the code.
Generalize a couple of functions in the units package using generics.
This allows removing duplicate code and simplifying callers by removing unnecessary integer conversions.
Additional cleanups:
- make "/s" part of the Printf format string ;
- simplify setSizeMBParameter;
- generalize cli.maybeHumanReadable*` helpers;
- remove unneeded receiver in commandRepositorySetParameters helpers.
Cleanups:
- use non-format variants of Log/Print with no additional args;
- fold in Fprintf call with no args into the following one;
- add missing arg placeholder in format strings;
- use require.Positive instead of Greater(..., 0);
- rename function to fillWithZeros to avoid collision with builtin clear;
- define type for context key to avoid collisions.
Objectives:
- Facilitate the generation of valid password hashes that can be used with
the `server user --user-password` CLI command.
- Encapsulate implementation details of password hashing in
the `user` package.
Adds a new `server user hash-password` CLI command to generate the
hash from a supplied password.
Modifies the `server user set/add --user-password-hash` CLI command
to accept the password hash generated using the `hash-password`
command.
Adds `GetNewProfile(ctx, rep, username)` helper to move implementation
details to the `user` package.
Includes CLI and unit tests.
Cleans up and removes unused functions.
Ensure repository disconnection at the end of the `server start` CLI command.
This was caught as a result of fixing the test below.
Fix `TestServerStartInsecure`:
Remove `--password=xxx` parameter, which causes a server start failure
due to incorrect repo password, and not for the case being checked,
which is the lack of the `--insecure` parameter.
Update test comments accordingly.
Use non-formatting logging functions for message without formatting.
For example, `log.Info("message")` instead of `log.Infof("message")`
Configure linter for printf-like functions
Code movement and simplification, no functional changes.
Objectives:
- Allow callers specifying the needed key (or hash) size, instead of
hard-coding it in the registered PBK derivers. Conceptually, the caller
needs to specify the key size, since that is a requirement of the
(encryption) algorithm being used in the caller. Now, the code changes
here do not result in any functional changes since the key size is
always 32 bytes.
- Remove a global definition for the default PB key deriver to use.
Instead, each of the 3 use case sets the default value.
Changes:
- `crypto.DeriveKeyFromPassword` now takes a key size.
- Adds new constants for the key sizes at the callers.
- Removes the global `crypto.MasterKeySize` const.
- Removes the global `crypto.DefaultKeyDerivationAlgorithm` const.
- Adds const for the default derivation algorithms for each use case.
- Adds a const for the salt length in the `internal/user` package, to ensure
the same salt length is used in both hash versions.
- Unexports various functions, variables and constants in the `internal/crypto`
& `internal/user` packages.
- Renames various constants for consistency.
- Removes unused functions and symbols.
- Renames files to be consistent and better reflect the structure of the code.
- Adds a couple of tests to ensure the const values are in sync and supported.
- Fixes a couple of typos
Followups to:
- #3725
- #3770
- #3779
- #3799
- #3816
The individual commits show the code transformations to simplify the
review of the changes.
Reverts to using the `PasswordHashVersion` in the user profile.
Adds a simple mechanism for translating between password hash
version and the corresponding password hashing algorithm (key
derivation algorithm).
* User profile add set cli changes
* Add additional test
* get salt length from key deriver
* Fixes for failing tests
* after shikhars review
* lint fixes
* Fix restoring objects with I prefix
set default of snapshot-time to 'latest' as noted in the help output
* Change test of restore to check it works without a time given
This is because --snapshot-time defaults to "latest" now.
