Addresses failure introduced in #3963
Change: allow `delete-random-snapID` action in the function `tryDeleteAction`.
Test: Manual local run was successful.
The robustness tests set soft limits for content cache size
and metadata cache size. This PR adds the hard limits for
both the caches, thus reducing the memory required to
run the robustness tests for long term.
The soft limits are set for content cache and metadata cache.
This may cause the cache to bloat beyond expectations.
Setting hard limits for the size would keep the cache bloat in
check, thus reducing the memory needed to required to run
the robustness tests over a long time.
---------
Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
Ensure repository disconnection at the end of the `server start` CLI command.
This was caught as a result of fixing the test below.
Fix `TestServerStartInsecure`:
Remove `--password=xxx` parameter, which causes a server start failure
due to incorrect repo password, and not for the case being checked,
which is the lack of the `--insecure` parameter.
Update test comments accordingly.
Adds the maintenance action to multi-client robustness jobs.
The robustness jobs do not run a maintenance action until now.
---------
Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
The robustness tests perform delete actions on random subdirectories and
files. If the actions encounter `directory not found` error, the framework
returns a `no-op` error.
This change ignores the `no-op` errors specifically for "delete" actions.
This change will reduce the failure frequency when a delete actions on the
source directory result in a no-op.
Use non-formatting logging functions for message without formatting.
For example, `log.Info("message")` instead of `log.Infof("message")`
Configure linter for printf-like functions
The robustness test `TestRandomizedSmall` generate file action gets
picked up four to eight times more frequently than the other actions.
This PR reduces the frequency of the file generation action so that the
test is more representative of the user workflow.
The robustness framework runs do not cover delete actions at the moment.
This change adds the following actions to the robustness jobs
- delete random subdirectories
- delete contents of directories
Code movement and simplification, no functional changes.
Objectives:
- Allow callers specifying the needed key (or hash) size, instead of
hard-coding it in the registered PBK derivers. Conceptually, the caller
needs to specify the key size, since that is a requirement of the
(encryption) algorithm being used in the caller. Now, the code changes
here do not result in any functional changes since the key size is
always 32 bytes.
- Remove a global definition for the default PB key deriver to use.
Instead, each of the 3 use case sets the default value.
Changes:
- `crypto.DeriveKeyFromPassword` now takes a key size.
- Adds new constants for the key sizes at the callers.
- Removes the global `crypto.MasterKeySize` const.
- Removes the global `crypto.DefaultKeyDerivationAlgorithm` const.
- Adds const for the default derivation algorithms for each use case.
- Adds a const for the salt length in the `internal/user` package, to ensure
the same salt length is used in both hash versions.
- Unexports various functions, variables and constants in the `internal/crypto`
& `internal/user` packages.
- Renames various constants for consistency.
- Removes unused functions and symbols.
- Renames files to be consistent and better reflect the structure of the code.
- Adds a couple of tests to ensure the const values are in sync and supported.
- Fixes a couple of typos
Followups to:
- #3725
- #3770
- #3779
- #3799
- #3816
The individual commits show the code transformations to simplify the
review of the changes.
* Fix restoring objects with I prefix
set default of snapshot-time to 'latest' as noted in the help output
* Change test of restore to check it works without a time given
This is because --snapshot-time defaults to "latest" now.
* Implement volume shadow copy support on Windows
* Update go-vss version
* Fix unused variables
* Rename upload_actions*.go files
* Move vss settings to a separate policy section
* Handle existing shadow copy root
* Fix tests
* Fix lint issues
* Add cli policy test
* Add OS snapshot integration test
* Add GitHub Actions VSS test
* Fix "Incorrect function" error for root VSS snapshots
* Rename err to finalErr in createOSSnapshot
* Add OSSnapshotMode test
* Do not modify paths starting with \\?\ on Windows
* Allow warning messages in logfile tests
* Fix ignorefs not wrapping OS snapshot directory
* Retry VSS creation if another op was in progress
---------
Co-authored-by: Jarek Kowalski <jaak@jkowalski.net>
* refactor(test): allow signaling sub-process from testenv.CLIExeRunner
* test(cli): add test for handling SIGTERM
* feat(general): catch and process SIGTERM for termination
* refactor(cli): rename function cli.App.onTerminate
Renames function from onCtrlC to a more generic onTerminate
* fix(repository): fixed handling of content.Info
Previously content.Info was an interface which was implemented by:
* index.InfoStruct
* index.indexEntryInfoV1
* index.indexEntryInfoV2
The last 2 implementations were relying on memory-mapped files
which in rare cases could be closed while Kopia was still processing
them leading to #2599.
This changes fixes the bug and strictly separates content.Info (which
is now always a struct) from the other two (which were renamed as
index.InfoReader and only used inside repo/content/...).
In addition to being safer, this _should_ reduce memory allocations.
* reduce the size of content.Info with proper alignment.
* pr feedback
* renamed index.InfoStruct to index.Info
* feat(server): reduce server refreshes of the repository
Previously each source would refresh itself from the repository
very frequently to determine the upcoming snapshot time. This change
refactors source manager so it does not own the repository connection
on its own but instead delegates all policy reads through the server.
Also introduces a new server scheduler that is responsible for
centrally managing the snapshot schedule and triggering snapshots
when they are due.
* Update cli/command_server_start.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update internal/server/server.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update internal/server/server_maintenance.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* pr feedback
---------
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* feat(repository): apply retention policies server-side
This allows append-only snapshots where the client can never delete
arbitrary manifests and policies are maintained on the server.
The client only needs permissions to create snapshots in a given, which
automatically gives them permission to invoke the server-side method
for their own snapshots only.
* Update cli/command_acl_add.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/api_manifest.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/api_manifest.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/grpc_session.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
---------
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
Adds a check for snapshot create when --all and a source path are
given simultaneously. Returns an error in this case since it would
otherwise create a duplicate snapshot for the specified source path.
---
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
- fixed directory iteration order
- reduced providervalidation memory usage
- disabled one test case of TestSnapshotSparseRestore
(filed https://github.com/kopia/kopia/issues/3178 to fix)
* - Added test for changing themes
- Encapsulated the creation of a test repository in a function
- Added test for changing the byte representation
- Added test for changing the pagination
* - Fixed byte representation test
* - Fixed lint
* - Another lint fix
---------
Co-authored-by: lupusA <lupuapps@gmail.com>
Changes kopia's behavior to match the exit code that would
have been returned when the `--json` flag was not specified.
`kopia snapshot create my/path --json` terminates with a 0
status code in cases where
`kopia snapshot create my/path` terminates with a
non-zero exit code.
One such case is when there are permissions errors reading
files or directories to snapshot.
Adds end-to-end tests for snapshot create with '--json' flag
* chore(ci): upgraded linter to 1.53.3
This flagged a bunch of unused parameters, so the PR is larger than
usual, but 99% mechanical.
* separate lint CI task
* run Lint in separate CI
