* Use an AssumeRole credential provider if a role was specified
* Use minio's assumerole call
* Update assume role test
---------
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
This allows persisting and reading back the timeout as a duration.
There are be no backwards compatibility issues, since the field is
not currently set, so it is not persisted in the configuration file.
Allow the use of a client certificate when authenticating
to an Azure Blob storage provider.
Tests included.
Credit: @DeepikaDixit
Authored-by: Deepika Dixit <DeepikaDixit@users.noreply.github.com>
* use uint8 for clarity
* unexport writeContentAsyncAndVerify
* fix typo in test function name
* remove commented interface functions
* use atomic.Int32
* cleanups in socket server activation test
* leverage stdlib's maps and slices packages
replace uses of `golang.org/x/exp/maps`
* nit: leverage `maps.Values`
Sets `repo.BuildInfo` and `repo.BuildVersion` when they are
not specified via link flags.
The behavior for binaries built via CI and Make remains the same.
Sets `repo.Version` to `v0-unofficial` during tests.
This was caused by the client using key derivation algorithm
from a config file (which did not have it when it was generated
using old version of Kopia).
Fixes#4254
Objective: make it easier to understand the tests
Changes:
- Introduce helpers to simplify the tests and increase clarity.
* verifySortedEntries
* addDeterministicContents
* addContentIDsWithDifferentPrefixes
* addIntsAsDeterministicContent
- Rename tests to better convey what they are testing.
- Use rand.Perm for simplicity
Ref: #4139
* Configure compressor for k and x prefixed content
Adds metadata compression setting to policy
Add support to configure compressor for k and x prefixed content
Set zstd-fastest as the default compressor for metadata in the policy
Adds support to set and show metadata compression to kopia policy commands
Adds metadata compression config to dir writer
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Pass concatenate options with ConcatenateOptions struct
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move content compression handling to caller
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move handling manifests to manifest pkg
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Correct const in server_test
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Remove unnecessary whitespace
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Disable metadata compression for < V2 format
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
---------
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
Changes:
* test that quick maintenance runs when epoch manager is enabled
* fix(general): run epoch maintenance for quick maintenance
Change based on a known-to-be-safe portion of the changes proposed in #3901
* cleanup: pass epoch manager to `runTaskEpochMaintenanceQuick`
The caller needs to get the epoch manager to determine
whether or not the epoch manager is enabled. The caller
now passes the epoch manager to `runTaskEpochMaintenanceQuick`
* wrap the error inside runTaskEpochMaintenanceQuick
* index builder for epoch index compaction
* index builder for epoch index compaction: fix CI errors
* index builder for epoch index compaction: UT for OneUseBuilder
* index builder for epoch index compaction: fix CI errors
* index builder for epoch index compaction: use *Info as builder item
* index builder for epoch index compaction: fix CI errors
* index builder for epoch index compaction: fix CI errors
* index builder for epoch index compaction: fix CI errors
* Change struct for tracking committed content
Committed content only ever has a value of 'true' for committed so use
an empty struct and an existance check instead.
* Don't copy committed manifest set for compaction
Assuming the number of committed manifests is much larger than the
number of manifest updates, it seems reasonable to update the logic to
write manifests out to not delete entries from the set being operated
on. Doing so allows us to avoid creating a duplicate of the set of all
committed manifests during compaction, which could also save some memory
as the temporary map wasn't being right-sized based on the the number of
committed entries. This also works because writing data either fails or
succeeds completely. That means there's no possibility of only some
entries being written out but not others, which means callers can use
the presence of an error to determine how to react (i.e. clear pending
set).
* Benchmarks for compaction
* use `getEnvVarOrSkip()` helper
* use creds from JSON env for test to actually run the test
* prefer require
* nit: relocate helpers
* fail tests if credentials cannot be decoded
The s3 storage provider had a different http transports for different cases:
- https without TLS verification: `&http.Transport{}` with default values;
- https with TLS verification: `http.DefaultTransport.Clone()`
This change uses `http.DefaultTransport` in all cases, instead of creating an
empty (zero-value) `http.Transport` for consistency.
Authored-by: aleksandr.samarin (@alexvbg)
* refactor(general): clarify comments
* refactor(general): rename `repodiag.logWriteSyncer`
This name reflects:
- the function of the struct; and
- the interface it implements (`zapcore.WriteSyncer`)
Create a basic test that requires all manifest fields to be populated
and then checks if deserialization works as expected. This is meant to
help ensure tests stay updated across changes.
Cleanups:
- use non-format variants of Log/Print with no additional args;
- fold in Fprintf call with no args into the following one;
- add missing arg placeholder in format strings;
- use require.Positive instead of Greater(..., 0);
- rename function to fillWithZeros to avoid collision with builtin clear;
- define type for context key to avoid collisions.
Use non-formatting logging functions for message without formatting.
For example, `log.Info("message")` instead of `log.Infof("message")`
Configure linter for printf-like functions
Previously, empty master keys were passed to the underlying
cryptographic primitives (HKDF, AEAD, etc.).
While this worked because the authentication mechanisms returned an
error, it's best to avoid passing empty master keys to these primitives
in the first place. This refactor avoids passing empty master keys and
enforces this via an assertion in the key derivation function.
Connecting to gRPC repository API using an IPv6 address does not
correctly configure the URI for the client's dial. Reconstructing the
parsed URL into `hostname + ":" + port` will remove the square brackets
required for IPv6 addressing, resulting in a `too many colons in
address` error.
Fix the issue by instead using the helper `net.JoinHostPort()`, which
will add square brackets for IPv6 hostnames.
Tested by running `TestServer` with `httptest.serve` flag set, forcing
the test server to listen on the IPv6 loopback:
```
cd internal/server
go test -v -run=TestServer$ --httptest.serve=[::1]:0 ./server
```
Fails without fix:
```
server_test.go:48:
Error Trace: /workspaces/kopia/internal/server/server_test.go:48
Error: Received unexpected error:
failed to exit idle mode: invalid target address ::1:45373, error info: address ::1:45373:443: too many colons in address
```
Passes with fix:
```
--- PASS: TestServer (0.81s)
PASS
```
Code movement and simplification, no functional changes.
Objectives:
- Allow callers specifying the needed key (or hash) size, instead of
hard-coding it in the registered PBK derivers. Conceptually, the caller
needs to specify the key size, since that is a requirement of the
(encryption) algorithm being used in the caller. Now, the code changes
here do not result in any functional changes since the key size is
always 32 bytes.
- Remove a global definition for the default PB key deriver to use.
Instead, each of the 3 use case sets the default value.
Changes:
- `crypto.DeriveKeyFromPassword` now takes a key size.
- Adds new constants for the key sizes at the callers.
- Removes the global `crypto.MasterKeySize` const.
- Removes the global `crypto.DefaultKeyDerivationAlgorithm` const.
- Adds const for the default derivation algorithms for each use case.
- Adds a const for the salt length in the `internal/user` package, to ensure
the same salt length is used in both hash versions.
- Unexports various functions, variables and constants in the `internal/crypto`
& `internal/user` packages.
- Renames various constants for consistency.
- Removes unused functions and symbols.
- Renames files to be consistent and better reflect the structure of the code.
- Adds a couple of tests to ensure the const values are in sync and supported.
- Fixes a couple of typos
Followups to:
- #3725
- #3770
- #3779
- #3799
- #3816
The individual commits show the code transformations to simplify the
review of the changes.
