* Use an AssumeRole credential provider if a role was specified
* Use minio's assumerole call
* Update assume role test
---------
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
This allows persisting and reading back the timeout as a duration.
There are be no backwards compatibility issues, since the field is
not currently set, so it is not persisted in the configuration file.
Allow the use of a client certificate when authenticating
to an Azure Blob storage provider.
Tests included.
Credit: @DeepikaDixit
Authored-by: Deepika Dixit <DeepikaDixit@users.noreply.github.com>
* Configure compressor for k and x prefixed content
Adds metadata compression setting to policy
Add support to configure compressor for k and x prefixed content
Set zstd-fastest as the default compressor for metadata in the policy
Adds support to set and show metadata compression to kopia policy commands
Adds metadata compression config to dir writer
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Pass concatenate options with ConcatenateOptions struct
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move content compression handling to caller
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move handling manifests to manifest pkg
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Correct const in server_test
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Remove unnecessary whitespace
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Disable metadata compression for < V2 format
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
---------
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* use `getEnvVarOrSkip()` helper
* use creds from JSON env for test to actually run the test
* prefer require
* nit: relocate helpers
* fail tests if credentials cannot be decoded
The s3 storage provider had a different http transports for different cases:
- https without TLS verification: `&http.Transport{}` with default values;
- https with TLS verification: `http.DefaultTransport.Clone()`
This change uses `http.DefaultTransport` in all cases, instead of creating an
empty (zero-value) `http.Transport` for consistency.
Authored-by: aleksandr.samarin (@alexvbg)
Cleanups:
- use non-format variants of Log/Print with no additional args;
- fold in Fprintf call with no args into the following one;
- add missing arg placeholder in format strings;
- use require.Positive instead of Greater(..., 0);
- rename function to fillWithZeros to avoid collision with builtin clear;
- define type for context key to avoid collisions.
Use non-formatting logging functions for message without formatting.
For example, `log.Info("message")` instead of `log.Infof("message")`
Configure linter for printf-like functions
When immutability is enabled on an Azure Blob storage account and/or container, a blob
may be protected by a policy that causes the deletion of the blob to fail.
This causes various kopia operations to fail, including the deletion of session marker
blobs and the deletion of blobs as part of kopia "maintenance" operations.
This changes allows creating an Azure-blob-native deletion marker when the blob is
protected on containers where **version-level immutability** is enabled. This
approach will likely NOT work with the many other flavors of immutability supported
by Azure Blob. That is, the approach is ONLY expected to work with storage
containers with "version-level immutability" enabled.
Added improved providervalidation logic which tests for read-after-write
property between connections. The new test was failing before the change
and is now passing for Google Drive, OneDrive and DropBox.
Change google client to accept more credentials type, including json config file for workload identity federation
Refactor tokenSourceFromCredentialsFile to remove duplicate code
Add unit tests
---------
Co-authored-by: Le Tran <le.tran@kasten.io>
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* Rename UnsupportedBlobRetention struct
Rename this struct to DefaultProviderImplementation in preparation for
adding other simple "default" functionality to it.
* Add other functions to default provider
Add other simple function implementations to the default provider so
that other providers can just embed this to get basic behavior.
* Cleanup existing users of default provider
* Add default provider to remaining storage types
Add the default provider to remaining storage providers and remove
functions that are now implemented by the default provider.
* Add new blob.Storage call to see if it's readonly
Return whether the storage is readonly so higher layers in the stack can
selectively disable some functionality if needed, like compaction.
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
This improves the performance of gather writes (p and q blobs) in
particular on high-latency links by reducing round-trips.
Tested by uploading Kopia working directory
(3 GB, files:32157 dirs:6051) to SFTP server on LAN (TrueNAS) over WiFi:
Before: 2m4s (~24 MB/s)
After: 1m13s (~41 MB/s)
* Store and return retention info in test storage
Add a new interface and function that allows getting retention
information during testing. This allows for more exact comparisons about
retention duration and mode in tests.
* Fixup how blobtesting retention extension works
Use the clock instead of the object's mod time so that extensions are
from the "current time." This aligns with how the S3 blob storage
functions.
* Update retention tests to use more precise checks
Where possible, use the information returned by GetRetention in tests
that deal with retention information. This allows for more precise
comparions of retention duration and mode instead of indirectly testing
duration by advancing the clock and attempting to modify blobs.
* More robust error comparisons in retention tests
Update tests for retention to use `ErrorIs` checks instead of comparing
error messages.
* Use `require.NoError` in retention tests
Minor cleanup to reduce branches in code by using `require.NoError`
instead of if-blocks and `t.Fatal`.
* refactor: return credentials.Value in createMinioSessionToken helper
* refactor: use minio SDK credentials package in STS test
* go mod tidy -compat=1.20 => removes dependency on AWS SDK v1 packages
* cleanup: unalias use of minio credentials package
* Implement ability to extend retention time on S3 buckets using Object Locks
* Move object-lock extension to maintenance.Params.
* Use a default function for unsupported extensions instead of duplicating code
* Fix potential lockup during object-lock extension
* Fix race condition. Add more code coverage
* rebase to V3
* Add checks to prevent user from setting Retention Period < Full Maintenance Interval
---------
Co-authored-by: Ashlie Martinez <ashmrtnz@alcion.ai>
* chore(ci): upgraded linter to 1.53.3
This flagged a bunch of unused parameters, so the PR is larger than
usual, but 99% mechanical.
* separate lint CI task
* run Lint in separate CI
We can't enable checklocks on CI yet until
https://github.com/google/gvisor/pull/8807 is merged upstream.
This was tested with private build of checklocks with this patch
applied and the results were clean.
