* Rename UnsupportedBlobRetention struct
Rename this struct to DefaultProviderImplementation in preparation for
adding other simple "default" functionality to it.
* Add other functions to default provider
Add other simple function implementations to the default provider so
that other providers can just embed this to get basic behavior.
* Cleanup existing users of default provider
* Add default provider to remaining storage types
Add the default provider to remaining storage providers and remove
functions that are now implemented by the default provider.
* Add new blob.Storage call to see if it's readonly
Return whether the storage is readonly so higher layers in the stack can
selectively disable some functionality if needed, like compaction.
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* Store and return retention info in test storage
Add a new interface and function that allows getting retention
information during testing. This allows for more exact comparisons about
retention duration and mode in tests.
* Fixup how blobtesting retention extension works
Use the clock instead of the object's mod time so that extensions are
from the "current time." This aligns with how the S3 blob storage
functions.
* Update retention tests to use more precise checks
Where possible, use the information returned by GetRetention in tests
that deal with retention information. This allows for more precise
comparions of retention duration and mode instead of indirectly testing
duration by advancing the clock and attempting to modify blobs.
* More robust error comparisons in retention tests
Update tests for retention to use `ErrorIs` checks instead of comparing
error messages.
* Use `require.NoError` in retention tests
Minor cleanup to reduce branches in code by using `require.NoError`
instead of if-blocks and `t.Fatal`.
* Implement ability to extend retention time on S3 buckets using Object Locks
* Move object-lock extension to maintenance.Params.
* Use a default function for unsupported extensions instead of duplicating code
* Fix potential lockup during object-lock extension
* Fix race condition. Add more code coverage
* rebase to V3
* Add checks to prevent user from setting Retention Period < Full Maintenance Interval
---------
Co-authored-by: Ashlie Martinez <ashmrtnz@alcion.ai>
* feat(repository): live cache eviction for persistent lru content cache
* Update internal/cache/persistent_lru_cache.go
Co-authored-by: Ali Dowair <adowair@umich.edu>
* merge the mutex cache into list cache
---------
Co-authored-by: Shikhar Mall <small@kopia.io>
Co-authored-by: Ali Dowair <adowair@umich.edu>
From https://github.com/google/gvisor/tree/master/tools/checklocks
This will perform static verification that we're using
`sync.Mutex`, `sync.RWMutex` and `atomic` correctly to guard access
to certain fields.
This was mostly just a matter of adding annotations to indicate which
fields are guarded by which mutex.
In a handful of places the code had to be refactored to allow static
analyzer to do its job better or to not be confused by some
constructs.
In one place this actually uncovered a bug where a function was not
releasing a lock properly in an error case.
The check is part of `make lint` but can also be invoked by
`make check-locks`.
* Introduce Volume sub-interface
The Volume interface defines APIs to access a storage provider's
volume (disk) capacity, usage, etc.. It is inherited by the Storage
interface, and is at the same hierarchical level as the Reader
interface.
* Add validations for new Volume method:
Check that GetCapacity() either returns `ErrNotAVolume`, or that it
returns a Capacity struct with values that make sense.
* Implement default (passthrough) GetCapacity:
Cloud providers do not have finite volumes, and WebDAV volumes have no
notion of volume size and usage. These implementations should just
return an error (ErrNotAVolume) when their GetCapacity() is called.
* Implement GetCapacity for sftp storage: Uses the sftp.Client interface
* Implement GetCapacity for logging, readonly store
* Implement GetCapacity() for blobtesting implementations
* Implement GetCapacity() for Google Drive:
Also modifies GetDriveClient to return the entire service instead of just the Files client.
* Implemented GetCapacity() for filesystem storage:
Implemented the function in a seperate file for each OS/architecture (Unix, OpenBSD, Windows).
* feat: persisting retention options in repository blob
- plumb retention parameters through wrapped storage
- generalize aes encryption mechanism
- rewrite the retention blob on password change
- do not write retention blob when empty
* handle retention-blob not-found failures
* cli params to set retention modes on repository create
* enable versioned map mock storage with retention settings
* adding unit tests
* write format and retention blob with retention settings if available
* rename certain functions and constants specific to format blob
* delete retention cache on password-change
* fix: replace SetTime() api call with TouchBlob()
* Update repo/repository_test.go
Co-authored-by: Nick <nick@kasten.io>
* pr feedback and codecov improvements
* fix: rename retention-blob structures to generic blob-cfg
* fix: remove minio dependency on retention constants
Co-authored-by: Shikhar Mall <shikhar@kasten.io>
Co-authored-by: Nick <nick@kasten.io>
* sharded: plumbed through blob.PutOptions
* blob: removed blob.Storage.SetTime() method
This was only used for `kopia repo sync-to` and got replaced with
an equivalent blob.PutOptions.SetTime, which wehn set to non-zero time
will attempt to set the modification time on a file.
Since some providers don't support changing modification time, we
are able to emulate it using per-blob metadata (on B2, Azure and GCS),
sadly S3 is still unsupported, because it does not support returning
metadata in list results.
Also added PutOptions.GetTime, which when set to not nil, will
populate the provided variable with actual time that got assigned
to the blob.
Added tests that verify that each provider supports GetTime
and SetTime according to this spec.
* blob: additional test coverage for filesystem storage
* blob: added PutBlobAndGetMetadata() helper and used where appropriate
* fixed test failures
* pr feedback
* Update repo/blob/azure/azure_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* blobtesting: fixed object_locking_map.go
* blobtesting: removed SetTime from ObjectLockingMap
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
