On Linux, Keyring is disabled by default and `--use-keyring` must
be specified on every command-line invocation, which gets quite
annoying.
This commit allows enabling keyring from environment variable
instead.
* blob: support for custom blob store sharding
This is experimental.
The .shards file can reside in the root of any blob storage that uses
sharding (filesystem/sftp/webdav) and can specify rules for sharding.
{
"default": [3,2,1],
"overrides": [
{ "prefix": "p", "shards": [2,2] },
{ "prefix": "x", "shards": [1,1,1] }
],
"maxNonShardedLength": 2
}
With this in place we'll be later able to do resharding of the
repository to optimize get/put/list performance for both repositories
and caches.
* cli: command line tools to manipulate shards in a directory
When --`config-file` is passed as a filename without any directory
(absolute or relative) it is resolved in OS-specific
config path.
For example on macOS:
`--config-file foo.config`
resolves to:
`~/Library/Application Support/kopia/foo.config`
* content: fixed repo upgrade version
Previously upgrade would enable epoch manager and index v2 but would
not set the version of the format itself. Everything worked fine
but it would not protect from old kopia opening the repository.
* ci: added compatibility test that uses real 0.8 and current binaries
* cli: include parameters in maintenance info JSON output
* e2e: add maintenance info checks in e2e test
* cli: simple test for 'maintenance info --json' command
* testing: KOPIA_TEST_LOG_OUTPUT logs subcommand outputs
* cli: additional flags for 'blob list'
* Makefile: run all tests against epoch-based index manager
* epoch: added support for deletion watermark, which keeps track of latest maintenance which dropped index entries
* content: added deletion watermark to content manager
* maintenance: improved maintenance without safety to force rewrites
* maintenance: skip quick maintenance when epoch manager is enabled
* maintenance: do not enable quick maintenance when epoch manager is used
* testing: skip TestIndexOptimize when running against epoch manager-backed index strutures
* webdav: added --atomic-writes (defaults to false) to assume WebDAV supports atomic writes
* rclone: added --atomic-writes (defaults to true)
This allows the provider to assume rclone backend supports atomic writes (most of them do).
* cli: added 'repository validate-provider' which runs a set of tests against blob storage provider to validate it
This implements a provider tests which exercises subtle behaviors which are not always correctly implemented by providers claiming compatibility with S3, for example.
The test checks:
- not found behavior
- prefix scans
- timestamps
- write atomicity
* retry: improved error message on failure
* rclone: fixed stats reporting and awaiting for completion
* webdav: prevent panic when attempting to mkdir with empty name
* testing: run providervalidation.ValidateProvider as part of regular provider tests
* cli: print a recommendation to validate provider after repository creation
* repo: added 'enable password change' flag (defaults to true for new repositories), which prevents embedding replicas of kopia.repository in pack blobs
* cli: added 'repo change-password' which can change the password of a connected repository
* repo: nit - renamed variables and functions dealing with key derivation
* repo: fixed cache validation HMAC secret to use stored HMAC secret instead of password-derived one
* cli: added test for repo change-password
* repo: negative cases for attempting to change password in an old repository
* Update cli/command_repository_change_password.go
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
* epoch: misc fixes and logging
* blob: misc helpers
* cli: removed useless 'repository upgrade', replaced by 'repository set-parameters'
* content: implemented indexBlobManagerV1 which uses epoch manager
* cli: commands to manipulate repository epoch parameters
* cli: commands to examine epoch-based indexes
* content: added test suite that uses epoch-based index manager
* content: fixed a ton of test data races caused by sharing blobtesting.DataMap
* cli: additional tests and validation for 'repository set-params'
* testing: replaced the use of suite with our own, since suite is not parallelizable
* Multiple placeholder expansion
It's inconvenient to only be able to expand a single placeholder file
at a time. Add support to the restore command to expand any number of
shallow placeholder arguments.
* Add placeholder looping
Loop over placeholder targets to expand from within the restore
command.
* Added multiple placeholder expansion test
Added a test to exercise the new code capable of expanding multiple
placeholder arguments to the restore command.
When doing a shallow restore, small files might take up less size than
storing the DirectoryEntry metadata. Add a minimum file size flag that
where files below that size will be written directly instead being
represented with shallow placeholders. This improves on #710.
* logging: added logger wrappers for Broadcast and Prefix
* nit: moved max hash size to a named constant
* content: added internal logger
* content: replaced context-based logging with explicit Loggers
This will capture the logger.Logger associated with the context when
the repository is opened and will reuse it for all logs instead of
creating new logger for each log message.
The new logger will also write logs to the internal logger in addition
to writing to a log file/console.
* cli: allow decrypting all blobs whose names start with _
* maintenance: added logs cleanup
* cli: commands to view logs
* cli: log selected command on each write session
* cli: added a flag to create repository with v2 index features
* content: plumb through compression.ID parameter to content.Manager.WriteContent()
* content: expose content.Manager.SupportsContentCompression
This allows object manager to decide whether to create compressed object
or let the content manager do it.
* object: if compression is requested and the repo supports it, pass compression ID to the content manager
* cli: show compression status in 'repository status'
* cli: output compression information in 'content list' and 'content stats'
* content: compression and decompression support
* content: unit tests for compression
* object: compression tests
* testing: added integration tests against v2 index
* testing: run all e2e tests with and without content-level compression
* htmlui: added UI for specifying index format on creation
* cli: additional tests for 'content ls' and 'content stats'
* applied pr suggestions
* cli: Added --max-examples-per-bucket flag to 'kopia snapshot estimate'
Added and cleaned up a bunch of unit tests.
Fixes#1054
* cli: misc tests to increase code coverage of the cli package
* ci: move code coverage run into separate GH job
* compression: add mapping from HeaderID to Name
* compression: minor refactoring
* compression: added 'lz4' based on 'github.com/pierrec/lz4'
* compression: added 'deflate' support
* cli: 'kopia benchmark compression' improvements
