* refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* chore: remove //nolint:gosec for os.ReadFile
At the time of this commit, the G304 rule of gosec does not include the
`os.ReadFile` function. We remove `//nolint:gosec` temporarily until
https://github.com/securego/gosec/pull/706 is merged.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* sftp: fixed performance reggression due to connection management
The previous pooling was causing serialization of all requests, which
was too slow.
This change effectively reverts connection pooling but adds
automatic reconnection + unit tests.
* sftp: fixed unwanted retry on initial connection
* Support setting AWS S3 storage class for all types of blobs
* Read .storageconfig file
* Improve loading logic
* Hide .storageconfig from ListBlobs()
On Linux, Keyring is disabled by default and `--use-keyring` must
be specified on every command-line invocation, which gets quite
annoying.
This commit allows enabling keyring from environment variable
instead.
* blob: support for custom blob store sharding
This is experimental.
The .shards file can reside in the root of any blob storage that uses
sharding (filesystem/sftp/webdav) and can specify rules for sharding.
{
"default": [3,2,1],
"overrides": [
{ "prefix": "p", "shards": [2,2] },
{ "prefix": "x", "shards": [1,1,1] }
],
"maxNonShardedLength": 2
}
With this in place we'll be later able to do resharding of the
repository to optimize get/put/list performance for both repositories
and caches.
* cli: command line tools to manipulate shards in a directory
When --`config-file` is passed as a filename without any directory
(absolute or relative) it is resolved in OS-specific
config path.
For example on macOS:
`--config-file foo.config`
resolves to:
`~/Library/Application Support/kopia/foo.config`
* content: fixed repo upgrade version
Previously upgrade would enable epoch manager and index v2 but would
not set the version of the format itself. Everything worked fine
but it would not protect from old kopia opening the repository.
* ci: added compatibility test that uses real 0.8 and current binaries
* cli: include parameters in maintenance info JSON output
* e2e: add maintenance info checks in e2e test
* cli: simple test for 'maintenance info --json' command
* testing: KOPIA_TEST_LOG_OUTPUT logs subcommand outputs
* cli: additional flags for 'blob list'
* Makefile: run all tests against epoch-based index manager
* epoch: added support for deletion watermark, which keeps track of latest maintenance which dropped index entries
* content: added deletion watermark to content manager
* maintenance: improved maintenance without safety to force rewrites
* maintenance: skip quick maintenance when epoch manager is enabled
* maintenance: do not enable quick maintenance when epoch manager is used
* testing: skip TestIndexOptimize when running against epoch manager-backed index strutures
* webdav: added --atomic-writes (defaults to false) to assume WebDAV supports atomic writes
* rclone: added --atomic-writes (defaults to true)
This allows the provider to assume rclone backend supports atomic writes (most of them do).
* cli: added 'repository validate-provider' which runs a set of tests against blob storage provider to validate it
This implements a provider tests which exercises subtle behaviors which are not always correctly implemented by providers claiming compatibility with S3, for example.
The test checks:
- not found behavior
- prefix scans
- timestamps
- write atomicity
* retry: improved error message on failure
* rclone: fixed stats reporting and awaiting for completion
* webdav: prevent panic when attempting to mkdir with empty name
* testing: run providervalidation.ValidateProvider as part of regular provider tests
* cli: print a recommendation to validate provider after repository creation
* repo: added 'enable password change' flag (defaults to true for new repositories), which prevents embedding replicas of kopia.repository in pack blobs
* cli: added 'repo change-password' which can change the password of a connected repository
* repo: nit - renamed variables and functions dealing with key derivation
* repo: fixed cache validation HMAC secret to use stored HMAC secret instead of password-derived one
* cli: added test for repo change-password
* repo: negative cases for attempting to change password in an old repository
* Update cli/command_repository_change_password.go
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
* epoch: misc fixes and logging
* blob: misc helpers
* cli: removed useless 'repository upgrade', replaced by 'repository set-parameters'
* content: implemented indexBlobManagerV1 which uses epoch manager
* cli: commands to manipulate repository epoch parameters
* cli: commands to examine epoch-based indexes
* content: added test suite that uses epoch-based index manager
* content: fixed a ton of test data races caused by sharing blobtesting.DataMap
* cli: additional tests and validation for 'repository set-params'
* testing: replaced the use of suite with our own, since suite is not parallelizable
* Multiple placeholder expansion
It's inconvenient to only be able to expand a single placeholder file
at a time. Add support to the restore command to expand any number of
shallow placeholder arguments.
* Add placeholder looping
Loop over placeholder targets to expand from within the restore
command.
* Added multiple placeholder expansion test
Added a test to exercise the new code capable of expanding multiple
placeholder arguments to the restore command.
When doing a shallow restore, small files might take up less size than
storing the DirectoryEntry metadata. Add a minimum file size flag that
where files below that size will be written directly instead being
represented with shallow placeholders. This improves on #710.
* logging: added logger wrappers for Broadcast and Prefix
* nit: moved max hash size to a named constant
* content: added internal logger
* content: replaced context-based logging with explicit Loggers
This will capture the logger.Logger associated with the context when
the repository is opened and will reuse it for all logs instead of
creating new logger for each log message.
The new logger will also write logs to the internal logger in addition
to writing to a log file/console.
* cli: allow decrypting all blobs whose names start with _
* maintenance: added logs cleanup
* cli: commands to view logs
* cli: log selected command on each write session
