The issue in #1439 was caused by goroutine context being associated
with the HTTP request so it became canceled soon after the request was
over, thus the goroutine to run maintenance never ran.
Fixed by adding ctxutil.Detach()
Also fixed logging by passing top-level contexts to requests
and added --log-server-requests flag to `server start` which enables
request logging.
This was introduced by a refactoring in #1361 - unlike
ioutil.ReadDir() which internally handles list/delete race and always
returns os.FileInfo, Info() on os.DirEntry can fail if a file
is deleted right after listing it.
Fixes#1486
* logging: added Logger.Debugw(message, key1, value1, ..., keyN, valueN)
This is based on ZAP and allows structural logs to be emitted.
* cli: added --json-log-console and --json-log-file flags
* logging: updated storage logging wrapper to use structural logging
* pr feedback
The dual time measurement is described in
https://go.googlesource.com/proposal/+/master/design/12914-monotonic.md
The fix is to discard hidden monotonic time component of time.Time
by converting to unix time and back.
Reviewed usage of clock.Now() and replaced with timetrack.StartTimer()
when measuring time.
The problem in #1402 was that passage of time was measured using
the monotonic time and not wall clock time. When the computer goes
to sleep, monotonic time is still monotonic while wall clock time makes
a leap when the computer wakes up. This is the behavior that
epoch manager (and most other compontents in Kopia) rely upon.
Fixes#1402
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
In particular this applies compression based on destination repository
policies.
By default ignore rules are disabled during migration to preserve all
files, but that can be optionally enabled using '--apply-ignore-rules'.
Fixes#1429
This was caused by additional resolution of path names only done in UI,
which caused \\hostname\share to be treated as relative and resolved
against the home directory.
Fixes#1385Fixes#1362
* refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* chore: remove //nolint:gosec for os.ReadFile
At the time of this commit, the G304 rule of gosec does not include the
`os.ReadFile` function. We remove `//nolint:gosec` temporarily until
https://github.com/securego/gosec/pull/706 is merged.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* sftp: fixed performance reggression due to connection management
The previous pooling was causing serialization of all requests, which
was too slow.
This change effectively reverts connection pooling but adds
automatic reconnection + unit tests.
* sftp: fixed unwanted retry on initial connection
* Support setting AWS S3 storage class for all types of blobs
* Read .storageconfig file
* Improve loading logic
* Hide .storageconfig from ListBlobs()
On Linux, Keyring is disabled by default and `--use-keyring` must
be specified on every command-line invocation, which gets quite
annoying.
This commit allows enabling keyring from environment variable
instead.
* blob: support for custom blob store sharding
This is experimental.
The .shards file can reside in the root of any blob storage that uses
sharding (filesystem/sftp/webdav) and can specify rules for sharding.
{
"default": [3,2,1],
"overrides": [
{ "prefix": "p", "shards": [2,2] },
{ "prefix": "x", "shards": [1,1,1] }
],
"maxNonShardedLength": 2
}
With this in place we'll be later able to do resharding of the
repository to optimize get/put/list performance for both repositories
and caches.
* cli: command line tools to manipulate shards in a directory
When --`config-file` is passed as a filename without any directory
(absolute or relative) it is resolved in OS-specific
config path.
For example on macOS:
`--config-file foo.config`
resolves to:
`~/Library/Application Support/kopia/foo.config`
* content: fixed repo upgrade version
Previously upgrade would enable epoch manager and index v2 but would
not set the version of the format itself. Everything worked fine
but it would not protect from old kopia opening the repository.
* ci: added compatibility test that uses real 0.8 and current binaries
* cli: include parameters in maintenance info JSON output
* e2e: add maintenance info checks in e2e test
* cli: simple test for 'maintenance info --json' command
* testing: KOPIA_TEST_LOG_OUTPUT logs subcommand outputs
* cli: additional flags for 'blob list'
* Makefile: run all tests against epoch-based index manager
* epoch: added support for deletion watermark, which keeps track of latest maintenance which dropped index entries
* content: added deletion watermark to content manager
* maintenance: improved maintenance without safety to force rewrites
* maintenance: skip quick maintenance when epoch manager is enabled
* maintenance: do not enable quick maintenance when epoch manager is used
* testing: skip TestIndexOptimize when running against epoch manager-backed index strutures
* webdav: added --atomic-writes (defaults to false) to assume WebDAV supports atomic writes
* rclone: added --atomic-writes (defaults to true)
This allows the provider to assume rclone backend supports atomic writes (most of them do).
* cli: added 'repository validate-provider' which runs a set of tests against blob storage provider to validate it
This implements a provider tests which exercises subtle behaviors which are not always correctly implemented by providers claiming compatibility with S3, for example.
The test checks:
- not found behavior
- prefix scans
- timestamps
- write atomicity
* retry: improved error message on failure
* rclone: fixed stats reporting and awaiting for completion
* webdav: prevent panic when attempting to mkdir with empty name
* testing: run providervalidation.ValidateProvider as part of regular provider tests
* cli: print a recommendation to validate provider after repository creation
* repo: added 'enable password change' flag (defaults to true for new repositories), which prevents embedding replicas of kopia.repository in pack blobs
* cli: added 'repo change-password' which can change the password of a connected repository
* repo: nit - renamed variables and functions dealing with key derivation
* repo: fixed cache validation HMAC secret to use stored HMAC secret instead of password-derived one
* cli: added test for repo change-password
* repo: negative cases for attempting to change password in an old repository
* Update cli/command_repository_change_password.go
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
Jarek Kowalski