This was caused by additional resolution of path names only done in UI,
which caused \\hostname\share to be treated as relative and resolved
against the home directory.
Fixes#1385Fixes#1362
* refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* chore: remove //nolint:gosec for os.ReadFile
At the time of this commit, the G304 rule of gosec does not include the
`os.ReadFile` function. We remove `//nolint:gosec` temporarily until
https://github.com/securego/gosec/pull/706 is merged.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* sftp: fixed performance reggression due to connection management
The previous pooling was causing serialization of all requests, which
was too slow.
This change effectively reverts connection pooling but adds
automatic reconnection + unit tests.
* sftp: fixed unwanted retry on initial connection
Directories containing symlinks could not be mounted on Windows and
were failing with 'invalid parameter'.
This change skips those entries and emits a warning to the console.
Symlinks appear to be unsupported in WebDAV at the protocol level
and servers usually follow symlinks server-side, which is not
feasible for Kopia. In the future we may be able to do more here, but
for now it's better to not show symlinks than not show anything.
Huge thanks to Joe Occhipinti for his help getting to the bottom of this
issue.
* upload: do not launch error group when doing non-parallel upload
* encryption: refactored code to reduce slice allocation, this uncovered a minor subtle bug in index recovery due to manual memory management
* content: fixed use-after-free bug during index recovery that started failing test after encryption memory management improvements
* gather: tweaked retention and chunk sizes
* gather: added typicalContiguousAllocator with 8MB chunks for default configuration
* content: fixed repo upgrade version
Previously upgrade would enable epoch manager and index v2 but would
not set the version of the format itself. Everything worked fine
but it would not protect from old kopia opening the repository.
* ci: added compatibility test that uses real 0.8 and current binaries
* testing: KOPIA_TEST_LOG_OUTPUT logs subcommand outputs
* cli: additional flags for 'blob list'
* Makefile: run all tests against epoch-based index manager
* epoch: added support for deletion watermark, which keeps track of latest maintenance which dropped index entries
* content: added deletion watermark to content manager
* maintenance: improved maintenance without safety to force rewrites
* maintenance: skip quick maintenance when epoch manager is enabled
* maintenance: do not enable quick maintenance when epoch manager is used
* testing: skip TestIndexOptimize when running against epoch manager-backed index strutures
* cli: added 'repository validate-provider' which runs a set of tests against blob storage provider to validate it
This implements a provider tests which exercises subtle behaviors which are not always correctly implemented by providers claiming compatibility with S3, for example.
The test checks:
- not found behavior
- prefix scans
- timestamps
- write atomicity
* retry: improved error message on failure
* rclone: fixed stats reporting and awaiting for completion
* webdav: prevent panic when attempting to mkdir with empty name
* testing: run providervalidation.ValidateProvider as part of regular provider tests
* cli: print a recommendation to validate provider after repository creation
* testing: removed testutil.Retry because all providers now have internal retries
* testing: simplified and unified cleanup for all cloud providers using shared buckets
* repo: added 'enable password change' flag (defaults to true for new repositories), which prevents embedding replicas of kopia.repository in pack blobs
* cli: added 'repo change-password' which can change the password of a connected repository
* repo: nit - renamed variables and functions dealing with key derivation
* repo: fixed cache validation HMAC secret to use stored HMAC secret instead of password-derived one
* cli: added test for repo change-password
* repo: negative cases for attempting to change password in an old repository
* Update cli/command_repository_change_password.go
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
* repo: big performance improvement for WriteContent with repo server
When re-uploading previously snapshotted directory we fetch directory
content `k<hash>` and very frequently end up writing the exact same
content. By caching last N content IDs we can avoid costly round-trip
to the server since we know that content ID was present in the session.
Also added small number of asynchronous writes, which also helps with
upload performance. Background writes are awaited before Flush().
Performance when snapshotting lots of small files (source code):
31.9 GB files:471205 dirs:75817, warm cache
Before: 260s
After: 55s (4-5x faster)
* fixed tests
* epoch: misc fixes and logging
* blob: misc helpers
* cli: removed useless 'repository upgrade', replaced by 'repository set-parameters'
* content: implemented indexBlobManagerV1 which uses epoch manager
* cli: commands to manipulate repository epoch parameters
* cli: commands to examine epoch-based indexes
* content: added test suite that uses epoch-based index manager
* content: fixed a ton of test data races caused by sharing blobtesting.DataMap
* cli: additional tests and validation for 'repository set-params'
* testing: replaced the use of suite with our own, since suite is not parallelizable
* refactor: extracted complete blob set functions to separate package
* completeset: added more functions
Also treat malformed blob IDs as their own sets for backwards compat.
* testing: refactored logs directory management
* content: fixed index mutex to be shared across all write sessions
added mutex protection during writecontent/refresh race
* testing: upload log artifacts
* content: bump revision number after index has been added
This fixes a bug where manifest manager in another session for
the same open repository may not see a content added, because they
will prematurely cache the incomplete set of contents.
This took 2 weeks to find.
* manifest: improved log output, fixed unnecessary mutex release
* testing: rewrote stress test to be model-based and more precise
