* sftp: support for external SSH command and host verfication improvements
- removed custom parsing of hostnames and verification and replaced with
standard 'knownhosts' implementation.
- added option to launch external SSH command which supports
aliases, agent, etc.
NOTE, we're still not supporting any cases where password needs to be
entered on the command line, since that would be incompatible with
the UI which uses client-server model.
Fixes#500Fixes#414
* site: updated SFTP repository connection instructions
Fixes#590
* [Robustness] Add command line parameters for kopia snapshotter
Add flags for:
- no-progress
- parallel
- cache sizes
- no update check
Add an integration test to validate snapshotter expected output
against a kopia executable.
Changed file read implementation from ReadAll() to a Handle to avoid OOMing
We don't have automated tests for this but I verified this by restoring
13GB file over fuse and memory usage never exceeded 400MB.
* logging: revamped logs from content manager to be machine parseable
Logs from the content manager (except reads) are sent to separate log
file that is always free from personally-identifiable information
(e.g. no file names, just content IDs and blob IDs).
Also moved CLI logs to a subdirectory (cli-logs) and put content logs
in a parallel directory (content-logs)
Also, the log file name will now include the type of the command that
was invoked:
kopia-20200913-134157-16110-snapshot-create.log
Fixes#588
* tests: moved all logs from tests to a separate directory
The benchmarks creates 20 GB of files in different configurations
* 10 x 2 GB files
* 100 x 200 MB files
* 1000 x 20 MB files
and backs them up to a local filesystem repository measuring time,
CPU and RAM usage.
The benchmarking script uses GCP instance (n1-standard-8) with fast NVME
flash to eliminate local filesystem latency.
Current performance numbers show major improvement in latency in
0.7.0-rc1 due to splitter throughput optimization (#606).
* cli: ensure advanced commands are not accidentally used
This prints an error when a dangerous command is used without
first setting KOPIA_ADVANCED_COMMANDS=enabled environment variable.
Co-authored-by: Julio López <julio+gh@kasten.io>
* site: edited installation page and switched Download link to point at it.
Emphasized the use of package managers to download and keep Kopia up-to-date.
Added instructions for using Scoop on Windows.
Co-authored-by: Julio López
content:Allow returning deleted content in GetContent
maintenance: check deleted contents as well
maintenance: test for when a directory content is reused after deletion
testing: add support for repo open options in repotesting
* Allow passing repo options to MustReopen
* Add repotesting.Environment.MustConnectOpenAnother
* Remove kopia.config.mlock file
* snapshot create helper
* Fix content delete related and e2e tests
* object: refactored writer to detect split points before writing
This introduces new primitive that will be moved into splitters
themselves in subsequent commits. I'm doing this in small steps to
ensure we don't regress at any time.
* splitter: refactored TestSplitters test
This is use slow (byte-by-byte) and fast (nextSplitPoint) methods of
determining split points.
Note nextSplitPoint is not implemented by splitters yet, but this
verifies that the test is expecting the right thing.
* object: splitter refactoring - replaced ShouldSplit() with NextSplitPoint() everywhere, still not optimized
* splitter: added additional dimension to splitter_test
We split either in large chunks or one byte at a time to catch
the corner cases in the splitter implementation.
* splitter: optimized splitters using NextSplitPoint primitive
This improves splitter performance by about 40% (buzhash) and makes
it virtually free for FIXED splitter.
* goreleaser: added signatures to RPM binaries
Currently goreleaser does not support it, so we're overriding
signing script and signing all RPMs that it produces.
Also changed goreleaser parameters to only publish binaries
when running on linux/amd64.
* build: added automatic publishing of RPMs to a YUM repository
Also fixed RPM file names to match local conventions.
* cli: added --tls-print-server-cert flag
This prints complete server certificate that is base64 and PEM-encoded.
It is needed for Electron to securely connect to the server outside of
the browser, since there's no way to trust certificate by fingerprint.
* server: added repo/exists API
* server: added ClientOptions to create and connect API
* server: exposed current-user API
* server: API to change description of a repository
* htmlui: refactored connect/create flow
This cleaned up the code a lot and made UX more obvious.
* kopia-ui: simplified repository management UX
Removed repository configuration window which was confusing due to
the notion of 'server'.
Now KopiaUI will automatically launch 'kopia server --ui' for each
config found in the kopia config directory and shut it down every
time repository is disconnected.
See https://youtu.be/P4Ll_LR4UVM for a quick demo.
Fixes#583
* repo: refactored client-specific options (hostname,username,description,readonly) into new struct that is JSON-compatible with current config
* cli: added 'repository set-client' to configure parameters of connected repository
* cli: cleaned up 'repository status' output
* cli: simplified mount command
See https://youtu.be/1Nt_HIl-NWQ
It will always use WebDAV on Windows and FUSE on Unix. Removed
confusing options.
New usage:
$ kopia mount [--browse]
Mounts all snapshots in a temporary filesystem directory
(both Unix and Windows).
$ kopia mount <object> [--browse]
Mounts given object in a temporary filesystem directory
(both Unix and Windows).
$ kopia mount <object> z: [--browse]
Mounts given object as a given drive letter in Windows (using
temporary WebDAV mount).
$ kopia mount <object> * [--browse]
Mounts given object as a random drive letter in Windows.
$ kopia mount <object> /mount/path [--browse]
Mounts given object in given path in Unix.
<object> can be the ID of a directory 'k<hash>' or 'all'
Optional --browse automatically opens OS-native file browser.
* htmlui: added UI for mounting directories
See https://youtu.be/T-9SshVa1d8 for a quick demo.
Also replaced some UI text with icons.
* lint: windows-specific fix
The problem was that while dropping deleted entries from the index
we were looking at index-local and nor merged result.
Say index1 has content1 created at t0.
Say index2 has content1 deleted at t1 > t0.
And we're compacting and dropping index entries deleted before t2,
such that t2 > t1.
Because create and delete records are in separate index files, the
compacted file would preserve "created t0" which is wrong.
This fix changes index compaction to always look at resolved content
status.
Co-authored-by: Julio López <julio+gh@kasten.io>
Fixes#564
cli: added 'kopia policy set --ignore-cache-dirs' option to control
whether to ignore caches (global default=true)
ui: added checkbox to control 'Ignore Cache Dirs' in policy editor
ignorefs: moved ignoring cache directories to ignorefs layer
Co-authored-by: Julio López <julio+gh@kasten.io>
