* upload: apply retention policy after each checkpoint
* policy: fixed applying retention policy to incomplete snapshots
- added limited retention to incomplete snapshots (keeping at least 3 snapshots and no less than 4 hours)
* nit: fixed minor data race
* upload: fixed minor data races when reporting stats
* upload: ensure we cancel estimation goroutine before upload finished
This fixes minor data race when gathering statistics.
Also avoids goroutine leaks.
* testing: don't use expensive scrypt-65536-8-1 in integration tests
* testing: use platform-specific encryption and hashing for arm and arm64 to speed up tests
* testing: manually manage log directory to be able to analyze integration test failures
* testing: snapshot_gc_test was too quick
* Makefile: renamed target building integration test binary
* object: fixed race condition between Result() and Checkpoint()
This would sometimes result in indirect objects having empty object IDs.
Fixes#648
* upload: ensure checkpoints never containt empty object IDs.
* testing: reduce armhf test weight
* restore: improved user experience
* 'snapshot restore' is now the same as 'restore' and both will
support restoring by manifest ID, root ID or root ID + subdirectory
* added support for restoring individual files
* implemented PR feedback and refactored object ID parsing
Moving helpers inside the snapshot/ package helped clean up the code
a lot.
* server: repro for zero-sized snapshot bug
As described in https://kopia.discourse.group/t/kopia-0-7-0-not-backing-up-any-files-repro-needed/136/5
* server: fixed zero-sized snapshots after repository is connected via API
The root cause was that source manager was inheriting HTTP call context
which was immediately closed after the 'connect' RPC returned thus
silently killing all uploads.
* Add Blake3 hash
Uses github.com/zeebo/blake3 optimized implementation
Hash info at https://github.com/BLAKE3-team/BLAKE3
Micro benchmarks show ~2X speedup for hashing+encryption.
- Caveat: not fully controlled environment, etc.
* go mod tidy
* Update .gitattributes for Go files
- Mark go source files as text, with LF line ending.
- Set git diff language mode for `*.go` files to `golang`
* Add .gitattributes entry for Makefiles
* sftp: support for external SSH command and host verfication improvements
- removed custom parsing of hostnames and verification and replaced with
standard 'knownhosts' implementation.
- added option to launch external SSH command which supports
aliases, agent, etc.
NOTE, we're still not supporting any cases where password needs to be
entered on the command line, since that would be incompatible with
the UI which uses client-server model.
Fixes#500Fixes#414
* site: updated SFTP repository connection instructions
Fixes#590
* [Robustness] Add command line parameters for kopia snapshotter
Add flags for:
- no-progress
- parallel
- cache sizes
- no update check
Add an integration test to validate snapshotter expected output
against a kopia executable.
Changed file read implementation from ReadAll() to a Handle to avoid OOMing
We don't have automated tests for this but I verified this by restoring
13GB file over fuse and memory usage never exceeded 400MB.
* logging: revamped logs from content manager to be machine parseable
Logs from the content manager (except reads) are sent to separate log
file that is always free from personally-identifiable information
(e.g. no file names, just content IDs and blob IDs).
Also moved CLI logs to a subdirectory (cli-logs) and put content logs
in a parallel directory (content-logs)
Also, the log file name will now include the type of the command that
was invoked:
kopia-20200913-134157-16110-snapshot-create.log
Fixes#588
* tests: moved all logs from tests to a separate directory
The benchmarks creates 20 GB of files in different configurations
* 10 x 2 GB files
* 100 x 200 MB files
* 1000 x 20 MB files
and backs them up to a local filesystem repository measuring time,
CPU and RAM usage.
The benchmarking script uses GCP instance (n1-standard-8) with fast NVME
flash to eliminate local filesystem latency.
Current performance numbers show major improvement in latency in
0.7.0-rc1 due to splitter throughput optimization (#606).
* cli: ensure advanced commands are not accidentally used
This prints an error when a dangerous command is used without
first setting KOPIA_ADVANCED_COMMANDS=enabled environment variable.
Co-authored-by: Julio López <julio+gh@kasten.io>
* site: edited installation page and switched Download link to point at it.
Emphasized the use of package managers to download and keep Kopia up-to-date.
Added instructions for using Scoop on Windows.
Co-authored-by: Julio López
content:Allow returning deleted content in GetContent
maintenance: check deleted contents as well
maintenance: test for when a directory content is reused after deletion
testing: add support for repo open options in repotesting
* Allow passing repo options to MustReopen
* Add repotesting.Environment.MustConnectOpenAnother
* Remove kopia.config.mlock file
* snapshot create helper
* Fix content delete related and e2e tests
* object: refactored writer to detect split points before writing
This introduces new primitive that will be moved into splitters
themselves in subsequent commits. I'm doing this in small steps to
ensure we don't regress at any time.
* splitter: refactored TestSplitters test
This is use slow (byte-by-byte) and fast (nextSplitPoint) methods of
determining split points.
Note nextSplitPoint is not implemented by splitters yet, but this
verifies that the test is expecting the right thing.
* object: splitter refactoring - replaced ShouldSplit() with NextSplitPoint() everywhere, still not optimized
* splitter: added additional dimension to splitter_test
We split either in large chunks or one byte at a time to catch
the corner cases in the splitter implementation.
* splitter: optimized splitters using NextSplitPoint primitive
This improves splitter performance by about 40% (buzhash) and makes
it virtually free for FIXED splitter.
