- added pooled splitters and ability to reset them without having to recreate
- added support for caller-provided compressor output to be able to pool it
- added pooling of compressor instances, since those are costly
non-optimized (0.5.0)
0. BLAKE2B-256-128 AES256-GCM-HMAC-SHA256 644.9 MiB / second
before this change:
0. BLAKE2B-256-128 AES256-GCM-HMAC-SHA256 655.9 MiB / second
after (this change):
0. BLAKE2B-256-128 AES256-GCM-HMAC-SHA256 781.5 MiB / second
* performance: plumbed through output buffer to encryption and hashing, so that the caller can pre-allocate/reuse it
* testing: fixed how we do comparison of byte slices to account for possible nils, which can be returned from encryption
* repo: added some initial metrics using OpenCensus
* cli: added flags to expose Prometheus metrics on a local endpoint
`--metrics-listen-addr=localhost:X` exposes prometheus metrics on
http://localhost:X/metrics
Also, kopia server will automatically expose /metrics endpoint on the
same port it runs as, without authentication.
- cleaned up migration progress output
- fixed migration idempotency
- added migration of policies
- renamed --parallelism to --parallel
- improved e2e test
- do not prompt for password to source repository if persisted
Now you can quickly verify that all contents are correctly backed
by existing blob without downloading much.
You can still use '--full' to cause full download and decryption.
- did not work on windows due to use of filepath which uses backslash
instead of slash
- added support for embedding SFTP key
- fixed UI controls
- misc fixes for KopiaUI
- added progress reporting
New ciphers are using authenticated encryption with associated data
(AEAD) and per-content key derived using HMAC-SHA256:
* AES256-GCM-HMAC-SHA256
* CHACHA20-POLY1305-HMAC-SHA256
They support content IDs of arbitrary length and are quite fast:
On my 2019 MBP:
- BLAKE2B-256 + AES256-GCM-HMAC-SHA256 - 648.7 MiB / second
- BLAKE2B-256 + CHACHA20-POLY1305-HMAC-SHA256 - 597.1 MiB / second
- HMAC-SHA256 + AES256-GCM-HMAC-SHA256 351 MiB / second
- HMAC-SHA256 + CHACHA20-POLY1305-HMAC-SHA256 316.2 MiB / second
Previous ciphers had several subtle issues:
* SALSA20 encryption, used weak nonce (64 bit prefix of content ID),
which means that for any two contents, whose IDs that have the same
64-bit prefix, their plaintext can be decoded from the ciphertext
alone.
* AES-{128,192,256}-CTR were not authenticated, so we were
required to hash plaintext after decryption to validate. This is not
recommended due to possibility of subtle timing attacks if an attacker
controls the ciphertext.
* SALSA20-HMAC was only validating checksum and not that the ciphertext
was for the correct content ID.
New repositories cannot be created using deprecated ciphers, but they
will still be supported for existing repositories, until at least 0.6.0.
The users are encouraged to migrate to one of new ciphers when 0.5.0 is
out.
The hostname/username are now persisted when connecting to repository
in a local config file.
This prevents weird behavior changes when hostname is suddenly changed,
such as when moving between networks.
repo.Repository will now expose Hostname/Username properties which
are always guarnateed to be set, and are used throughout.
Removed --hostname/--username overrides when taking snapshot et.al.
This is mostly mechanical and changes how loggers are instantiated.
Logger is now associated with a context, passed around all methods,
(most methods had ctx, but had to add it in a few missing places).
By default Kopia does not produce any logs, but it can be overridden,
either locally for a nested context, by calling
ctx = logging.WithLogger(ctx, newLoggerFunc)
To override logs globally, call logging.SetDefaultLogger(newLoggerFunc)
This refactoring allowed removing dependency from Kopia repo
and go-logging library (the CLI still uses it, though).
It is now also possible to have all test methods emit logs using
t.Logf() so that they show up in failure reports, which should make
debugging of test failures suck less.
Percentage based on last-known snapshot size
* server: exposed last completed snapshot size in the API
* cli: added support for progress indicator (percentage based on last-known snapshot size)
* htmlui: added progress indicator in the UI (percentage based on last-known snapshot size)
/api/v1/repo/create
/api/v1/repo/connect
/api/v1/repo/disconnect
Refactored server code and fixed a number of outstanding robustness
issues. Tweaked the API responses a bit to make more sense when consumed
by the UI.
Fixes print for error handling policy. Previously was printing
pointer address, now dereference pointer to print boolean itself.
Nil check in case the caller passes a policy with nil. In that
case print "false" which is the behavior in uploader for nil
pointer.
* Ignore read errors based on policy settings
Added an error handling policy section. Can independently control error handling for directory and file read errors, toggle-able from the `policy set` command to either "true", "false", or "inherit". If any read error is hit, the error handling will check the effective policy on whether to ignore it or not. Currently there is no differentiation between read error types, though in the future we may want to add the `errors.Is(err, os.ErrPermission)` conditional.
Fix was implemented such that the policy ignores read errors ONLY on child entries of the source. So a snapshot will still fail if the source root directory itself can't be read, but you can ignore the error if a file or a subdirectory in the snapshot source root can't be read. I did this to address some otherwise strange behavior where you would successfully snapshot (because you ignored the error), but couldn't restore that snapshot because nothing really happened during the operation.
Those will make it possible to securely host 'kopia server' embedded
in a desktop app that runs in the background and can access UI.
- added support for using and generating TLS certificates
- added /api/v1/shutdown API to remotely trigger server shutdown
- added support for automatically shutting down server if no requests
arrive in certain amount of time
- added support for generating and printing random password to STDERR
TLS supports 3 modes:
1. serve TLS using externally-provided cert/key PEM files
2. generate & write PEM files, then serve TLS using them
3. generate and use emphemeral cert/key (prints SHA256 fingerprint)
```
make lint
kopia/tools/.tools/bin/golangci-lint --deadline 180s run | tee .linterr.txt
cli/command_snapshot_delete.go:5: File is not `goimports`-ed with -local github.com/kopia/kopia (goimports)
tests/end_to_end_test/end_to_end_test.go:303: Function 'TestSnapshotDelete' is too long (187 > 100) (funlen)
func TestSnapshotDelete(t *testing.T) {
tests/end_to_end_test/end_to_end_test.go:306:2: only one cuddle assignment allowed before range statement (wsl)
for _, tc := range []struct {
^
tests/end_to_end_test/end_to_end_test.go:517:4: only one cuddle assignment allowed before if statement (wsl)
if expectDeleteSucceeds {
^
tests/end_to_end_test/end_to_end_test.go:537:2: assignments should only be cuddled with other assignments (wsl)
line := lines[0]
^
tests/end_to_end_test/end_to_end_test.go:542:2: only one cuddle assignment allowed before if statement (wsl)
if typeVal != "policy" {
^
tests/end_to_end_test/end_to_end_test.go:558:2: assignments should only be cuddled with other assignments (wsl)
restoreDir := filepath.Join(e.dataDir, "restored")
^
tests/end_to_end_test/end_to_end_test.go:568:2: if statements should only be cuddled with assignments (wsl)
if got, want := len(si[0].snapshots), 1; got != want {
^
tests/end_to_end_test/end_to_end_test.go:571:2: assignments should only be cuddled with other assignments (wsl)
snapID := si[0].snapshots[0].snapshotID
^
tests/end_to_end_test/end_to_end_test.go:604:2: if statements should only be cuddled with assignments (wsl)
if len(fileInfo) != 0 {
^
cli/command_snapshot_delete.go:25:2: only one cuddle assignment allowed before if statement (wsl)
if err != nil {
^
cli/command_snapshot_delete.go:36:3: if statements should only be cuddled with assignments (wsl)
if labels["username"] != getUserName() {
^
```
Snapshot restore will take a snapshot ID and restore the
associated snapshot to the target path.
- Looks up the manifest with the snapshot ID
- Gets the snapshot root entry
- Copies the snapshot from the root entry to the target path
Because it uses the parent manifest with the copied permissions,
the restored directory will have the permissions of the original
source directory.
Implemented snapshot delete command. Behaves similarly to manifest rm, but with extra verification steps.
- Checks that the referenced manifest is of type "snapshot"
- Checks that the ID points to a snapshot, checks that the host name, user name, and path provided by flag or defaults match the source of the snapshot ID. Command will fail if they do not match, except if given --unsafe-ignore-source, which will bypass the associated safety requirement and delete anyway.
Added end to end tests for input combinations, restore in conjunction with delete, and trying to snapshot delete a manifest by ID of a non-snapshot manifest.
