* trivial: move CachingOptions out of content.Manager, where it's not needed
* trivial: removed newManagerWithOptions which was the same as NewManager
also moved one-time initialization to newReadManager()
* linter: upgraded to 1.33, disabled some linters
* lint: fixed 'errorlint' errors
This ensures that all error comparisons use errors.Is() or errors.As().
We will be wrapping more errors going forward so it's important that
error checks are not strict everywhere.
Verified that there are no exceptions for errorlint linter which
guarantees that.
* lint: fixed or suppressed wrapcheck errors
* lint: nolintlint and misc cleanups
Co-authored-by: Julio López <julio+gh@kasten.io>
* restore: improved user experience
* 'snapshot restore' is now the same as 'restore' and both will
support restoring by manifest ID, root ID or root ID + subdirectory
* added support for restoring individual files
* implemented PR feedback and refactored object ID parsing
Moving helpers inside the snapshot/ package helped clean up the code
a lot.
* server: repro for zero-sized snapshot bug
As described in https://kopia.discourse.group/t/kopia-0-7-0-not-backing-up-any-files-repro-needed/136/5
* server: fixed zero-sized snapshots after repository is connected via API
The root cause was that source manager was inheriting HTTP call context
which was immediately closed after the 'connect' RPC returned thus
silently killing all uploads.
* cli: added --tls-print-server-cert flag
This prints complete server certificate that is base64 and PEM-encoded.
It is needed for Electron to securely connect to the server outside of
the browser, since there's no way to trust certificate by fingerprint.
* server: added repo/exists API
* server: added ClientOptions to create and connect API
* server: exposed current-user API
* server: API to change description of a repository
* htmlui: refactored connect/create flow
This cleaned up the code a lot and made UX more obvious.
* kopia-ui: simplified repository management UX
Removed repository configuration window which was confusing due to
the notion of 'server'.
Now KopiaUI will automatically launch 'kopia server --ui' for each
config found in the kopia config directory and shut it down every
time repository is disconnected.
See https://youtu.be/P4Ll_LR4UVM for a quick demo.
Fixes#583
* repo: refactored client-specific options (hostname,username,description,readonly) into new struct that is JSON-compatible with current config
* cli: added 'repository set-client' to configure parameters of connected repository
* cli: cleaned up 'repository status' output
* cli: simplified mount command
See https://youtu.be/1Nt_HIl-NWQ
It will always use WebDAV on Windows and FUSE on Unix. Removed
confusing options.
New usage:
$ kopia mount [--browse]
Mounts all snapshots in a temporary filesystem directory
(both Unix and Windows).
$ kopia mount <object> [--browse]
Mounts given object in a temporary filesystem directory
(both Unix and Windows).
$ kopia mount <object> z: [--browse]
Mounts given object as a given drive letter in Windows (using
temporary WebDAV mount).
$ kopia mount <object> * [--browse]
Mounts given object as a random drive letter in Windows.
$ kopia mount <object> /mount/path [--browse]
Mounts given object in given path in Unix.
<object> can be the ID of a directory 'k<hash>' or 'all'
Optional --browse automatically opens OS-native file browser.
* htmlui: added UI for mounting directories
See https://youtu.be/T-9SshVa1d8 for a quick demo.
Also replaced some UI text with icons.
* lint: windows-specific fix
Globally replaced all use of time with internal 'clock' package
which provides indirection to time.Now()
Added support for faking clock in Kopia via KOPIA_FAKE_CLOCK_ENDPOINT
logfile: squelch annoying log message
testenv: added faketimeserver which serves time over HTTP
testing: added endurance test which tests kopia over long time scale
This creates kopia repository and simulates usage of Kopia over multiple
months (using accelerated fake time) to trigger effects that are only
visible after long time passage (maintenance, compactions, expirations).
The test is not used part of any test suite yet but will run in
post-submit mode only, preferably 24/7.
testing: refactored internal/clock to only support injection when
'testing' build tag is present
* kopia-ui: added ability to connect to kopia server
* kopia-ui: update status page to show some data for repositories connected to API server
* kopia-ui: hide user@host selection dropdown for kopia server repositories
Added test that verifies that when client performs Flush (which happens
at the end of each snapshot and when repository is closed), the
server writes new blobs to the storage.
Fixes#464
instead moved to run as part of maintenance ('kopia maintenance run')
added 'kopia maintenance run --force' flag which runs maintenance even
if not owned
Support for remote content repository where all contents and
manifests are fetched over HTTP(S) instead of locally
manipulating blob storage
* server: implement content and manifest access APIs
* apiclient: moved Kopia API client to separate package
* content: exposed content.ValidatePrefix()
* manifest: added JSON serialization attributes to EntryMetadata
* repo: changed repo.Open() to return Repository instead of *DirectRepository
* repo: added apiServerRepository
* cli: added 'kopia repository connect server'
This sets up repository connection via the API server instead of
directly-manipulated storage.
* server: add support for specifying a list of usernames/password via --htpasswd-file
* tests: added API server repository E2E test
* server: only return manifests (policies and snapshots) belonging to authenticated user
* maintenance: encrypt maintenance schedule block
* maintenance: created snapshotmaintenance package that wraps maintenance and performs snapshot GC + regular maintenance in one shot, used in CLI and server
* PR feedback.
Maintenance: support for automatic GC
Moved maintenance algorithms from 'cli' to 'repo/maintenance' package
Added support for CLI commands:
kopia gc - performs quick maintenance
kopia gc --full- perform full maintenance
Full maintenance performs snapshot gc, but it's not safe to do this automatically possibly in parallel to snapshots being taken. This will be addressed ~0.7 timeframe.
* This is 99% mechanical:
Extracted repo.Repository interface that only exposes high-level object and manifest management methods, but not blob nor content management.
Renamed old *repo.Repository to *repo.DirectRepository
Reviewed codebase to only depend on repo.Repository as much as possible, but added way for low-level CLI commands to use DirectRepository.
* PR fixes
New ciphers are using authenticated encryption with associated data
(AEAD) and per-content key derived using HMAC-SHA256:
* AES256-GCM-HMAC-SHA256
* CHACHA20-POLY1305-HMAC-SHA256
They support content IDs of arbitrary length and are quite fast:
On my 2019 MBP:
- BLAKE2B-256 + AES256-GCM-HMAC-SHA256 - 648.7 MiB / second
- BLAKE2B-256 + CHACHA20-POLY1305-HMAC-SHA256 - 597.1 MiB / second
- HMAC-SHA256 + AES256-GCM-HMAC-SHA256 351 MiB / second
- HMAC-SHA256 + CHACHA20-POLY1305-HMAC-SHA256 316.2 MiB / second
Previous ciphers had several subtle issues:
* SALSA20 encryption, used weak nonce (64 bit prefix of content ID),
which means that for any two contents, whose IDs that have the same
64-bit prefix, their plaintext can be decoded from the ciphertext
alone.
* AES-{128,192,256}-CTR were not authenticated, so we were
required to hash plaintext after decryption to validate. This is not
recommended due to possibility of subtle timing attacks if an attacker
controls the ciphertext.
* SALSA20-HMAC was only validating checksum and not that the ciphertext
was for the correct content ID.
New repositories cannot be created using deprecated ciphers, but they
will still be supported for existing repositories, until at least 0.6.0.
The users are encouraged to migrate to one of new ciphers when 0.5.0 is
out.
The hostname/username are now persisted when connecting to repository
in a local config file.
This prevents weird behavior changes when hostname is suddenly changed,
such as when moving between networks.
repo.Repository will now expose Hostname/Username properties which
are always guarnateed to be set, and are used throughout.
Removed --hostname/--username overrides when taking snapshot et.al.
This is mostly mechanical and changes how loggers are instantiated.
Logger is now associated with a context, passed around all methods,
(most methods had ctx, but had to add it in a few missing places).
By default Kopia does not produce any logs, but it can be overridden,
either locally for a nested context, by calling
ctx = logging.WithLogger(ctx, newLoggerFunc)
To override logs globally, call logging.SetDefaultLogger(newLoggerFunc)
This refactoring allowed removing dependency from Kopia repo
and go-logging library (the CLI still uses it, though).
It is now also possible to have all test methods emit logs using
t.Logf() so that they show up in failure reports, which should make
debugging of test failures suck less.
Percentage based on last-known snapshot size
* server: exposed last completed snapshot size in the API
* cli: added support for progress indicator (percentage based on last-known snapshot size)
* htmlui: added progress indicator in the UI (percentage based on last-known snapshot size)
- added ability to make new snapshots from the UI
- added directory picker
- hide/show macOS dock icon automatically
- fixed copy/paste on Mac (apparently if you don't have 'Edit' menu
in your app, copy/paste and many other shortcut keys simply don't
work)
- added smart time formatting ("X minutes ago", etc.) in lists
using 'moment' library
- added progress information to snapshots
CreateSnapshotSource API for ensuring source exists
Upload - starts upload on a given source or matching sources
Cancel - cancels upload on a given source or matching sources
/api/v1/repo/create
/api/v1/repo/connect
/api/v1/repo/disconnect
Refactored server code and fixed a number of outstanding robustness
issues. Tweaked the API responses a bit to make more sense when consumed
by the UI.
Those will make it possible to securely host 'kopia server' embedded
in a desktop app that runs in the background and can access UI.
- added support for using and generating TLS certificates
- added /api/v1/shutdown API to remotely trigger server shutdown
- added support for automatically shutting down server if no requests
arrive in certain amount of time
- added support for generating and printing random password to STDERR
TLS supports 3 modes:
1. serve TLS using externally-provided cert/key PEM files
2. generate & write PEM files, then serve TLS using them
3. generate and use emphemeral cert/key (prints SHA256 fingerprint)
This is enabled by `kopia server --ui` and can be viewed in a browser
at http://localhost:51515/
Right now it can only list snapshots and policies (barely).
