* Add multi-language support
* Adding a guidline to outline the localisation process.
* Moved doc to separate pr
---------
Co-authored-by: lupusA <lupuapps@gmail.com>
* feat(snapshots): added ability to use cron expressions to schedule snapshots
We use `github.com/hashicorp/cronexpr` to parse and evaluate expressions,
as documented in https://github.com/hashicorp/cronexpr#implementation
* upgrade ui
* pr feedback
* chore(ci): upgraded linter to 1.53.3
This flagged a bunch of unused parameters, so the PR is larger than
usual, but 99% mechanical.
* separate lint CI task
* run Lint in separate CI
* Update display on repository summary
* Apply throughout app
* Situate units_test
* Update Command Line documentation
* Envar cleanup
* Rename to BytesString
* Restore envar string available for test
* Remove extraneous empty check and restore UIPreferences field for frontend
* PR: config bool cleanup and missed `BaseEnv`s
* Fix lint and test
Some compression algorithms are not recommended because they
allocate disproportionate amounts of memory. They are still
possible to use, just marked as NOT RECOMMENDED in the UI.
- removed a bunch of hacks and should improve the logging
performance by avoiding interfaces and data translation. This will
allow using of de-sugared loggers in performance-critical
logging situations.
- this will also allow using features of ZAP more directly without
having to reimplement them.
- moved logging.Printf() to testlogging
- refactored `uitask` to store logs in a structural format and
present them as JSON only in the UI
- renamed printf_logger.go to printf.go so that fewer columns are used
in the logs
This allows KopiaUI server to start when the repository directory
is not mounted or otherwise unavailable. Connection attempts will
be retried indefinitely and user will see new `Initializing` page.
This also exposes `Open` and `Connect` as tasks allowing the user to see
logs directly in the UI and cancel the operation.
This adds new set of APIs `/api/v1/control/*` which can be used to administratively control a running server.
Once the server is started, the administrative user can control it
using CLI commands:
export KOPIA_SERVER_ADDRESS=...
export KOPIA_SERVER_CERT_FINGERPRINT=...
export KOPIA_SERVER_PASSWORD=...
* `kopia server status` - displays status of sources managed by the server
* `kopia server snapshot` - triggers server-side upload of snapshots for managed sources
* `kopia server cancel` - cancels upload of snapshots for managed sources
* `kopia server pause` - pauses scheduled snapshots for managed sources
* `kopia server resume` - resumes scheduled snapshots for managed sources
* `kopia server refresh` - causes server to resynchronize with externally-made changes, such as policies or new sources
* `kopia server flush` - causes server to flush all pending writes
* `kopia server shutdown` - graceful shutdown of the server
Authentication uses new user `server-control` and is disabled
by default. To enable it when starting the server, provide the password
using one of the following methods:
* `--server-control-password`
* `--random-server-control-password`
* `.htpasswd` file
* `KOPIA_SERVER_CONTROL_PASSWORD` environment variable
This change allows us to tighten the API security and remove some
methods that UI user was able to call, but which were not needed.
* cli: refactored snapshot list
* cli: show range tags in snapshot list
For example if N snapshots are coalesced together because they
have identical roots we may emit now:
```
2021-03-31 23:09:27 PDT ked3400debc7dd61baffab070bafd59cd (monthly-10)
2021-04-30 06:12:53 PDT kd0576d212e55a831b7ff1636f90a7233 (monthly-4..9)
+ 5 identical snapshots until 2021-09-30 23:00:19 PDT
2021-10-31 23:22:25 PDT k846bf22aa2863d27f05e820f840b14f8 (monthly-3)
2021-11-08 21:29:31 PST k5793ddcd61ef27b93c75ab74a5828176 (latest-1..3,hourly-1..13,daily-1..7,weekly-1..4,monthly-1..2,annual-1)
+ 18 identical snapshots until 2021-12-04 10:09:54 PST
```
* server: server-side coalescing of snapshot
* ui: added coalescing of retention tags
* ui: changed how PolicyEditor is instantiated via a route
* server: added paths/resolve API
* server: refresh affected source manager after policy change
Also switched 15-second refresh cycle which is way too aggressive
to 30-minute cycle (manual refresh button can be used if needed).
* policy: allow overriding top-level policy for estimation
* server: changed source create API to always require policy
* ui: streamlined new snapshot and estimate flow
* linter fix
* throtting: implemented a Throttler based on token bucket and configurable window.
* cli: rewired throttle options to use common Limits structure and helpers
The JSON is backwards compatible.
* blob: remove explicit throttling from gcs,s3,b2 & azure
* cleanup: removed internal/throttle
* repo: add throttling wrapper around storage at the repository level
* throttling: expose APIs to get limits and add validation
* server: expose API to get/set throttle in a running server
* pr feedback
* cli: added a flag to create repository with v2 index features
* content: plumb through compression.ID parameter to content.Manager.WriteContent()
* content: expose content.Manager.SupportsContentCompression
This allows object manager to decide whether to create compressed object
or let the content manager do it.
* object: if compression is requested and the repo supports it, pass compression ID to the content manager
* cli: show compression status in 'repository status'
* cli: output compression information in 'content list' and 'content stats'
* content: compression and decompression support
* content: unit tests for compression
* object: compression tests
* testing: added integration tests against v2 index
* testing: run all e2e tests with and without content-level compression
* htmlui: added UI for specifying index format on creation
* cli: additional tests for 'content ls' and 'content stats'
* applied pr suggestions
* cli: Added --max-examples-per-bucket flag to 'kopia snapshot estimate'
Added and cleaned up a bunch of unit tests.
Fixes#1054
* cli: misc tests to increase code coverage of the cli package
* ci: move code coverage run into separate GH job
* uitask: added support for reporting string progress info
* server: report current directory as task progress
* snapshot: created reusable Estimate() method to be used during upload, cli estimate and via API
* cli: switched to snapshotfs.Estimate()
* server: added API to estimate snapshot size
* kopia-ui: fixed directory selector
* htmlui: streamlined new snapshot flow and cleaned up policy setting
See https://youtu.be/8p6csuoB3kg
This formalizes the concept of a 'UI user' which is a local
user that can call APIs the same way that UI does it.
The server will now allow access to:
- UI user (identified using `--server-username` with password specified
using `--server-password' or `--random-password`)
- remote users with usersnames/passwords specified in `--htpasswd-file`
- remote users defined in the repository using `kopia users add`
when `--allow-repository-users` is passed.
The UI user only has access to methods specifically designated as such
(normally APIs used by the UI + few special ones such as 'shutdown').
Remote users (identified via `user@host`) don't get access to UI APIs.
There are some APIs that can be accessed by any authenticated
caller (UI or remote):
- /api/v1/flush
- /api/v1/repo/status
- /api/v1/repo/sync
- /api/v1/repo/parameters
To make this easier to understand in code, refactored server handlers
to require specifing what kind of authorization is required
at registration time.
* repo: refactored connect code set up cache for server repositories
- improved logic to close the cache on last connection
- preemptively add all contents with a prefix to the cache
- refactored how config is loaded and saved
Now cache dir will be stored as relative and resolved to absolute as
part of loading and saving the file, in all other places cache dir
is expected to be absolute.
* server: removed cache directory from the API and UI
This won't be easily available and does not seem useful to expose
anyway.
* cli: enabled cache commands for server repositories
* cli: added KOPIA_CACHE_DIRECTORY environment variable
This is used on two occassions - when setting up connection (it gets
persisted in the config) and later when opening (to override the
cache location from config). It makes setting up docker container with
mounted cache somewhat easier with one environment variable.
* cli: show cache size for the server cache
* tls: present more helpful error message that includes SHA256 fingerprint of the TLS server on mismatch
* server: return the name of user who attempted to login when authentication fails
* uitask: added package for managing and introspection into tasks running inside the process
* server: added API for getting details of tasks running inside the server
* htmlui: added new tab called 'Tasks'
This allows access to progres, logs and cancelation for long-running
tasks (Snapshots, Maintenance, and in the future Restore, Estimate,
Verify)
* snapshot: improve counters returned from the upload
* server: reworked authn/authz
Previously authentication was done as an wrapper handler and
authorization was inlined. This change moves authn/authz handlers
inside the server and implements separate authorization module that's
individually tested.
Also fixed an issue where server users were not able to see global
or host-level policies.
* PR feedback
* linter: upgraded to 1.33, disabled some linters
* lint: fixed 'errorlint' errors
This ensures that all error comparisons use errors.Is() or errors.As().
We will be wrapping more errors going forward so it's important that
error checks are not strict everywhere.
Verified that there are no exceptions for errorlint linter which
guarantees that.
* lint: fixed or suppressed wrapcheck errors
* lint: nolintlint and misc cleanups
Co-authored-by: Julio López <julio+gh@kasten.io>
* cli: added --tls-print-server-cert flag
This prints complete server certificate that is base64 and PEM-encoded.
It is needed for Electron to securely connect to the server outside of
the browser, since there's no way to trust certificate by fingerprint.
* server: added repo/exists API
* server: added ClientOptions to create and connect API
* server: exposed current-user API
* server: API to change description of a repository
* htmlui: refactored connect/create flow
This cleaned up the code a lot and made UX more obvious.
* kopia-ui: simplified repository management UX
Removed repository configuration window which was confusing due to
the notion of 'server'.
Now KopiaUI will automatically launch 'kopia server --ui' for each
config found in the kopia config directory and shut it down every
time repository is disconnected.
See https://youtu.be/P4Ll_LR4UVM for a quick demo.
Fixes#583
* repo: refactored client-specific options (hostname,username,description,readonly) into new struct that is JSON-compatible with current config
* cli: added 'repository set-client' to configure parameters of connected repository
* cli: cleaned up 'repository status' output
* cli: simplified mount command
See https://youtu.be/1Nt_HIl-NWQ
It will always use WebDAV on Windows and FUSE on Unix. Removed
confusing options.
New usage:
$ kopia mount [--browse]
Mounts all snapshots in a temporary filesystem directory
(both Unix and Windows).
$ kopia mount <object> [--browse]
Mounts given object in a temporary filesystem directory
(both Unix and Windows).
$ kopia mount <object> z: [--browse]
Mounts given object as a given drive letter in Windows (using
temporary WebDAV mount).
$ kopia mount <object> * [--browse]
Mounts given object as a random drive letter in Windows.
$ kopia mount <object> /mount/path [--browse]
Mounts given object in given path in Unix.
<object> can be the ID of a directory 'k<hash>' or 'all'
Optional --browse automatically opens OS-native file browser.
* htmlui: added UI for mounting directories
See https://youtu.be/T-9SshVa1d8 for a quick demo.
Also replaced some UI text with icons.
* lint: windows-specific fix
* kopia-ui: added ability to connect to kopia server
* kopia-ui: update status page to show some data for repositories connected to API server
* kopia-ui: hide user@host selection dropdown for kopia server repositories
Support for remote content repository where all contents and
manifests are fetched over HTTP(S) instead of locally
manipulating blob storage
* server: implement content and manifest access APIs
* apiclient: moved Kopia API client to separate package
* content: exposed content.ValidatePrefix()
* manifest: added JSON serialization attributes to EntryMetadata
* repo: changed repo.Open() to return Repository instead of *DirectRepository
* repo: added apiServerRepository
* cli: added 'kopia repository connect server'
This sets up repository connection via the API server instead of
directly-manipulated storage.
* server: add support for specifying a list of usernames/password via --htpasswd-file
* tests: added API server repository E2E test
* server: only return manifests (policies and snapshots) belonging to authenticated user
* server: when serving HTML UI, prefix the title with string from KOPIA_UI_TITLE_PREFIX envar
* kopia-ui: support for multiple repositories + portability
This is a major rewrite of the app/ codebase which changes
how configuration for repositories is maintained and how it flows
through the component hierarchy.
Portable mode is enabled by creating 'repositories' subdirectory before
launching the app.
on macOS:
<parent>/KopiaUI.app
<parent>/repositories/
On Windows, option #1 - nested directory
<parent>\KopiaUI.exe
<parent>\repositories\
On Windows, option #2 - parallel directory
<parent>\some-dir\KopiaUI.exe
<parent>\repositories\
In portable mode, repositories will have 'cache' and 'logs' nested
in it.
This is mostly mechanical and changes how loggers are instantiated.
Logger is now associated with a context, passed around all methods,
(most methods had ctx, but had to add it in a few missing places).
By default Kopia does not produce any logs, but it can be overridden,
either locally for a nested context, by calling
ctx = logging.WithLogger(ctx, newLoggerFunc)
To override logs globally, call logging.SetDefaultLogger(newLoggerFunc)
This refactoring allowed removing dependency from Kopia repo
and go-logging library (the CLI still uses it, though).
It is now also possible to have all test methods emit logs using
t.Logf() so that they show up in failure reports, which should make
debugging of test failures suck less.
Percentage based on last-known snapshot size
* server: exposed last completed snapshot size in the API
* cli: added support for progress indicator (percentage based on last-known snapshot size)
* htmlui: added progress indicator in the UI (percentage based on last-known snapshot size)
