Commit Graph

32 Commits

Author SHA1 Message Date
Jarek Kowalski
40510c043d Support for content-level compression (#1076)
* cli: added a flag to create repository with v2 index features

* content: plumb through compression.ID parameter to content.Manager.WriteContent()

* content: expose content.Manager.SupportsContentCompression

This allows object manager to decide whether to create compressed object
or let the content manager do it.

* object: if compression is requested and the repo supports it, pass compression ID to the content manager

* cli: show compression status in 'repository status'

* cli: output compression information in 'content list' and 'content stats'

* content: compression and decompression support

* content: unit tests for compression

* object: compression tests

* testing: added integration tests against v2 index

* testing: run all e2e tests with and without content-level compression

* htmlui: added UI for specifying index format on creation

* cli: additional tests for 'content ls' and 'content stats'

* applied pr suggestions
2021-05-22 05:35:27 -07:00
Jarek Kowalski
5179ad2cd2 cli: test + misc improvements (#1083)
* cli: Added --max-examples-per-bucket flag to 'kopia snapshot estimate'

Added and cleaned up a bunch of unit tests.

Fixes #1054

* cli: misc tests to increase code coverage of the cli package

* ci: move code coverage run into separate GH job
2021-05-17 21:47:11 -07:00
Jarek Kowalski
132e2eef50 New snapshot UX - streamlined snapshot creation and policy setting (#878)
* uitask: added support for reporting string progress info

* server: report current directory as task progress

* snapshot: created reusable Estimate() method to be used during upload, cli estimate and via API

* cli: switched to snapshotfs.Estimate()

* server: added API to estimate snapshot size

* kopia-ui: fixed directory selector

* htmlui: streamlined new snapshot flow and cleaned up policy setting

See https://youtu.be/8p6csuoB3kg
2021-03-10 23:04:55 -08:00
Jarek Kowalski
1f1465f4ba Improvements and cleanups for connecting to kopia server (#870)
* repo: refactored connect code set up cache for server repositories

- improved logic to close the cache on last connection
- preemptively add all contents with a prefix to the cache
- refactored how config is loaded and saved

Now cache dir will be stored as relative and resolved to absolute as
part of loading and saving the file, in all other places cache dir
is expected to be absolute.

* server: removed cache directory from the API and UI

This won't be easily available and does not seem useful to expose
anyway.

* cli: enabled cache commands for server repositories

* cli: added KOPIA_CACHE_DIRECTORY environment variable

This is used on two occassions - when setting up connection (it gets
persisted in the config) and later when opening (to override the
cache location from config). It makes setting up docker container with
mounted cache somewhat easier with one environment variable.

* cli: show cache size for the server cache

* tls: present more helpful error message that includes SHA256 fingerprint of the TLS server on mismatch

* server: return the name of user who attempted to login when authentication fails
2021-03-07 11:25:21 -08:00
Jarek Kowalski
504238df7a Restore UI (#823)
* server: added restore api
* htmlui: restore UI
2021-02-11 02:08:47 -08:00
Jarek Kowalski
dc3d9f53a3 Added support for Tasks in the UI (#818)
* uitask: added package for managing and introspection into tasks running inside the process

* server: added API for getting details of tasks running inside the server

* htmlui: added new tab called 'Tasks'

This allows access to progres, logs and cancelation for long-running
tasks (Snapshots, Maintenance, and in the future Restore, Estimate,
Verify)

* snapshot: improve counters returned from the upload
2021-02-08 17:54:39 -08:00
Jarek Kowalski
5912247f29 server: reworked authn/authz (#788)
* server: reworked authn/authz

Previously authentication was done as an wrapper handler and
authorization was inlined. This change moves authn/authz handlers
inside the server and implements separate authorization module that's
individually tested.

Also fixed an issue where server users were not able to see global
or host-level policies.

* PR feedback
2021-01-21 07:31:34 -08:00
Jarek Kowalski
1f3b8d4da4 upgrade linter to 1.35 (#786)
* lint: added test that enforces Makefile and GH action linter versions are in sync
* workaround for linter gomnd problem - https://github.com/golangci/golangci-lint/issues/1653
2021-01-16 18:21:16 -08:00
Jarek Kowalski
3b87902433 Kopia UI improvements for repository management (#592)
* cli: added --tls-print-server-cert flag

This prints complete server certificate that is base64 and PEM-encoded.

It is needed for Electron to securely connect to the server outside of
the browser, since there's no way to trust certificate by fingerprint.

* server: added repo/exists API

* server: added ClientOptions to create and connect API

* server: exposed current-user API

* server: API to change description of a repository

* htmlui: refactored connect/create flow

This cleaned up the code a lot and made UX more obvious.

* kopia-ui: simplified repository management UX

Removed repository configuration window which was confusing due to
the notion of 'server'.

Now KopiaUI will automatically launch 'kopia server --ui' for each
config found in the kopia config directory and shut it down every
time repository is disconnected.

See https://youtu.be/P4Ll_LR4UVM for a quick demo.

Fixes #583
2020-09-07 08:00:19 -07:00
Jarek Kowalski
29ce1819cb Added support for setting and changing repository client options (description, read-only, hostname, username) (#589)
* repo: refactored client-specific options (hostname,username,description,readonly) into new struct that is JSON-compatible with current config

* cli: added 'repository set-client' to configure parameters of connected repository

* cli: cleaned up 'repository status' output
2020-09-04 13:57:15 -07:00
Jarek Kowalski
a5838ff34c Improvements to UX for mounting directories (both CLI and KopiaUI) (#573)
* cli: simplified mount command

See https://youtu.be/1Nt_HIl-NWQ

It will always use WebDAV on Windows and FUSE on Unix. Removed
confusing options.

New usage:

$ kopia mount [--browse]
    Mounts all snapshots in a temporary filesystem directory
    (both Unix and Windows).

$ kopia mount <object> [--browse]
    Mounts given object in a temporary filesystem directory
    (both Unix and Windows).

$ kopia mount <object> z: [--browse]
    Mounts given object as a given drive letter in Windows (using
    temporary WebDAV mount).

$ kopia mount <object> * [--browse]
    Mounts given object as a random drive letter in Windows.

$ kopia mount <object> /mount/path [--browse]
    Mounts given object in given path in Unix.

<object> can be the ID of a directory 'k<hash>' or 'all'

Optional --browse automatically opens OS-native file browser.

* htmlui: added UI for mounting directories

See https://youtu.be/T-9SshVa1d8 for a quick demo.

Also replaced some UI text with icons.

* lint: windows-specific fix
2020-09-03 17:46:48 -07:00
Jarek Kowalski
48f253173b kopia-ui: added ability to connect to kopia server and few other minor tweaks (#546)
* kopia-ui: added ability to connect to kopia server

* kopia-ui: update status page to show some data for repositories connected to API server

* kopia-ui: hide user@host selection dropdown for kopia server repositories
2020-08-16 17:57:37 -07:00
Jarek Kowalski
9a6dea898b Linter upgrade to v1.30.0 (#526)
* fixed godot linter errors
* reformatted source with gofumpt
* disabled some linters
* fixed nolintlint warnings
* fixed gci warnings
* lint: fixed 'nestif' warnings
* lint: fixed 'exhaustive' warnings
* lint: fixed 'gocritic' warnings
* lint: fixed 'noctx' warnings
* lint: fixed 'wsl' warnings
* lint: fixed 'goerr113' warnings
* lint: fixed 'gosec' warnings
* lint: upgraded linter to 1.30.0
* lint: more 'exhaustive' warnings

Co-authored-by: Nick <nick@kasten.io>
2020-08-12 19:28:53 -07:00
Jarek Kowalski
9b68a631e6 Highlight snapshot errors in the UI and CLI (#376)
* upload: exposed numFailed and failedEntries on directory summary

* cli: better present snapshot errors

* htmlui: display snapshot errors
2020-03-22 14:18:47 -07:00
Jarek Kowalski
897483299f Kopia UI & CLI: support for progress indicator (#268)
Percentage based on last-known snapshot size

* server: exposed last completed snapshot size in the API
* cli: added support for progress indicator (percentage based on last-known snapshot size)
* htmlui: added progress indicator in the UI (percentage based on last-known snapshot size)
2020-02-24 17:55:02 -08:00
Jarek Kowalski
27854d85ed server: report local username and hostname when listing sources 2020-02-22 19:27:10 -08:00
Jarek Kowalski
fde2f2e0e6 server: additional status code from CreateSnapshotSource, more tests 2020-02-22 19:27:10 -08:00
Jarek Kowalski
ab2c906f2c server: implemented remaining server API methods
CreateSnapshotSource API for ensuring source exists
Upload - starts upload on a given source or matching sources
Cancel - cancels upload on a given source or matching sources
2020-02-22 19:27:10 -08:00
Jarek Kowalski
c42b5cd89f server: API server for CRUD on individual Policies 2020-02-16 23:04:17 -08:00
Jarek Kowalski
4cb898927c server: new APIs and error codes to support UI flow for connecting to repository 2020-02-16 22:43:36 -08:00
Jarek Kowalski
0f79279f5e server: added support for new verbs in the API
/api/v1/repo/create
/api/v1/repo/connect
/api/v1/repo/disconnect

Refactored server code and fixed a number of outstanding robustness
issues. Tweaked the API responses a bit to make more sense when consumed
by the UI.
2020-02-13 17:23:50 -08:00
Jarek Kowalski
ac70a38101 lint: upgraded to 1.22.2 and make lint issues a build failure
fixed or silenced linter warnings, mostly due to magic numeric constants
2020-01-03 16:39:30 -08:00
Jarek Kowalski
72520029b0 golangci-lint: added more linters
Also fixed pre-existing lint errors.
2019-06-02 22:56:57 -07:00
Jarek Kowalski
54edb97b3a refactoring: renamed repo/block to repo/content
Also introduced strongly typed content.ID and manifest.ID (instead of string)

This aligns identifiers across all layers of repository:

blob.ID
content.ID
object.ID
manifest.ID
2019-06-01 22:24:19 -07:00
Jarek Kowalski
63303904e1 switched remaining fmt.Errorf to errors.Wrap() 2019-06-01 10:57:05 -07:00
Jarek Kowalski
0c41d41276 Fixed up paths after merge 2019-05-27 15:48:39 -07:00
Jarek Kowalski
327d8317d8 refactored repo/ into separate github.com/kopia/repo/ git repository 2018-10-26 20:40:57 -07:00
Jarek Kowalski
d47f662316 refactoring: moved kopia/kopia/policy to kopia/kopia/snapshot/policy 2018-10-20 09:55:52 -07:00
Jarek Kowalski
91066f2469 reorganized low-level repository packages by moving them all under kopia/kopia/repo/ 2018-08-30 22:01:05 -07:00
Jarek Kowalski
3ef4593b03 refactored policy management into separate policy/ package 2018-08-29 23:08:40 -07:00
Jarek Kowalski
5386370906 emit snapshot size 2018-08-04 09:57:10 -07:00
Jarek Kowalski
2f1cf8129b added skeleton of CLI actions for manipulating server sources 2018-06-18 18:02:36 -07:00