Commit Graph

138 Commits

Author SHA1 Message Date
Jarek Kowalski
239d809075 performance: introduced buf.Pool which helps reuse memory buffers (#345)
* performance: added buf.Pool which can be used to manage ephemeral buffers for encryption and compression
* repo: switched object writer to buf.Pool
* content: switched encryption to use buf.Pool
* object: switched compression to use buf.Pool
* testing: added missing content manager Close()
2020-03-18 20:42:16 -07:00
Jarek Kowalski
c9877bf130 performance: refactored content manager to avoid copying data
Previously we would store special field Payload for contents
that were added but never flushed to the store and it was not
encrypted. This required special handling different for pending
vs flushed contents.

This change maintains pending pack buffer ready to be flushed
and appends encrypted contents to it, which avoids data copying.
The buffers are pooled to avoid allocations.
2020-03-17 18:07:10 -07:00
Jarek Kowalski
e80f5536c3 performance: plumbed through output buffer to encryption and hashing,… (#333)
* performance: plumbed through output buffer to encryption and hashing, so that the caller can pre-allocate/reuse it

* testing: fixed how we do comparison of byte slices to account for possible nils, which can be returned from encryption
2020-03-12 08:27:44 -07:00
Jarek Kowalski
514df69afa performance: added wrapper around io.Copy()
this pools copy buffers so they can be reused instead of throwing away
after each io.Copy()
2020-03-10 21:52:30 -07:00
Jarek Kowalski
ec56ed3d70 Makefile: fixed layering test 2020-03-10 17:43:56 -07:00
Jarek Kowalski
f07d164829 Makefile: added layering-test, fixes #324 2020-03-10 13:18:23 -07:00
Julio López
d9ce3d0ad6 Inject time in Kopia components (#314)
Motivation: Allow time injection for (unit) tests, to more easily test and
verify time-dependent invariants.

Add time injection support for:

* repo.Manager
* manifest.Manager
* snapshot.Uploader

Then, wire up to these components. The content.Manager already had support for
time injection, but was not wired up from the time function passed to repo creation.

Add an internal/faketime package for testing. Mainly code movement from testing
code in the repo/content package. Motivation: make it available to other packages
outside content Also, add simple tests for faketime functions.
2020-03-10 00:42:10 -07:00
Nick
2c72fbd514 Remove FIO_USE_DOCKER env 2020-03-03 20:36:43 -08:00
Nick
b98236a535 Use fio image from dockerhub
Changing image to ljishen/fio instead of building
an image in kopia.
2020-03-03 20:36:43 -08:00
Nick
c5d8c9a271 Using docker to wrap fio execution to add robustness tool tests to Travis
Update the fio runner to use a docker image if the appropriate environment variable is set. Docker image is built via a makefile target and used in the robustness tool tests.
2020-03-03 20:36:43 -08:00
Jarek Kowalski
d95e6a3d09 sftp: Fixed issues in SFTP provider, Fixes #216
- did not work on windows due to use of filepath which uses backslash
  instead of slash
- added support for embedding SFTP key
- fixed UI controls
- misc fixes for KopiaUI
- added progress reporting
2020-03-01 18:56:06 -08:00
Jarek Kowalski
fd91186ead travis: used own retry script, since travis_retry does not work in makefile 2020-02-29 10:55:46 -08:00
Jarek Kowalski
007890ebfd travis: used own retry script, since travis_retry does not work in makefile 2020-02-29 10:46:42 -08:00
Jarek Kowalski
6d4d66621e travis: wrap setup and release steps with travis_retry to reduce flakes 2020-02-29 10:33:46 -08:00
Nick
173e18c97d Fix typo in makefile so CI can pass (upload-coverage target)
Fixing a typo in the Makefile target upload-coverage. Without fix, the target it is not found if Travis is running against a PR.
2020-02-28 18:10:10 -08:00
Jarek Kowalski
c55e53041c travis: fixed --skip-publish logic 2020-02-27 21:49:00 -08:00
Jarek Kowalski
7512f581f1 travis: checkout go.mod/go.sum at the end of travis-setup 2020-02-27 21:30:35 -08:00
Jarek Kowalski
2d214d2670 Makefile: refactored travis-release to run goreleaser first, which should avoid dirty git state 2020-02-27 21:04:57 -08:00
Jarek Kowalski
4ca9bee898 Makefile: print git diff before goreleaser 2020-02-26 22:22:19 -08:00
Jarek Kowalski
9b50a6e891 test: increased e2e test timeout
Added linear retry support when waiting for snapshots
2020-02-22 19:27:10 -08:00
Jarek Kowalski
98f1b26f39 kopia-ui: support for auto-update and publishing prerelease binaries from Travis to kopia/kopia-ui-release 2020-02-19 23:08:12 -08:00
Jarek Kowalski
11d6eb1c6c Added unit tests for HTML UI to make it a bit harder to regress.
Covered are:

- connect and create flow
- parameter pages for all providers
- connect using token
2020-02-19 18:22:45 -08:00
Jarek Kowalski
f9db94ca77 travis: disable windows signing by setting CSC_LINK and CSC_KEY_PASSWORD to empty, attempt 2 2020-02-18 06:50:35 -08:00
Jarek Kowalski
795b4f14e6 travis: disable windows signing by setting CSC_LINK and CSC_KEY_PASSWORD to empty 2020-02-17 23:29:45 -08:00
Jarek Kowalski
8208990321 travis: fix --skip-sign logic 2020-02-17 21:01:26 -08:00
Jarek Kowalski
98e877f437 travis: disable signing on windows 2020-02-17 20:19:07 -08:00
Jarek Kowalski
35a7bb6038 travis: deparallelize tasks within build to reduce flakes and improve logging 2020-02-17 20:19:07 -08:00
Jarek Kowalski
7f79c77d73 Makefile: refactored tools to be installable on Windows with only minimal deps (make/curl/unzip)
This allows full kopia and kopia-ui to be built on Windows along with
running lint and integration tests.
2020-02-17 18:45:08 -08:00
Jarek Kowalski
f8720eda62 travis: re-enabled integration tests on Windows after disabling GPG 2020-02-10 19:13:43 -08:00
Jarek Kowalski
8d9a042a44 travis: don't run GPG on Windows 2020-02-10 18:57:58 -08:00
Jarek Kowalski
24c425ab2f travis: disable integration tests on windows to see if that fixes builds hanging 2020-02-10 18:42:17 -08:00
Jarek Kowalski
0d9f1dc043 temporary: print task list after running tests on windows, some left-over process may be causing travis builds to fail 2020-02-10 17:52:08 -08:00
Jarek Kowalski
75929f65e9 travis: add integration tests and install-noui on Windows 2020-02-09 22:46:19 -08:00
Jarek Kowalski
29e5750686 travis: added bare-bones Windows build that does go test
fixed some issues that prevented go test from passing on Windows:

- webdav client used \ instead of /
- need retries around mmap.Open()
- paths are prefixed with C:\ on windows
- time.Now() does not always move forward on Windows
2020-02-09 20:22:14 -08:00
Nick Wright
67e8b6d262 Add lint and test to osx travis-releaser target
Add lint and test to the travis-releaser target for osx.
2020-02-09 18:42:38 -08:00
Jarek Kowalski
80f5a403ce travis: only build the UI on macOS since Linux will build everything except it 2020-02-03 19:08:38 -08:00
Jarek Kowalski
6493a56270 travis: fixed --skip-sign logic 2020-02-02 22:23:08 -08:00
Jarek Kowalski
2b07045ef9 travis: refactored release to be easier to understand
There's now one target, `travis-release` that can be run
locally, or on Travis CI except for:

- code signing (Travis on non-PR runs)
- publishing artifacts to GH releases (Travis on tagged releases)
- creating long-term repository for testing (Travis on tagged releases)
2020-02-02 22:23:08 -08:00
Jarek Kowalski
c3ead4bc3e Kopia UI: added desktop app shell based on Electron that runs in the tray (#183)
* app: added desktop app shell based on Electron that runs in the tray, starts a background kopia server and allows access to the UI

* icons: updated icons for the app

* htmlui: flexible containers
2020-02-01 11:58:22 -08:00
Jarek Kowalski
2bc8383e47 testing: customized e2e test directory tree shapes
based on PR feedback, instead of 3 uniform directories,
have 3 different shapes.
2020-01-16 19:40:34 -08:00
Jarek Kowalski
7d39be976f testing: split end_to_end_test into separate files.
refactored test helpers to separate package
made all tests parallel and improved the code structure
2020-01-16 19:40:34 -08:00
Jarek Kowalski
aa1b24762d lint: added retry loop around linter invocations 2020-01-03 16:39:30 -08:00
Jarek Kowalski
ac70a38101 lint: upgraded to 1.22.2 and make lint issues a build failure
fixed or silenced linter warnings, mostly due to magic numeric constants
2020-01-03 16:39:30 -08:00
Jarek Kowalski
c1454db2a9 makefile: added quick-install target which only rebuilds the binary and embeds pre-built HTMLUI 2019-12-14 19:30:09 -08:00
Jarek Kowalski
bd7a82ab94 makefile: added clean target 2019-12-14 10:48:31 -08:00
Jarek Kowalski
b93e58760a makefile: install and install-noui target 2019-12-14 10:19:08 -08:00
Jarek Kowalski
095828b140 Makefile: fixed default rule 2019-11-30 13:00:40 -08:00
Jarek Kowalski
e8a1eaa9ee htmlui: added experimental HTML-based UI
This is enabled by `kopia server --ui` and can be viewed in a browser
at http://localhost:51515/

Right now it can only list snapshots and policies (barely).
2019-11-20 14:41:38 -08:00
Jarek Kowalski
c3ffc435d7 site:reorganized NPM installation into main Makefile 2019-11-11 19:23:22 -08:00
Jarek Kowalski
0654e30b26 linter: upgraded to golangci-lint 1.18.0, fixed or suppressed warnings 2019-09-11 06:42:08 -07:00