Commit Graph

229 Commits

Author SHA1 Message Date
Jarek Kowalski
8a4ac4dec3 Upgraded linter to 1.43.0 (#1505)
* fixed new gocritic violations
* fixed new 'contextcheck' violations
* fixed 'gosec' warnings
* suppressed ireturn and varnamelen linters
* fixed tenv violations, enabled building robustness tests on arm64
* fixed remaining linux failures
* makefile: fixed 'lint-all' target when running on arm64
* linter: increase deadline
* disable nilnil linter - to be enabled in separate PR
2021-11-11 17:03:11 -08:00
Jarek Kowalski
e7ed4951d6 ci: also build armv7l version of KopiaUI (#1483)
Also fixed embedded server paths for KopiaUI.

Fixes #1448
2021-11-04 16:50:47 -07:00
Julio Lopez
0629d007fc nit: remove spurious space in Makefile commands (#1311) 2021-09-22 13:44:46 -07:00
Jarek Kowalski
928150fe6b linter: upgrade to 1.42.1 (#1292) 2021-09-14 19:11:39 -07:00
Jarek Kowalski
bd7e9641da fixed minor buffer leak from internal logger, added allocation tracker (#1290) 2021-09-14 06:13:19 -07:00
Jarek Kowalski
8b2b91f9f9 content: fixed repo upgrade version (#1286)
* content: fixed repo upgrade version

Previously upgrade would enable epoch manager and index v2 but would
not set the version of the format itself. Everything worked fine
but it would not protect from old kopia opening the repository.

* ci: added compatibility test that uses real 0.8 and current binaries
2021-09-10 22:51:51 -07:00
Jarek Kowalski
5941d63130 ci: increased test timeout to 15 minutes 2021-09-08 20:43:42 -07:00
Jarek Kowalski
135c307c6c ci: fixed rclone installation 2021-09-08 20:26:23 -07:00
Jarek Kowalski
7e68d8e4c1 Consolidated format version flags (#1284) 2021-09-08 18:44:03 -07:00
Jarek Kowalski
56edb042ae content: switched defaults to use v2 index format and epoch-based indexes (#1275) 2021-09-02 05:53:01 -07:00
Jarek Kowalski
35d0f31c0d huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)
This helps recycle buffers more efficiently during snapshots.
Also, improved memory tracking, enabled profiling flags and added pprof
by default.
2021-08-20 08:45:10 -07:00
Jarek Kowalski
d6d9a1fb5f Maintenance improvements for epoch-based index structures (#1225)
* testing: KOPIA_TEST_LOG_OUTPUT logs subcommand outputs

* cli: additional flags for 'blob list'

* Makefile: run all tests against epoch-based index manager

* epoch: added support for deletion watermark, which keeps track of latest maintenance which dropped index entries

* content: added deletion watermark to content manager

* maintenance: improved maintenance without safety to force rewrites

* maintenance: skip quick maintenance when epoch manager is enabled

* maintenance: do not enable quick maintenance when epoch manager is used

* testing: skip TestIndexOptimize when running against epoch manager-backed index strutures
2021-08-02 21:08:54 -07:00
Jarek Kowalski
bdd53f05d5 ci: install provider test dependencies separately 2021-07-18 09:18:14 -07:00
Jarek Kowalski
49d8a32783 testing: split provider tests into separate steps 2021-07-18 09:15:40 -07:00
Jarek Kowalski
7b70446b74 ci: switch license NPM check to 'onlyAllow' instead of 'failOn' 2021-07-17 10:31:21 -07:00
Jarek Kowalski
42e5661bf4 ci: added license check using github.com/frapposelli/wwhrd (#1200)
* ci: added license check using github.com/frapposelli/wwhrd

* ci: also check NPM licenses

* readme: removed FOSSA badge

* Makefile: install NPM dependencies before license check
2021-07-17 09:58:25 -07:00
Jarek Kowalski
f702e65a31 deps: removed dependency on github.com/minio/minio/pkg, replaced with command line tool (#1185) 2021-07-10 10:15:31 -07:00
Jarek Kowalski
61af6399c8 cli: fixed profiling flags (#1181) 2021-07-09 21:35:00 -07:00
Jarek Kowalski
e64d5b8eab Fixed few subtle threading bugs uncovered by stress test and rewrote the test to be model-based (#1157)
* testing: refactored logs directory management

* content: fixed index mutex to be shared across all write sessions

added mutex protection during writecontent/refresh race

* testing: upload log artifacts

* content: bump revision number after index has been added

This fixes a bug where manifest manager in another session for
the same open repository may not see a content added, because they
will prematurely cache the incomplete set of contents.

This took 2 weeks to find.

* manifest: improved log output, fixed unnecessary mutex release

* testing: rewrote stress test to be model-based and more precise
2021-07-01 21:37:27 -07:00
Jarek Kowalski
40510c043d Support for content-level compression (#1076)
* cli: added a flag to create repository with v2 index features

* content: plumb through compression.ID parameter to content.Manager.WriteContent()

* content: expose content.Manager.SupportsContentCompression

This allows object manager to decide whether to create compressed object
or let the content manager do it.

* object: if compression is requested and the repo supports it, pass compression ID to the content manager

* cli: show compression status in 'repository status'

* cli: output compression information in 'content list' and 'content stats'

* content: compression and decompression support

* content: unit tests for compression

* object: compression tests

* testing: added integration tests against v2 index

* testing: run all e2e tests with and without content-level compression

* htmlui: added UI for specifying index format on creation

* cli: additional tests for 'content ls' and 'content stats'

* applied pr suggestions
2021-05-22 05:35:27 -07:00
Jarek Kowalski
5179ad2cd2 cli: test + misc improvements (#1083)
* cli: Added --max-examples-per-bucket flag to 'kopia snapshot estimate'

Added and cleaned up a bunch of unit tests.

Fixes #1054

* cli: misc tests to increase code coverage of the cli package

* ci: move code coverage run into separate GH job
2021-05-17 21:47:11 -07:00
Jarek Kowalski
30ca3e2e6c Upgraded linter to 1.40.1 (#1072)
* tools: upgraded linter to 1.40.1

* lint: fixed nolintlint vionlations

* lint: disabled tagliatele linter

* lint: fixed remaining warnings
2021-05-15 12:12:34 -07:00
Jarek Kowalski
fcd507a56d Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)
* cli: fixed remaining testability indirections for output and logging

* cli: added cli.RunSubcommand() which is used in testing to execute a subcommand in the same process

* tests: refactored most e2e tests to invoke kopia subcommands in-process

* Makefile: enable code coverage for cli/ and internal/

* testing: pass 'testing' tag to unit tests which uses much faster (insecure) password hashing scheme

* Makefile: push coverage from PRs again

* tests: disable buffer management to reduce memory usage on ARM

* cli: fixed misaligned atomic field on ARMHF

also temporarily fixed statup-time benign race condition when setting
default on the timeZone variable, which is the last global variable.
2021-05-11 22:26:28 -07:00
Steve Joachim
d2b816934c robustness: add tests for kopia server repos (#1029)
* Add error handling for robustness Checker
* robustness engine updates for concurrency
* add client-server utils for multiclient robustness
* create client package for robustness tests
* create multi-client robustness test framework
* multi-client robustness test definitions
* update robustness test runner scripts
* Address unparam lint errors

Co-authored-by: Julio Lopez <julio+gh@kasten.io>
2021-04-29 21:45:13 -07:00
Denis Fondras
28b21ac900 be consistent in $(MAKE) usage (#992) 2021-04-18 11:58:45 -07:00
Jarek Kowalski
922bc7cbc8 ci: refactored credentials handling (#987)
This strengthens credential handling after our signing keys may have
been leaked in the [codecov.io breach](https://about.codecov.io/security-update/)

* pass only minimal credentials to each build step to avoid
  exposing sensitive tokens to tools that don't need them
  (like code coverage)
* removed encrypted credential files and replaced with environment-based
* allow full ci/cd including publishing artifacts from forks
* regenerated all passwords, tokens and service accounts
* do not install Google Cloud SDK on GHA - it's already there
* moved RPM signing to 'Stage And Publish Artifacts' phase
* generated new GPG signing key

See https://kopia.discourse.group/t/important-impact-of-codecov-io-security-issue-on-kopia-build-pipeline/377
2021-04-16 08:17:13 -07:00
Jarek Kowalski
7c088338ce testing: upload endurance test logs as artifacts, run more frequently 2021-04-07 14:11:39 -07:00
Jarek Kowalski
3e438dd2b8 ci: run provider-tests as separate GH job (#952) 2021-04-06 21:46:54 -07:00
Jarek Kowalski
79adef0f33 ci: run endurance test 2021-04-03 19:10:36 -07:00
Jarek Kowalski
9a756c719f Enabled race detector in CI, fixed a few data races (#919)
* content: fixed data race in IterateUnreferencedBlobs

* upload: fixed data race between uploader and estimator

* testing: fixed data race in repo/blob/logging test

* makefile: run tests on CI/linux/amd64 with -race

* robustness: fixed test race

* content: fixed data race getContentDataUnlocked that triggers TestParallelWrites - looks scary but in practice very hard to trigger in real life and does not cause data corruption

* testing: reduce test complexity under race detector

* server: fixed minor race in refreshStatus()

* testing: reduced depth of sharedTestDataDir2

* ci: run race detector in separate job

* ci: run unit test race detector in parallel to integration tests
2021-04-02 18:21:04 -07:00
Jarek Kowalski
c9225dc429 Makefile: replace single slash with double slash when signing
on Windows bash is messing up flags starting with / for some reason
2021-03-23 09:52:13 -07:00
Jarek Kowalski
f69253899c ci: disable retry when building kopia 2021-03-23 09:08:40 -07:00
Jarek Kowalski
21af8914b1 ci: fixed windows signing and scoop url 2021-03-23 09:07:16 -07:00
Jarek Kowalski
16282b5b61 Makefile refactoring + publish scoop package (#907)
* Makefile: refactoring

- added signing of windows exe in a zip package
- removed Windows build from goreleaser
- removed homebrew from goreleaser
- enabled incremental builds

* ci: publish scoop package
2021-03-22 22:39:24 -07:00
Jarek Kowalski
2d0491d997 ci: disable output when installing Google Cloud SDK 2021-03-21 22:23:42 -07:00
Jarek Kowalski
f0b789c985 ci: install Google Cloud SDK in a retry loop to fix tar: write error 2021-03-21 22:13:20 -07:00
Jarek Kowalski
a3d4b1de8e Update Makefile 2021-03-21 19:38:03 -07:00
Jarek Kowalski
2e5a3ad3a3 ci: build and publish Homebrew package 2021-03-21 18:41:08 -07:00
Jarek Kowalski
bcfba334da ci: stop using goreleaser to produce macOS binaries, make universal binary as separate .tar.gz 2021-03-21 17:58:04 -07:00
Jarek Kowalski
335ff79e78 CI refactoring (#903)
* ci: publish additional binaries as artifacts

* ci: install {htmlui,app}/node_modules as part of ci-setup

* ci: added retry to improve robustness of builds

* ci: continue-on-error for non-pull-requests

* ci: moved publish-packages to Stage And Publish Artifacts phase
2021-03-21 15:17:39 -07:00
Jarek Kowalski
342919892e ci: install signing tools just before building the binary with UI 2021-03-20 12:09:57 -07:00
Jarek Kowalski
10774d5acd ci/cd: refactored how GH releases are published (#867)
- stop publishing to github directly from goreleaser and electron builder
- upload dist/ as artifact at the end of each build
- stage release in a separate directory, generated checksum, GPG sign and publish
- removed spaces from Windows setup exe
2021-03-20 11:46:05 -07:00
Jarek Kowalski
3b6cf5cc7b docker: publish docker images to docker hub (#896)
docker: push dockerhub tags

* major.minor - tracks latest major.minor release (except major==0)
* latest - tracks official stable releases
* testing - tracks official stable releases or pre-releases (beta, rc)
* unstable - tracks nightly/unstable builds
2021-03-19 12:51:24 -07:00
Jarek Kowalski
a6e96c31f8 Makefie: don't sign macOS binaries unless MACOS_SIGNING_IDENTITY is defined 2021-03-04 19:55:39 -08:00
Jarek Kowalski
5928df3498 Makefile: build fat macOS binary (amd64 and arm64) (#866)
This will be embedded in KopiaUI which is built on macOS runner.
2021-03-04 19:35:26 -08:00
Jarek Kowalski
01cd9ba134 Makefile: run goreleaser only on linux/amd64 in non-PR mode (#861) 2021-03-02 08:29:37 -08:00
Jarek Kowalski
ff158ea763 goreleaser: updated to 0.157.0 to produce darwin/arm64 build (#855) 2021-02-27 12:10:20 -08:00
Jarek Kowalski
5ddc896c55 coverage: publish coverage results on a best-effort basis 2021-02-27 09:36:39 -08:00
Jarek Kowalski
e6637fe112 Makefile: only install cloud sdk on amd64 2021-02-23 06:20:06 -08:00
Jarek Kowalski
8efe86d3a3 Makefile: typo fix 2021-02-23 06:08:20 -08:00