Commit Graph

61 Commits

Author SHA1 Message Date
Jarek Kowalski
5d9133a72b ci: switched code coverage tool to use GH action (#1391) 2021-10-14 23:51:28 -07:00
Z
33c8733750 add unicode filename test, add env switches for long filenames/unicode filenames, update workflow file to include env variables (#1371) 2021-10-09 12:38:36 -07:00
Julio Lopez
6472d171d4 Skip publishing artifacts on fork repos (#1364) 2021-10-07 08:42:40 -07:00
Jarek Kowalski
8b2b91f9f9 content: fixed repo upgrade version (#1286)
* content: fixed repo upgrade version

Previously upgrade would enable epoch manager and index v2 but would
not set the version of the format itself. Everything worked fine
but it would not protect from old kopia opening the repository.

* ci: added compatibility test that uses real 0.8 and current binaries
2021-09-10 22:51:51 -07:00
Julio Lopez
1077cc24d3 blob/s3: point-in-time support for s3 versioned stores (#1259)
Adds a `--point-in-time' flag for `repo connect`
2021-09-02 21:38:08 -07:00
Jarek Kowalski
cc692c40af ci: delete golang installation on armhf before installing a new one (#1276)
Per https://github.com/golang/go/issues/48134#issuecomment-911431265
2021-09-02 19:52:49 -07:00
Jarek Kowalski
06451aa7c2 upgrade golang to 1.17 + various dependencies (#1256) 2021-09-01 17:08:09 -07:00
Julio Lopez
082785af06 ci: run stress test on kopia/kopia repo (#1257) 2021-09-01 17:01:37 -07:00
Julio Lopez
310f45101f ci: avoid double workflow executions on PRs (#1258) 2021-09-01 06:06:10 -07:00
Jarek Kowalski
d77e771df7 provider test: fixed typo 2021-07-29 06:26:06 -07:00
Jarek Kowalski
7dd9e332ad ci: fixed syntax 2021-07-18 09:33:38 -07:00
Jarek Kowalski
e2e0cb37d7 ci: changed continue-on-error: true to if: succeeded() || failed() 2021-07-18 09:31:23 -07:00
Jarek Kowalski
bdd53f05d5 ci: install provider test dependencies separately 2021-07-18 09:18:14 -07:00
Jarek Kowalski
49d8a32783 testing: split provider tests into separate steps 2021-07-18 09:15:40 -07:00
Jarek Kowalski
33cf322941 ci: run provider test more often to get better stability signal 2021-07-18 09:09:43 -07:00
Jarek Kowalski
42e5661bf4 ci: added license check using github.com/frapposelli/wwhrd (#1200)
* ci: added license check using github.com/frapposelli/wwhrd

* ci: also check NPM licenses

* readme: removed FOSSA badge

* Makefile: install NPM dependencies before license check
2021-07-17 09:58:25 -07:00
Jarek Kowalski
f702e65a31 deps: removed dependency on github.com/minio/minio/pkg, replaced with command line tool (#1185) 2021-07-10 10:15:31 -07:00
Jarek Kowalski
f95ef881ce testing: fixed typo 2021-07-10 08:55:17 -07:00
Jarek Kowalski
a1aa6ae31f tests: install minio mc tool (#1184) 2021-07-10 08:53:54 -07:00
Jarek Kowalski
e64d5b8eab Fixed few subtle threading bugs uncovered by stress test and rewrote the test to be model-based (#1157)
* testing: refactored logs directory management

* content: fixed index mutex to be shared across all write sessions

added mutex protection during writecontent/refresh race

* testing: upload log artifacts

* content: bump revision number after index has been added

This fixes a bug where manifest manager in another session for
the same open repository may not see a content added, because they
will prematurely cache the incomplete set of contents.

This took 2 weeks to find.

* manifest: improved log output, fixed unnecessary mutex release

* testing: rewrote stress test to be model-based and more precise
2021-07-01 21:37:27 -07:00
Jarek Kowalski
e15a79474c azure: added support for using SAS Tokens instead of storage keys (#1093)
Fixes #1071
2021-05-22 12:43:55 -07:00
Jarek Kowalski
5179ad2cd2 cli: test + misc improvements (#1083)
* cli: Added --max-examples-per-bucket flag to 'kopia snapshot estimate'

Added and cleaned up a bunch of unit tests.

Fixes #1054

* cli: misc tests to increase code coverage of the cli package

* ci: move code coverage run into separate GH job
2021-05-17 21:47:11 -07:00
Jarek Kowalski
862298f979 testing: provider test fixes (#1073) 2021-05-15 20:50:28 -07:00
Jarek Kowalski
d81f3f1131 ci: do not upload artifacts from self-hosted runners, fixes build race (#1015)
Fixes #1014
2021-04-25 20:12:25 -07:00
Jarek Kowalski
922bc7cbc8 ci: refactored credentials handling (#987)
This strengthens credential handling after our signing keys may have
been leaked in the [codecov.io breach](https://about.codecov.io/security-update/)

* pass only minimal credentials to each build step to avoid
  exposing sensitive tokens to tools that don't need them
  (like code coverage)
* removed encrypted credential files and replaced with environment-based
* allow full ci/cd including publishing artifacts from forks
* regenerated all passwords, tokens and service accounts
* do not install Google Cloud SDK on GHA - it's already there
* moved RPM signing to 'Stage And Publish Artifacts' phase
* generated new GPG signing key

See https://kopia.discourse.group/t/important-impact-of-codecov-io-security-issue-on-kopia-build-pipeline/377
2021-04-16 08:17:13 -07:00
Jarek Kowalski
52220a75a9 testing: upload endurance test logs as artifacts regardless of status, don't ignore faulures 2021-04-07 14:21:05 -07:00
Jarek Kowalski
7c088338ce testing: upload endurance test logs as artifacts, run more frequently 2021-04-07 14:11:39 -07:00
Jarek Kowalski
3e438dd2b8 ci: run provider-tests as separate GH job (#952) 2021-04-06 21:46:54 -07:00
Jarek Kowalski
79adef0f33 ci: run endurance test 2021-04-03 19:10:36 -07:00
Jarek Kowalski
9a756c719f Enabled race detector in CI, fixed a few data races (#919)
* content: fixed data race in IterateUnreferencedBlobs

* upload: fixed data race between uploader and estimator

* testing: fixed data race in repo/blob/logging test

* makefile: run tests on CI/linux/amd64 with -race

* robustness: fixed test race

* content: fixed data race getContentDataUnlocked that triggers TestParallelWrites - looks scary but in practice very hard to trigger in real life and does not cause data corruption

* testing: reduce test complexity under race detector

* server: fixed minor race in refreshStatus()

* testing: reduced depth of sharedTestDataDir2

* ci: run race detector in separate job

* ci: run unit test race detector in parallel to integration tests
2021-04-02 18:21:04 -07:00
Jarek Kowalski
d0f2ef53d7 blob: improved startup error handling of rclone and webdav PutBlob race (#915)
* added framework for unit testing against remote real rclone remotes,
  added google drive backend
* added parallelism to blobtesting which revealed some races during
  PutBlob with WebDAV.
2021-03-28 08:26:35 -07:00
Jarek Kowalski
b8e89c2808 s3: refactored testing to support multiple S3-compatible providers (#914)
The credentials are passed via JSON matching s3.Options struct
Fixed some verification issue with Wasabi.
2021-03-26 14:33:20 -07:00
Jarek Kowalski
16282b5b61 Makefile refactoring + publish scoop package (#907)
* Makefile: refactoring

- added signing of windows exe in a zip package
- removed Windows build from goreleaser
- removed homebrew from goreleaser
- enabled incremental builds

* ci: publish scoop package
2021-03-22 22:39:24 -07:00
Jarek Kowalski
2e5a3ad3a3 ci: build and publish Homebrew package 2021-03-21 18:41:08 -07:00
Jarek Kowalski
94b3544653 ci: moved Docker installation to 'publish' phase 2021-03-21 16:28:08 -07:00
Jarek Kowalski
3987973964 ci: install credentials before pushing APT/RPM packages 2021-03-21 15:56:23 -07:00
Jarek Kowalski
335ff79e78 CI refactoring (#903)
* ci: publish additional binaries as artifacts

* ci: install {htmlui,app}/node_modules as part of ci-setup

* ci: added retry to improve robustness of builds

* ci: continue-on-error for non-pull-requests

* ci: moved publish-packages to Stage And Publish Artifacts phase
2021-03-21 15:17:39 -07:00
Jarek Kowalski
97eb7b23dc ci: store changelog along with artifacts 2021-03-20 15:10:53 -07:00
Jarek Kowalski
10774d5acd ci/cd: refactored how GH releases are published (#867)
- stop publishing to github directly from goreleaser and electron builder
- upload dist/ as artifact at the end of each build
- stage release in a separate directory, generated checksum, GPG sign and publish
- removed spaces from Windows setup exe
2021-03-20 11:46:05 -07:00
Jarek Kowalski
3b6cf5cc7b docker: publish docker images to docker hub (#896)
docker: push dockerhub tags

* major.minor - tracks latest major.minor release (except major==0)
* latest - tracks official stable releases
* testing - tracks official stable releases or pre-releases (beta, rc)
* unstable - tracks nightly/unstable builds
2021-03-19 12:51:24 -07:00
Jarek Kowalski
dd740d42e6 Revert "ci: run build every 4 hours to get more test samples (temporary)"
This reverts commit e4bb295a1c.
2021-03-16 22:32:44 -07:00
Jarek Kowalski
e4bb295a1c ci: run build every 4 hours to get more test samples (temporary) 2021-03-16 08:45:58 -07:00
Jarek Kowalski
5928df3498 Makefile: build fat macOS binary (amd64 and arm64) (#866)
This will be embedded in KopiaUI which is built on macOS runner.
2021-03-04 19:35:26 -08:00
Jarek Kowalski
e2b9a81ac3 Major CI/CD refactoring and re-added support for ARM/ARM64 runners (#849)
* ci: refactored CI/CD logic & Makefile

- removed all travis CI emulation environment variables and replaced with:

CI_TAG=<empty>|tag
IS_PULL_REQUEST=false|true

- refactored all OS and architecture-specific decisions to use around standard GOOS/GOARCH values instead of uname/OS
- re-added self-hosted runner for ARMHF (3 replicas)
- added brand new self-hosted runner for ARM64 (3 replicas)
- disabled attempts to publish and sign on forks
- improved integration test log output to better see timings and sub-tests
- print longest tests (unit tests and integration) after each run
- verified that all configurations build successfully on a clone (jkowalski/kopia)
- run make setup in parallel

* testing: fixed tests on ARM and ARM64

- fixed ARM-specific alignment issue
- cleaned up test logging
- fixed huge params warning threshold because it was tripping on ARM.
- reduced test complexity to make them fit in 15 minutes
2021-02-23 00:52:54 -08:00
Julio López
48fe5e96b8 Upgrade to Go 1.16 (#842)
* Use 1.16 in GHA workflows
* Require Go 1.16 in `go.mod`
2021-02-17 21:52:23 -08:00
Jarek Kowalski
fef23a7e2c disable code coverage 2021-02-14 21:45:56 -08:00
Jarek Kowalski
0b1d6d3159 codecov: run only on ubuntu 2021-02-14 21:18:09 -08:00
Jarek Kowalski
0581a07220 coverage: use GH action to upload coverage 2021-02-14 20:45:26 -08:00
Jarek Kowalski
70b0d8d9c9 integrate with codecov.io (#836)
* integrate with codecov.io
2021-02-14 20:17:05 -08:00
Jarek Kowalski
a655d7b9e2 minor build tweaks (#830) 2021-02-12 18:36:01 -08:00