Commit Graph

33 Commits

Author SHA1 Message Date
Z
33c8733750 add unicode filename test, add env switches for long filenames/unicode filenames, update workflow file to include env variables (#1371) 2021-10-09 12:38:36 -07:00
Julio Lopez
6472d171d4 Skip publishing artifacts on fork repos (#1364) 2021-10-07 08:42:40 -07:00
Jarek Kowalski
cc692c40af ci: delete golang installation on armhf before installing a new one (#1276)
Per https://github.com/golang/go/issues/48134#issuecomment-911431265
2021-09-02 19:52:49 -07:00
Jarek Kowalski
06451aa7c2 upgrade golang to 1.17 + various dependencies (#1256) 2021-09-01 17:08:09 -07:00
Jarek Kowalski
e64d5b8eab Fixed few subtle threading bugs uncovered by stress test and rewrote the test to be model-based (#1157)
* testing: refactored logs directory management

* content: fixed index mutex to be shared across all write sessions

added mutex protection during writecontent/refresh race

* testing: upload log artifacts

* content: bump revision number after index has been added

This fixes a bug where manifest manager in another session for
the same open repository may not see a content added, because they
will prematurely cache the incomplete set of contents.

This took 2 weeks to find.

* manifest: improved log output, fixed unnecessary mutex release

* testing: rewrote stress test to be model-based and more precise
2021-07-01 21:37:27 -07:00
Jarek Kowalski
5179ad2cd2 cli: test + misc improvements (#1083)
* cli: Added --max-examples-per-bucket flag to 'kopia snapshot estimate'

Added and cleaned up a bunch of unit tests.

Fixes #1054

* cli: misc tests to increase code coverage of the cli package

* ci: move code coverage run into separate GH job
2021-05-17 21:47:11 -07:00
Jarek Kowalski
d81f3f1131 ci: do not upload artifacts from self-hosted runners, fixes build race (#1015)
Fixes #1014
2021-04-25 20:12:25 -07:00
Jarek Kowalski
922bc7cbc8 ci: refactored credentials handling (#987)
This strengthens credential handling after our signing keys may have
been leaked in the [codecov.io breach](https://about.codecov.io/security-update/)

* pass only minimal credentials to each build step to avoid
  exposing sensitive tokens to tools that don't need them
  (like code coverage)
* removed encrypted credential files and replaced with environment-based
* allow full ci/cd including publishing artifacts from forks
* regenerated all passwords, tokens and service accounts
* do not install Google Cloud SDK on GHA - it's already there
* moved RPM signing to 'Stage And Publish Artifacts' phase
* generated new GPG signing key

See https://kopia.discourse.group/t/important-impact-of-codecov-io-security-issue-on-kopia-build-pipeline/377
2021-04-16 08:17:13 -07:00
Jarek Kowalski
d0f2ef53d7 blob: improved startup error handling of rclone and webdav PutBlob race (#915)
* added framework for unit testing against remote real rclone remotes,
  added google drive backend
* added parallelism to blobtesting which revealed some races during
  PutBlob with WebDAV.
2021-03-28 08:26:35 -07:00
Jarek Kowalski
b8e89c2808 s3: refactored testing to support multiple S3-compatible providers (#914)
The credentials are passed via JSON matching s3.Options struct
Fixed some verification issue with Wasabi.
2021-03-26 14:33:20 -07:00
Jarek Kowalski
16282b5b61 Makefile refactoring + publish scoop package (#907)
* Makefile: refactoring

- added signing of windows exe in a zip package
- removed Windows build from goreleaser
- removed homebrew from goreleaser
- enabled incremental builds

* ci: publish scoop package
2021-03-22 22:39:24 -07:00
Jarek Kowalski
2e5a3ad3a3 ci: build and publish Homebrew package 2021-03-21 18:41:08 -07:00
Jarek Kowalski
94b3544653 ci: moved Docker installation to 'publish' phase 2021-03-21 16:28:08 -07:00
Jarek Kowalski
3987973964 ci: install credentials before pushing APT/RPM packages 2021-03-21 15:56:23 -07:00
Jarek Kowalski
335ff79e78 CI refactoring (#903)
* ci: publish additional binaries as artifacts

* ci: install {htmlui,app}/node_modules as part of ci-setup

* ci: added retry to improve robustness of builds

* ci: continue-on-error for non-pull-requests

* ci: moved publish-packages to Stage And Publish Artifacts phase
2021-03-21 15:17:39 -07:00
Jarek Kowalski
97eb7b23dc ci: store changelog along with artifacts 2021-03-20 15:10:53 -07:00
Jarek Kowalski
10774d5acd ci/cd: refactored how GH releases are published (#867)
- stop publishing to github directly from goreleaser and electron builder
- upload dist/ as artifact at the end of each build
- stage release in a separate directory, generated checksum, GPG sign and publish
- removed spaces from Windows setup exe
2021-03-20 11:46:05 -07:00
Jarek Kowalski
3b6cf5cc7b docker: publish docker images to docker hub (#896)
docker: push dockerhub tags

* major.minor - tracks latest major.minor release (except major==0)
* latest - tracks official stable releases
* testing - tracks official stable releases or pre-releases (beta, rc)
* unstable - tracks nightly/unstable builds
2021-03-19 12:51:24 -07:00
Jarek Kowalski
dd740d42e6 Revert "ci: run build every 4 hours to get more test samples (temporary)"
This reverts commit e4bb295a1c.
2021-03-16 22:32:44 -07:00
Jarek Kowalski
e4bb295a1c ci: run build every 4 hours to get more test samples (temporary) 2021-03-16 08:45:58 -07:00
Jarek Kowalski
5928df3498 Makefile: build fat macOS binary (amd64 and arm64) (#866)
This will be embedded in KopiaUI which is built on macOS runner.
2021-03-04 19:35:26 -08:00
Jarek Kowalski
e2b9a81ac3 Major CI/CD refactoring and re-added support for ARM/ARM64 runners (#849)
* ci: refactored CI/CD logic & Makefile

- removed all travis CI emulation environment variables and replaced with:

CI_TAG=<empty>|tag
IS_PULL_REQUEST=false|true

- refactored all OS and architecture-specific decisions to use around standard GOOS/GOARCH values instead of uname/OS
- re-added self-hosted runner for ARMHF (3 replicas)
- added brand new self-hosted runner for ARM64 (3 replicas)
- disabled attempts to publish and sign on forks
- improved integration test log output to better see timings and sub-tests
- print longest tests (unit tests and integration) after each run
- verified that all configurations build successfully on a clone (jkowalski/kopia)
- run make setup in parallel

* testing: fixed tests on ARM and ARM64

- fixed ARM-specific alignment issue
- cleaned up test logging
- fixed huge params warning threshold because it was tripping on ARM.
- reduced test complexity to make them fit in 15 minutes
2021-02-23 00:52:54 -08:00
Julio López
48fe5e96b8 Upgrade to Go 1.16 (#842)
* Use 1.16 in GHA workflows
* Require Go 1.16 in `go.mod`
2021-02-17 21:52:23 -08:00
Jarek Kowalski
fef23a7e2c disable code coverage 2021-02-14 21:45:56 -08:00
Jarek Kowalski
0b1d6d3159 codecov: run only on ubuntu 2021-02-14 21:18:09 -08:00
Jarek Kowalski
0581a07220 coverage: use GH action to upload coverage 2021-02-14 20:45:26 -08:00
Jarek Kowalski
70b0d8d9c9 integrate with codecov.io (#836)
* integrate with codecov.io
2021-02-14 20:17:05 -08:00
Jarek Kowalski
a655d7b9e2 minor build tweaks (#830) 2021-02-12 18:36:01 -08:00
Jarek Kowalski
38fc17bae8 ci: changed cron schedule to run once a week, instead of twice daily 2021-01-24 09:02:15 -08:00
Jarek Kowalski
3cce502017 ci: disabled armhf runner until it's fixed 2021-01-14 20:51:12 -08:00
Jarek Kowalski
fd0e8ed2c9 ci: fix coveralls publishing 2020-12-24 11:24:31 -08:00
Jarek Kowalski
2541f0013c ci: GitHub actions support (#739)
This mimics Travis behavior and also allows making full local release
from a GH owner other than 'kopia' as long as required secrets are
provided.

Supports:

- basic testing on PRs
- publishing kopia RPM, DEB and TAR.GZ release to <owner>/kopia
  (on tags)
- publishing KopiaUI release to <owner>/kopia (on tags)
  or <owner>/kopia-ui-release (on master)
- publishing RPM and APT packages from official kopia/kopia and
  master branch only

Verified on jkowalski/kopia clone:

Tags:
https://github.com/jkowalski/kopia/actions/runs/439735808
https://github.com/jkowalski/kopia/releases/tag/v0.8.0-testing4

Post-submit:
https://github.com/jkowalski/kopia/actions/runs/439720039
https://github.com/jkowalski/kopia-ui-release/releases/tag/v20201222.0.225118

* ci: remove .travis.yml
* testing: skip tests if environment values are empty
* ci: run heavyweight tests on amd64 only
* ci: remove .appveyor.yml
* makefile: removed appveyor travis emulation
2020-12-24 01:11:57 -08:00
Jarek Kowalski
213a557435 ci: run make on GitHub actions (#736) 2020-12-21 17:15:46 -08:00