Commit Graph

75 Commits

Author SHA1 Message Date
Jarek Kowalski
30ca3e2e6c Upgraded linter to 1.40.1 (#1072)
* tools: upgraded linter to 1.40.1

* lint: fixed nolintlint vionlations

* lint: disabled tagliatele linter

* lint: fixed remaining warnings
2021-05-15 12:12:34 -07:00
Steve Joachim
d2b816934c robustness: add tests for kopia server repos (#1029)
* Add error handling for robustness Checker
* robustness engine updates for concurrency
* add client-server utils for multiclient robustness
* create client package for robustness tests
* create multi-client robustness test framework
* multi-client robustness test definitions
* update robustness test runner scripts
* Address unparam lint errors

Co-authored-by: Julio Lopez <julio+gh@kasten.io>
2021-04-29 21:45:13 -07:00
Denis Fondras
3f29141c33 User must install npm manually on OpenBSD (#995)
* user must install npm manually with OpenBSD

* simplify and add freebsd rules
2021-04-20 19:51:29 -07:00
Jarek Kowalski
922bc7cbc8 ci: refactored credentials handling (#987)
This strengthens credential handling after our signing keys may have
been leaked in the [codecov.io breach](https://about.codecov.io/security-update/)

* pass only minimal credentials to each build step to avoid
  exposing sensitive tokens to tools that don't need them
  (like code coverage)
* removed encrypted credential files and replaced with environment-based
* allow full ci/cd including publishing artifacts from forks
* regenerated all passwords, tokens and service accounts
* do not install Google Cloud SDK on GHA - it's already there
* moved RPM signing to 'Stage And Publish Artifacts' phase
* generated new GPG signing key

See https://kopia.discourse.group/t/important-impact-of-codecov-io-security-issue-on-kopia-build-pipeline/377
2021-04-16 08:17:13 -07:00
Jarek Kowalski
b59a1131a9 ci: publish scoop and homebrew from tags 2021-04-11 12:50:40 -07:00
Jarek Kowalski
601ea66fd9 docker: fix executable permissions 2021-04-09 23:13:35 -07:00
Jarek Kowalski
76ce309589 site: upgraded hugo to 0.82.0 (#938) 2021-04-03 23:11:56 -07:00
Jarek Kowalski
21af8914b1 ci: fixed windows signing and scoop url 2021-03-23 09:07:16 -07:00
Jarek Kowalski
4d77bd109e ci: added publishing to test-builds Scoop bucket 2021-03-23 08:01:28 -07:00
Jarek Kowalski
c0db53da11 ci: reduce number of old versions in APT/RPM unstable channels 2021-03-23 06:43:38 -07:00
Jarek Kowalski
16282b5b61 Makefile refactoring + publish scoop package (#907)
* Makefile: refactoring

- added signing of windows exe in a zip package
- removed Windows build from goreleaser
- removed homebrew from goreleaser
- enabled incremental builds

* ci: publish scoop package
2021-03-22 22:39:24 -07:00
Jarek Kowalski
e8d14af0db ci: publish homebrew test builds to kopia/test-builds TAP 2021-03-21 20:57:46 -07:00
Jarek Kowalski
2e5a3ad3a3 ci: build and publish Homebrew package 2021-03-21 18:41:08 -07:00
Jarek Kowalski
bcfba334da ci: stop using goreleaser to produce macOS binaries, make universal binary as separate .tar.gz 2021-03-21 17:58:04 -07:00
Jarek Kowalski
335ff79e78 CI refactoring (#903)
* ci: publish additional binaries as artifacts

* ci: install {htmlui,app}/node_modules as part of ci-setup

* ci: added retry to improve robustness of builds

* ci: continue-on-error for non-pull-requests

* ci: moved publish-packages to Stage And Publish Artifacts phase
2021-03-21 15:17:39 -07:00
Jarek Kowalski
342919892e ci: install signing tools just before building the binary with UI 2021-03-20 12:09:57 -07:00
Jarek Kowalski
3a94c16678 Dockerfile: switched to distroless, specified defaults environment variables for containerized kopia (#897)
* Dockerfile: specified reasonable defaults options for containerized kopia

* addressed pr comments, switched to gcr.io/distroless/static:nonroot

distroless has no executable code, so this requires KOPIA_PASSWORD
to always be provided via env, b/c distroless does not have
/bin/stty to disable TTY echo (we should not require that, BTW)

* site: added docker image documentation
2021-03-19 21:54:48 -07:00
Jarek Kowalski
3b6cf5cc7b docker: publish docker images to docker hub (#896)
docker: push dockerhub tags

* major.minor - tracks latest major.minor release (except major==0)
* latest - tracks official stable releases
* testing - tracks official stable releases or pre-releases (beta, rc)
* unstable - tracks nightly/unstable builds
2021-03-19 12:51:24 -07:00
Jarek Kowalski
36c65855ee ci: prevent locking macos keychain if the build takes more than 5 minutes 2021-03-14 13:42:59 -07:00
Jarek Kowalski
b421af9b46 one more attempt at fixing macOS signing hangs 2021-03-10 23:45:12 -08:00
Jarek Kowalski
3c696c4f03 one more attempt at fixing macOS signing hangs 2021-03-10 23:25:01 -08:00
Jarek Kowalski
01ee5980db Makefile: set default-keychain when signing macOS binaries 2021-03-10 20:34:28 -08:00
Jarek Kowalski
5928df3498 Makefile: build fat macOS binary (amd64 and arm64) (#866)
This will be embedded in KopiaUI which is built on macOS runner.
2021-03-04 19:35:26 -08:00
Jarek Kowalski
01cd9ba134 Makefile: run goreleaser only on linux/amd64 in non-PR mode (#861) 2021-03-02 08:29:37 -08:00
Jarek Kowalski
8d43715acf ci: upgrade goreleaser to 0.158.0 2021-03-01 18:03:34 -08:00
Jarek Kowalski
ff158ea763 goreleaser: updated to 0.157.0 to produce darwin/arm64 build (#855) 2021-02-27 12:10:20 -08:00
Julio López
7bafe51dcc Replace go-bindata with //go:embed (#844)
* Replace htmlui_fallback.go with go:embed
* Replace go-bindata generated UI with go:embed
* Update site Go version to 1.16
* Update BUILD.md to reflect workflow with go:embed
2021-02-23 01:09:40 -08:00
Jarek Kowalski
e2b9a81ac3 Major CI/CD refactoring and re-added support for ARM/ARM64 runners (#849)
* ci: refactored CI/CD logic & Makefile

- removed all travis CI emulation environment variables and replaced with:

CI_TAG=<empty>|tag
IS_PULL_REQUEST=false|true

- refactored all OS and architecture-specific decisions to use around standard GOOS/GOARCH values instead of uname/OS
- re-added self-hosted runner for ARMHF (3 replicas)
- added brand new self-hosted runner for ARM64 (3 replicas)
- disabled attempts to publish and sign on forks
- improved integration test log output to better see timings and sub-tests
- print longest tests (unit tests and integration) after each run
- verified that all configurations build successfully on a clone (jkowalski/kopia)
- run make setup in parallel

* testing: fixed tests on ARM and ARM64

- fixed ARM-specific alignment issue
- cleaned up test logging
- fixed huge params warning threshold because it was tripping on ARM.
- reduced test complexity to make them fit in 15 minutes
2021-02-23 00:52:54 -08:00
Jarek Kowalski
d9b58fb6e2 linter: upgraded to 1.37.1 (#846) 2021-02-20 11:34:14 -08:00
Jarek Kowalski
4bf42e337d fix long filenames on Windows (#822)
* windows: fixed handling of long filenames
2021-02-12 09:09:42 -08:00
Jarek Kowalski
2a58166c43 lint: upgraded to 1.36 which fixes flakes (#801)
* lint: upgraded to 1.36 which fixes flakes

* ci: removed separate lint action - normal lint is fast enough and less flaky

* ci: also deleted linter_version_test.go
2021-01-26 19:24:30 -08:00
Jarek Kowalski
c3e6aebfb6 rclone: provider improvements (#795)
switched local passwd hash to APR1 to support older rclones
install well-known version of rclone for testing
2021-01-25 14:53:55 -08:00
Jarek Kowalski
6a63c207a4 Upgraded Node and all NPM dependencies to latest versions (#793)
* htmlui: upgraded all NPM dependencies to latest versions

* app: upgraded all NPM dependencies to latest versions

* tools: upgraded node to 14.15.4 LTS

* app: fixed deprecated methods
2021-01-23 18:40:50 -08:00
Jarek Kowalski
1f3b8d4da4 upgrade linter to 1.35 (#786)
* lint: added test that enforces Makefile and GH action linter versions are in sync
* workaround for linter gomnd problem - https://github.com/golangci/golangci-lint/issues/1653
2021-01-16 18:21:16 -08:00
Jarek Kowalski
2079770654 ci: don't allocate tty when running docker, since it's not supported on GH 2020-12-24 10:03:42 -08:00
Jarek Kowalski
97d630cc65 CI tweaks (#742)
* ci: run full release only on amd64

* ci: logging for apt-publish.sh

* ci: stop building website in Makefile as it's already handled by Netlify
2020-12-24 09:00:30 -08:00
Jarek Kowalski
2541f0013c ci: GitHub actions support (#739)
This mimics Travis behavior and also allows making full local release
from a GH owner other than 'kopia' as long as required secrets are
provided.

Supports:

- basic testing on PRs
- publishing kopia RPM, DEB and TAR.GZ release to <owner>/kopia
  (on tags)
- publishing KopiaUI release to <owner>/kopia (on tags)
  or <owner>/kopia-ui-release (on master)
- publishing RPM and APT packages from official kopia/kopia and
  master branch only

Verified on jkowalski/kopia clone:

Tags:
https://github.com/jkowalski/kopia/actions/runs/439735808
https://github.com/jkowalski/kopia/releases/tag/v0.8.0-testing4

Post-submit:
https://github.com/jkowalski/kopia/actions/runs/439720039
https://github.com/jkowalski/kopia-ui-release/releases/tag/v20201222.0.225118

* ci: remove .travis.yml
* testing: skip tests if environment values are empty
* ci: run heavyweight tests on amd64 only
* ci: remove .appveyor.yml
* makefile: removed appveyor travis emulation
2020-12-24 01:11:57 -08:00
Jarek Kowalski
e03971fc59 Upgraded linter to v1.33.0 (#734)
* linter: upgraded to 1.33, disabled some linters

* lint: fixed 'errorlint' errors

This ensures that all error comparisons use errors.Is() or errors.As().
We will be wrapping more errors going forward so it's important that
error checks are not strict everywhere.

Verified that there are no exceptions for errorlint linter which
guarantees that.

* lint: fixed or suppressed wrapcheck errors

* lint: nolintlint and misc cleanups

Co-authored-by: Julio López <julio+gh@kasten.io>
2020-12-21 22:39:22 -08:00
Jarek Kowalski
213a557435 ci: run make on GitHub actions (#736) 2020-12-21 17:15:46 -08:00
Julio López
4eb8f71567 Helper script to run robustness job (#721) 2020-12-15 20:32:19 -08:00
Jarek Kowalski
7d21eaa6c1 Cleanup old RPM and DEB packages from 'unstable' repositories (#677)
* tools: cleanup old RPM packages in the 'unstable' track automatically

* tools: remove old DEB automatically from the 'unstable' channel
2020-10-10 14:43:32 -07:00
Jarek Kowalski
a01bfde39a Makefile: plug in gotestsum for better test output (#652) 2020-09-29 21:14:20 -07:00
Jarek Kowalski
44968af7b6 release: fixed RPM signing by moving all of this to a single invocation of sign.sh 2020-09-13 17:07:49 -07:00
Jarek Kowalski
f3fc2ea3b7 tools: misc fixes to RPM publishing 2020-09-10 22:11:33 -07:00
Jarek Kowalski
5494746e67 tools: clean up repomd directory before updating 2020-09-10 18:59:52 -07:00
Jarek Kowalski
4ef314bee5 Added RPM repository (#600)
* goreleaser: added signatures to RPM binaries

Currently goreleaser does not support it, so we're overriding
signing script and signing all RPMs that it produces.

Also changed goreleaser parameters to only publish binaries
when running on linux/amd64.

* build: added automatic publishing of RPMs to a YUM repository

Also fixed RPM file names to match local conventions.
2020-09-09 23:18:20 -07:00
Jarek Kowalski
3d24fd8875 apt-publish: publish dummy i386 release, even though we don't make packages for it as Ubuntu on amd64 is sometimes confused and tries downloading i386 anyway (#574) 2020-08-31 17:22:17 -07:00
Jarek Kowalski
f0751233a0 apt: update caching parameters after push 2020-08-14 20:05:10 -07:00
Jarek Kowalski
3f71aef13d apt: fixed matching unstable version numbers 2020-08-14 15:34:05 -07:00
Jarek Kowalski
b381d31eb8 tools: added apt-publish tool that push to APT repository
The repository is in GCS and the script will automatically copy
DEB files from dist/ to the proper locations and will regenerate
APT package index.
2020-08-14 14:08:10 -07:00