Commit Graph

117 Commits

Author SHA1 Message Date
Jarek Kowalski
f69424961f chore(ci): upgrade golang to 1.19.2 and linter to 1.50.1 (#2526)
Lack of generics support is blocking various dependency upgrades,
so this unblocks that.

Temporarily disabled `checklocks` linter until it is fixed upstream.
2022-10-28 11:02:47 -07:00
Yanik39
7cb57d4b6b docs(site): example docker-compose.yml (#2253) 2022-10-01 20:19:11 -07:00
Jarek Kowalski
704c07178e fix(ci): fixed git working directory cleanup issue (#2383)
https://github.com/kopia/kopia/runs/8243012442?check_suite_focus=true

This is caused by weird permissions set on files by `go install`
which makes it hard to remove the entire directory using `rm -rf`.

The solution is to clean (local) module cache after `go install` which
makes subsequent `rm -rf` work without a problem.
2022-09-08 07:39:56 -07:00
Jarek Kowalski
afdfad6691 feat(general): move kopia binary in Docker container to /bin/kopia (#2368) 2022-09-04 21:02:15 +00:00
s-h-a-r-d
63257d9537 feat(general): add curl to Docker container image (#2363) 2022-09-03 21:30:13 +00:00
chaitalisg
b61a7627e9 test(general): Add test for snapshot fix command (#2018)
* add robustness test for snapshot fix commands

* temporary change to see if plumbing works

* add robustness test for snapshot fix commands

* temporary change to see if plumbing works

* debug

* revert changes to robustness main_test

* Move recovery tests to a new package

* Revert unwanted changes

* Updated make target

* Temporary change for a pipeline run

* Resolve go package issues

* Remove local path used for debugging

* Debugging

* Debugging

* Add new orchestration script for recovery test

* Add comments

* Add comments

* Revert changes to robustness job script

* Remove unawanted files

* Change permissions for the orchestration script

* Fix typo

* Executable

* Update orchestration script

* Fix linter errors

* Fix linter errors

* Fix linter errors

* fix: linter grouping issue

* Fix linter errors

* Fix linter errors

* Update tests/recovery/blobmanipulator/blobmanipulator.go

Co-authored-by: Aaron Alpar <55999015+aaron-kasten@users.noreply.github.com>

* Implement feedback

* Implement feedback

* Fix linter errors

* Address feedback

* Add maintenance functions

* Added mechanism to interrupt kopia operation during the test run

* Fix linter errors

* Revert debugging changes

* Fix linter errors

* Fix linter errors

* Replace syscall

* Skip test when KOPIA_EXE is not set

* Remove debugging changes

* Add kopia command interruption to simulate corruption

* Check if command is running before killing

* Delete a snapshot before calling maintenance

* Remove nil check for the process

Co-authored-by: Shikhar Mall <small@kopia.io>
Co-authored-by: Aaron Alpar <55999015+aaron-kasten@users.noreply.github.com>
2022-08-09 09:44:18 -07:00
Jarek Kowalski
51dcaa985d chore(ci): upgraded linter to 1.48.0 (#2294)
Mechanically fixed all issues, added `lint-fix` make target.
2022-08-09 06:07:54 +00:00
Jarek Kowalski
21a0e4c1f6 fix(general): install missing ca-certificates in Docker (#2280)
Fixes #2278
2022-08-05 07:24:30 -07:00
Yanik39
93ede2844b feat(general): container image based on Ubuntu (#2252) 2022-07-31 10:03:46 -07:00
Jarek Kowalski
31c5bd94bc build(deps): bumped rclone binary to 1.59.0 (#2221) 2022-07-23 06:50:43 +00:00
Jarek Kowalski
1a82061e49 chore(ci): upgraded linter to 1.47.0, added 15s ReadHeaderTimeout in web server (#2206) 2022-07-18 22:47:36 -07:00
Jarek Kowalski
d25e2ad026 feat(infra): added make kopia-ui-with-local-htmlui-changes (#2062)
This allows building of KopiaUI with local changes make to htmlui/
repository.

In order for this to work you must have 3 repositories checked out
side-by-side:

* https://github.com/kopia/kopia
* https://github.com/kopia/htmlui
* https://github.com/kopia/htmluibuild
2022-06-18 22:49:39 +00:00
Jarek Kowalski
1ffad15069 chore(ci): upgraded linter to 1.46.2 (#2027) 2022-06-10 05:30:07 +00:00
Jarek Kowalski
daa62de3e4 chore(ci): added checklocks static analyzer (#1838)
From https://github.com/google/gvisor/tree/master/tools/checklocks

This will perform static verification that we're using
`sync.Mutex`, `sync.RWMutex` and `atomic` correctly to guard access
to certain fields.

This was mostly just a matter of adding annotations to indicate which
fields are guarded by which mutex.

In a handful of places the code had to be refactored to allow static
analyzer to do its job better or to not be confused by some
constructs.

In one place this actually uncovered a bug where a function was not
releasing a lock properly in an error case.

The check is part of `make lint` but can also be invoked by
`make check-locks`.
2022-03-19 22:42:59 -07:00
Jarek Kowalski
9054fd0dc2 chore(ci): upgraded linter to 1.45 (#1836)
* chore(deps): upgraded linter to 1.45

* fixed linter warning
2022-03-18 22:24:42 -07:00
Ali Dowair
7bc32c485a Fix failed robustness job script if TEST_RC unset (#1780)
If the TEST_RC environment variable is unset, the robustness_job.sh
script exits prematurely due to an unbound variable error, instead of
starting the robustness job without that argument. This commit
evaluates said variable conditionally so that the script doesn't fail.
2022-03-01 13:32:36 -08:00
Ali Dowair
1778131b0c Introduce robustness job pre-test rc (#1770)
Before running a robustness test using tools/robustness_job.sh, it
may be useful to perform some other work or to set some environment
variables to prepare the environment/test repo (for example, to clean
up files, redirect input/output). This commit adds an additional
optional argument TEST_RC, to be consumed as an environment variable.
If TEST_RC is set, then the script sources its value before launching
the robustness job.
2022-02-24 17:36:01 -08:00
Jarek Kowalski
c9d9bf2de2 chore(infra): upgraded linter to 1.44.2 (#1768) 2022-02-22 21:46:02 -08:00
Jarek Kowalski
d3edab594e build(ci): added rclone 1.57.0 to Docker container images (#1718)
* build(ci): added rclone 1.57.0 to Docker container images

/bin/rclone is configured to use config in /app/rclone/rclone.conf
so the user can pass -v ~/.config/rclone:/app/rclone

* pr feedback
2022-02-02 21:06:43 -08:00
Jarek Kowalski
e67f84e0ba chore(general): updated linter to 1.44.0 (#1681) 2022-01-25 21:21:13 -08:00
Jarek Kowalski
c2ab5020dc build(ci): auto-generate change log (#1678) 2022-01-23 15:22:11 -08:00
Jarek Kowalski
32ed220a6c build(lint): enabled gochecknoglobals and tagged existing globals (#1664) 2022-01-15 12:54:56 -08:00
Jarek Kowalski
6438d28094 fix(ci): fix regression when publishing APT repository (#1658) 2022-01-14 08:06:51 -08:00
Jarek Kowalski
a223fb8bdc ci: do not upload armv7 deb packages (kopia-ui) into the apt repository bucket 2021-12-29 10:43:51 -08:00
Jarek Kowalski
daacc6a8f4 htmlui: moved HTML UI to separate repository (#1600)
The source code for htmlui is now in github.com/kopia/htmlui
GitHub Actions will drop compiled builds in github.com/kopia/htmluibuild
where they are now used as a go module dependency.

This greatly simplifies the build and improves the security,
because Kopia will be consuming pre-built htmlui.

This also means kopia can now installed with embedded UI using:
`go install github.com/kopia/kopia@latest`
2021-12-18 12:28:30 -08:00
Jarek Kowalski
8a4ac4dec3 Upgraded linter to 1.43.0 (#1505)
* fixed new gocritic violations
* fixed new 'contextcheck' violations
* fixed 'gosec' warnings
* suppressed ireturn and varnamelen linters
* fixed tenv violations, enabled building robustness tests on arm64
* fixed remaining linux failures
* makefile: fixed 'lint-all' target when running on arm64
* linter: increase deadline
* disable nilnil linter - to be enabled in separate PR
2021-11-11 17:03:11 -08:00
Jarek Kowalski
ca4bf81b2f ci: dependency upgrades (#1504)
* ci: upgraded Node to 16.13.0 LTS

* ci: upgraded Hugo to 0.89.2
2021-11-09 05:50:48 -08:00
Jarek Kowalski
5931425dc4 ci: ignore unsupported architectures when publishing RPM repo 2021-11-04 19:08:39 -07:00
t-rutter
b156ff7606 ci: fix Homebrew upgrade/cleanup warning (#1447) 2021-10-25 20:28:23 -07:00
Jarek Kowalski
191a51b278 ui: fixed snapshotting UNC roots (#1401)
This was caused by additional resolution of path names only done in UI,
which caused \\hostname\share to be treated as relative and resolved
against the home directory.

Fixes #1385
Fixes #1362
2021-10-17 13:25:12 -07:00
CrendKing
e7297d7111 Remove unnecessary subdirectory from Windows archive (#1352)
* Remove unnecessary subdirectory from Windows archive

* Use Scoop extract_dir instead of changing archive structure
2021-10-03 18:10:54 -07:00
Jarek Kowalski
928150fe6b linter: upgrade to 1.42.1 (#1292) 2021-09-14 19:11:39 -07:00
Jarek Kowalski
8b2b91f9f9 content: fixed repo upgrade version (#1286)
* content: fixed repo upgrade version

Previously upgrade would enable epoch manager and index v2 but would
not set the version of the format itself. Everything worked fine
but it would not protect from old kopia opening the repository.

* ci: added compatibility test that uses real 0.8 and current binaries
2021-09-10 22:51:51 -07:00
Jarek Kowalski
135c307c6c ci: fixed rclone installation 2021-09-08 20:26:23 -07:00
Jarek Kowalski
7e68d8e4c1 Consolidated format version flags (#1284) 2021-09-08 18:44:03 -07:00
Jarek Kowalski
6c83cc6a21 ui: upgraded all NPM dependencies for htmlui/ and app/ (#1278) 2021-09-06 17:04:30 -07:00
Jarek Kowalski
59b5a44ca6 Makefile: refactored build tools installation (#1274)
Instead of using hardcore Makefile scripts and external tools, created a
specialized Go tool that combines curl, sha256sum, gunzip, tar, unzip
and automatically figures out the correct URL to download based on
the current GOOS/GOARCH combination.
2021-09-01 20:50:14 -07:00
Jarek Kowalski
9e182f131a linter: upgraded to 1.42.0 (#1246) 2021-08-20 18:26:45 -07:00
Jarek Kowalski
42e5661bf4 ci: added license check using github.com/frapposelli/wwhrd (#1200)
* ci: added license check using github.com/frapposelli/wwhrd

* ci: also check NPM licenses

* readme: removed FOSSA badge

* Makefile: install NPM dependencies before license check
2021-07-17 09:58:25 -07:00
Jarek Kowalski
f702e65a31 deps: removed dependency on github.com/minio/minio/pkg, replaced with command line tool (#1185) 2021-07-10 10:15:31 -07:00
Jarek Kowalski
61af6399c8 cli: fixed profiling flags (#1181) 2021-07-09 21:35:00 -07:00
Jarek Kowalski
9e059a1277 upgraded linter to 1.41.0 (#1144) 2021-06-16 19:44:55 -07:00
Jarek Kowalski
30ca3e2e6c Upgraded linter to 1.40.1 (#1072)
* tools: upgraded linter to 1.40.1

* lint: fixed nolintlint vionlations

* lint: disabled tagliatele linter

* lint: fixed remaining warnings
2021-05-15 12:12:34 -07:00
Steve Joachim
d2b816934c robustness: add tests for kopia server repos (#1029)
* Add error handling for robustness Checker
* robustness engine updates for concurrency
* add client-server utils for multiclient robustness
* create client package for robustness tests
* create multi-client robustness test framework
* multi-client robustness test definitions
* update robustness test runner scripts
* Address unparam lint errors

Co-authored-by: Julio Lopez <julio+gh@kasten.io>
2021-04-29 21:45:13 -07:00
Denis Fondras
3f29141c33 User must install npm manually on OpenBSD (#995)
* user must install npm manually with OpenBSD

* simplify and add freebsd rules
2021-04-20 19:51:29 -07:00
Jarek Kowalski
922bc7cbc8 ci: refactored credentials handling (#987)
This strengthens credential handling after our signing keys may have
been leaked in the [codecov.io breach](https://about.codecov.io/security-update/)

* pass only minimal credentials to each build step to avoid
  exposing sensitive tokens to tools that don't need them
  (like code coverage)
* removed encrypted credential files and replaced with environment-based
* allow full ci/cd including publishing artifacts from forks
* regenerated all passwords, tokens and service accounts
* do not install Google Cloud SDK on GHA - it's already there
* moved RPM signing to 'Stage And Publish Artifacts' phase
* generated new GPG signing key

See https://kopia.discourse.group/t/important-impact-of-codecov-io-security-issue-on-kopia-build-pipeline/377
2021-04-16 08:17:13 -07:00
Jarek Kowalski
b59a1131a9 ci: publish scoop and homebrew from tags 2021-04-11 12:50:40 -07:00
Jarek Kowalski
601ea66fd9 docker: fix executable permissions 2021-04-09 23:13:35 -07:00
Jarek Kowalski
76ce309589 site: upgraded hugo to 0.82.0 (#938) 2021-04-03 23:11:56 -07:00
Jarek Kowalski
21af8914b1 ci: fixed windows signing and scoop url 2021-03-23 09:07:16 -07:00