Commit Graph
121 Commits
Author SHA1 Message Date
Julio Lopez aabf183cca Revert "build(tests): fix mc installation (#2910)" (#2912)
This reverts commit cb2d117.

The issue was fixed upstream in minio/mc#4541

Fixes #2908
2023-04-10 16:00:44 +00:00
Julio Lopez cb2d117533 build(tests): fix mc installation (#2910) 2023-04-10 01:36:53 +00:00
YFdyh000 fed807f20e fix(repository): fixed connect broken since rclone v1.61.0 (#2853)
* fix(repository): fixed connect broken since rclone v1.61.0

https://github.com/rclone/rclone/commit/4444d2d102cc187f8cc4907f3e06611576c10a36#diff-3bfe93b28218b8e93a7480a5684d21ac566795a0fa8e540ec1bc6cbe7028439eR295

* build(deps): bumped rclone binary to 1.62.2
2023-03-28 01:55:13 +00:00
Jarek Kowalski 939152cfda chore(ci): upgraded linter to 1.51.1, minor style fixes (#2748) 2023-02-09 06:47:36 +00:00
Jarek Kowalski f69424961f chore(ci): upgrade golang to 1.19.2 and linter to 1.50.1 (#2526)
Lack of generics support is blocking various dependency upgrades,
so this unblocks that.

Temporarily disabled `checklocks` linter until it is fixed upstream.
2022-10-28 11:02:47 -07:00
Yanik39 7cb57d4b6b docs(site): example docker-compose.yml (#2253) 2022-10-01 20:19:11 -07:00
Jarek Kowalski 704c07178e fix(ci): fixed git working directory cleanup issue (#2383)
https://github.com/kopia/kopia/runs/8243012442?check_suite_focus=true

This is caused by weird permissions set on files by `go install`
which makes it hard to remove the entire directory using `rm -rf`.

The solution is to clean (local) module cache after `go install` which
makes subsequent `rm -rf` work without a problem.
2022-09-08 07:39:56 -07:00
Jarek Kowalski afdfad6691 feat(general): move kopia binary in Docker container to /bin/kopia (#2368) 2022-09-04 21:02:15 +00:00
s-h-a-r-d 63257d9537 feat(general): add curl to Docker container image (#2363) 2022-09-03 21:30:13 +00:00
b61a7627e9 test(general): Add test for snapshot fix command (#2018)
* add robustness test for snapshot fix commands

* temporary change to see if plumbing works

* add robustness test for snapshot fix commands

* temporary change to see if plumbing works

* debug

* revert changes to robustness main_test

* Move recovery tests to a new package

* Revert unwanted changes

* Updated make target

* Temporary change for a pipeline run

* Resolve go package issues

* Remove local path used for debugging

* Debugging

* Debugging

* Add new orchestration script for recovery test

* Add comments

* Add comments

* Revert changes to robustness job script

* Remove unawanted files

* Change permissions for the orchestration script

* Fix typo

* Executable

* Update orchestration script

* Fix linter errors

* Fix linter errors

* Fix linter errors

* fix: linter grouping issue

* Fix linter errors

* Fix linter errors

* Update tests/recovery/blobmanipulator/blobmanipulator.go

Co-authored-by: Aaron Alpar <55999015+aaron-kasten@users.noreply.github.com>

* Implement feedback

* Implement feedback

* Fix linter errors

* Address feedback

* Add maintenance functions

* Added mechanism to interrupt kopia operation during the test run

* Fix linter errors

* Revert debugging changes

* Fix linter errors

* Fix linter errors

* Replace syscall

* Skip test when KOPIA_EXE is not set

* Remove debugging changes

* Add kopia command interruption to simulate corruption

* Check if command is running before killing

* Delete a snapshot before calling maintenance

* Remove nil check for the process

Co-authored-by: Shikhar Mall <small@kopia.io>
Co-authored-by: Aaron Alpar <55999015+aaron-kasten@users.noreply.github.com>
2022-08-09 09:44:18 -07:00
Jarek Kowalski 51dcaa985d chore(ci): upgraded linter to 1.48.0 (#2294)
Mechanically fixed all issues, added `lint-fix` make target.
2022-08-09 06:07:54 +00:00
Jarek Kowalski 21a0e4c1f6 fix(general): install missing ca-certificates in Docker (#2280)
Fixes #2278
2022-08-05 07:24:30 -07:00
Yanik39 93ede2844b feat(general): container image based on Ubuntu (#2252) 2022-07-31 10:03:46 -07:00
Jarek Kowalski 31c5bd94bc build(deps): bumped rclone binary to 1.59.0 (#2221) 2022-07-23 06:50:43 +00:00
Jarek Kowalski 1a82061e49 chore(ci): upgraded linter to 1.47.0, added 15s ReadHeaderTimeout in web server (#2206) 2022-07-18 22:47:36 -07:00
Jarek Kowalski d25e2ad026 feat(infra): added make kopia-ui-with-local-htmlui-changes (#2062)
This allows building of KopiaUI with local changes make to htmlui/
repository.

In order for this to work you must have 3 repositories checked out
side-by-side:

* https://github.com/kopia/kopia
* https://github.com/kopia/htmlui
* https://github.com/kopia/htmluibuild
2022-06-18 22:49:39 +00:00
Jarek Kowalski 1ffad15069 chore(ci): upgraded linter to 1.46.2 (#2027) 2022-06-10 05:30:07 +00:00
Jarek Kowalski daa62de3e4 chore(ci): added checklocks static analyzer (#1838)
From https://github.com/google/gvisor/tree/master/tools/checklocks

This will perform static verification that we're using
`sync.Mutex`, `sync.RWMutex` and `atomic` correctly to guard access
to certain fields.

This was mostly just a matter of adding annotations to indicate which
fields are guarded by which mutex.

In a handful of places the code had to be refactored to allow static
analyzer to do its job better or to not be confused by some
constructs.

In one place this actually uncovered a bug where a function was not
releasing a lock properly in an error case.

The check is part of `make lint` but can also be invoked by
`make check-locks`.
2022-03-19 22:42:59 -07:00
Jarek Kowalski 9054fd0dc2 chore(ci): upgraded linter to 1.45 (#1836)
* chore(deps): upgraded linter to 1.45

* fixed linter warning
2022-03-18 22:24:42 -07:00
Ali Dowair 7bc32c485a Fix failed robustness job script if TEST_RC unset (#1780)
If the TEST_RC environment variable is unset, the robustness_job.sh
script exits prematurely due to an unbound variable error, instead of
starting the robustness job without that argument. This commit
evaluates said variable conditionally so that the script doesn't fail.
2022-03-01 13:32:36 -08:00
Ali Dowair 1778131b0c Introduce robustness job pre-test rc (#1770)
Before running a robustness test using tools/robustness_job.sh, it
may be useful to perform some other work or to set some environment
variables to prepare the environment/test repo (for example, to clean
up files, redirect input/output). This commit adds an additional
optional argument TEST_RC, to be consumed as an environment variable.
If TEST_RC is set, then the script sources its value before launching
the robustness job.
2022-02-24 17:36:01 -08:00
Jarek Kowalski c9d9bf2de2 chore(infra): upgraded linter to 1.44.2 (#1768) 2022-02-22 21:46:02 -08:00
Jarek Kowalski d3edab594e build(ci): added rclone 1.57.0 to Docker container images (#1718)
* build(ci): added rclone 1.57.0 to Docker container images

/bin/rclone is configured to use config in /app/rclone/rclone.conf
so the user can pass -v ~/.config/rclone:/app/rclone

* pr feedback
2022-02-02 21:06:43 -08:00
Jarek Kowalski e67f84e0ba chore(general): updated linter to 1.44.0 (#1681) 2022-01-25 21:21:13 -08:00
Jarek Kowalski c2ab5020dc build(ci): auto-generate change log (#1678) 2022-01-23 15:22:11 -08:00
Jarek Kowalski 32ed220a6c build(lint): enabled gochecknoglobals and tagged existing globals (#1664) 2022-01-15 12:54:56 -08:00
Jarek Kowalski 6438d28094 fix(ci): fix regression when publishing APT repository (#1658) 2022-01-14 08:06:51 -08:00
Jarek Kowalski a223fb8bdc ci: do not upload armv7 deb packages (kopia-ui) into the apt repository bucket 2021-12-29 10:43:51 -08:00
Jarek Kowalski daacc6a8f4 htmlui: moved HTML UI to separate repository (#1600)
The source code for htmlui is now in github.com/kopia/htmlui
GitHub Actions will drop compiled builds in github.com/kopia/htmluibuild
where they are now used as a go module dependency.

This greatly simplifies the build and improves the security,
because Kopia will be consuming pre-built htmlui.

This also means kopia can now installed with embedded UI using:
`go install github.com/kopia/kopia@latest`
2021-12-18 12:28:30 -08:00
Jarek Kowalski 8a4ac4dec3 Upgraded linter to 1.43.0 (#1505)
* fixed new gocritic violations
* fixed new 'contextcheck' violations
* fixed 'gosec' warnings
* suppressed ireturn and varnamelen linters
* fixed tenv violations, enabled building robustness tests on arm64
* fixed remaining linux failures
* makefile: fixed 'lint-all' target when running on arm64
* linter: increase deadline
* disable nilnil linter - to be enabled in separate PR
2021-11-11 17:03:11 -08:00
Jarek Kowalski ca4bf81b2f ci: dependency upgrades (#1504)
* ci: upgraded Node to 16.13.0 LTS

* ci: upgraded Hugo to 0.89.2
2021-11-09 05:50:48 -08:00
Jarek Kowalski 5931425dc4 ci: ignore unsupported architectures when publishing RPM repo 2021-11-04 19:08:39 -07:00
t-rutter b156ff7606 ci: fix Homebrew upgrade/cleanup warning (#1447) 2021-10-25 20:28:23 -07:00
Jarek Kowalski 191a51b278 ui: fixed snapshotting UNC roots (#1401)
This was caused by additional resolution of path names only done in UI,
which caused \\hostname\share to be treated as relative and resolved
against the home directory.

Fixes #1385
Fixes #1362
2021-10-17 13:25:12 -07:00
CrendKing e7297d7111 Remove unnecessary subdirectory from Windows archive (#1352)
* Remove unnecessary subdirectory from Windows archive

* Use Scoop extract_dir instead of changing archive structure
2021-10-03 18:10:54 -07:00
Jarek Kowalski 928150fe6b linter: upgrade to 1.42.1 (#1292) 2021-09-14 19:11:39 -07:00
Jarek Kowalski 8b2b91f9f9 content: fixed repo upgrade version (#1286)
* content: fixed repo upgrade version

Previously upgrade would enable epoch manager and index v2 but would
not set the version of the format itself. Everything worked fine
but it would not protect from old kopia opening the repository.

* ci: added compatibility test that uses real 0.8 and current binaries
2021-09-10 22:51:51 -07:00
Jarek Kowalski 135c307c6c ci: fixed rclone installation 2021-09-08 20:26:23 -07:00
Jarek Kowalski 7e68d8e4c1 Consolidated format version flags (#1284) 2021-09-08 18:44:03 -07:00
Jarek Kowalski 6c83cc6a21 ui: upgraded all NPM dependencies for htmlui/ and app/ (#1278) 2021-09-06 17:04:30 -07:00
Jarek Kowalski 59b5a44ca6 Makefile: refactored build tools installation (#1274)
Instead of using hardcore Makefile scripts and external tools, created a
specialized Go tool that combines curl, sha256sum, gunzip, tar, unzip
and automatically figures out the correct URL to download based on
the current GOOS/GOARCH combination.
2021-09-01 20:50:14 -07:00
Jarek Kowalski 9e182f131a linter: upgraded to 1.42.0 (#1246) 2021-08-20 18:26:45 -07:00
Jarek Kowalski 42e5661bf4 ci: added license check using github.com/frapposelli/wwhrd (#1200)
* ci: added license check using github.com/frapposelli/wwhrd

* ci: also check NPM licenses

* readme: removed FOSSA badge

* Makefile: install NPM dependencies before license check
2021-07-17 09:58:25 -07:00
Jarek Kowalski f702e65a31 deps: removed dependency on github.com/minio/minio/pkg, replaced with command line tool (#1185) 2021-07-10 10:15:31 -07:00
Jarek Kowalski 61af6399c8 cli: fixed profiling flags (#1181) 2021-07-09 21:35:00 -07:00
Jarek Kowalski 9e059a1277 upgraded linter to 1.41.0 (#1144) 2021-06-16 19:44:55 -07:00
Jarek Kowalski 30ca3e2e6c Upgraded linter to 1.40.1 (#1072)
* tools: upgraded linter to 1.40.1

* lint: fixed nolintlint vionlations

* lint: disabled tagliatele linter

* lint: fixed remaining warnings
2021-05-15 12:12:34 -07:00
Steve JoachimandJulio Lopez d2b816934c robustness: add tests for kopia server repos (#1029)
* Add error handling for robustness Checker
* robustness engine updates for concurrency
* add client-server utils for multiclient robustness
* create client package for robustness tests
* create multi-client robustness test framework
* multi-client robustness test definitions
* update robustness test runner scripts
* Address unparam lint errors

Co-authored-by: Julio Lopez <julio+gh@kasten.io>
2021-04-29 21:45:13 -07:00
Denis Fondras 3f29141c33 User must install npm manually on OpenBSD (#995)
* user must install npm manually with OpenBSD

* simplify and add freebsd rules
2021-04-20 19:51:29 -07:00
Jarek Kowalski 922bc7cbc8 ci: refactored credentials handling (#987)
This strengthens credential handling after our signing keys may have
been leaked in the [codecov.io breach](https://about.codecov.io/security-update/)

* pass only minimal credentials to each build step to avoid
  exposing sensitive tokens to tools that don't need them
  (like code coverage)
* removed encrypted credential files and replaced with environment-based
* allow full ci/cd including publishing artifacts from forks
* regenerated all passwords, tokens and service accounts
* do not install Google Cloud SDK on GHA - it's already there
* moved RPM signing to 'Stage And Publish Artifacts' phase
* generated new GPG signing key

See https://kopia.discourse.group/t/important-impact-of-codecov-io-security-issue-on-kopia-build-pipeline/377
2021-04-16 08:17:13 -07:00