Commit Graph
2200 Commits
Author SHA1 Message Date
dependabot[bot] 5932a660e6 build(deps): bump github.com/minio/minio-go/v7 from 7.0.21 to 7.0.23 (#1794)
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.21 to 7.0.23.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.21...v7.0.23)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-02 19:31:17 -08:00
dependabot[bot] 8f778ce951 build(deps): bump github.com/chromedp/chromedp from 0.7.6 to 0.7.8 (#1793)
Bumps [github.com/chromedp/chromedp](https://github.com/chromedp/chromedp) from 0.7.6 to 0.7.8.
- [Release notes](https://github.com/chromedp/chromedp/releases)
- [Commits](https://github.com/chromedp/chromedp/compare/v0.7.6...v0.7.8)

---
updated-dependencies:
- dependency-name: github.com/chromedp/chromedp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 22:25:16 -08:00
dependabot[bot] 1617d51315 build(deps): bump github.com/klauspost/compress from 1.14.2 to 1.14.4 (#1792)
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.14.2 to 1.14.4.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.14.2...v1.14.4)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 22:03:23 -08:00
dependabot[bot] 1d50a343f7 build(deps): bump nanoid from 3.1.30 to 3.3.1 in /app (#1790)
Bumps [nanoid](https://github.com/ai/nanoid) from 3.1.30 to 3.3.1.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/3.1.30...3.3.1)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 21:26:00 -08:00
dependabot[bot] 7d5abf87f4 build(deps): bump electron-updater from 4.6.1 to 4.6.5 in /app (#1781)
Bumps [electron-updater](https://github.com/electron-userland/electron-builder/tree/HEAD/packages/electron-updater) from 4.6.1 to 4.6.5.
- [Release notes](https://github.com/electron-userland/electron-builder/releases)
- [Changelog](https://github.com/electron-userland/electron-builder/blob/master/packages/electron-updater/CHANGELOG.md)
- [Commits](https://github.com/electron-userland/electron-builder/commits/electron-updater@4.6.5/packages/electron-updater)

---
updated-dependencies:
- dependency-name: electron-updater
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 21:23:50 -08:00
dependabot[bot] dc52d624a4 build(deps): bump go.uber.org/multierr from 1.7.0 to 1.8.0 (#1783)
Bumps [go.uber.org/multierr](https://github.com/uber-go/multierr) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/uber-go/multierr/releases)
- [Changelog](https://github.com/uber-go/multierr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/multierr/compare/v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: go.uber.org/multierr
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 21:23:17 -08:00
dependabot[bot] c7ccf4c79d build(deps): bump actions/checkout from 2 to 3 (#1782)
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 21:22:55 -08:00
dependabot[bot] 1005ba0ec4 build(deps): bump github.com/golang-jwt/jwt/v4 from 4.2.0 to 4.3.0 (#1787)
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.2.0 to 4.3.0.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.2.0...v4.3.0)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 21:21:29 -08:00
dependabot[bot] ff8d64c1a9 build(deps): bump electron-log from 4.4.5 to 4.4.6 in /app (#1788)
Bumps [electron-log](https://github.com/megahertz/electron-log) from 4.4.5 to 4.4.6.
- [Release notes](https://github.com/megahertz/electron-log/releases)
- [Changelog](https://github.com/megahertz/electron-log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/megahertz/electron-log/compare/v4.4.5...v4.4.6)

---
updated-dependencies:
- dependency-name: electron-log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 21:21:11 -08:00
dependabot[bot] de0f7c4dfb build(deps): bump cloud.google.com/go/storage from 1.19.0 to 1.21.0 (#1789)
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.19.0 to 1.21.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.19.0...spanner/v1.21.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-01 21:20:58 -08:00
Ali Dowair 7bc32c485a Fix failed robustness job script if TEST_RC unset (#1780)
If the TEST_RC environment variable is unset, the robustness_job.sh
script exits prematurely due to an unbound variable error, instead of
starting the robustness job without that argument. This commit
evaluates said variable conditionally so that the script doesn't fail.
2022-03-01 13:32:36 -08:00
Ali Dowair 813927fe7e Add PR Checklist to contribution guidelines (#1775)
To save reviewers' time and CI/CD cloud spend (if any), this commit adds
a short checklist to the contribution guidelines, outlining some checks
developers can run locally against their code before creating/updating
a pull request on Github.
2022-02-25 20:55:44 -08:00
Ali Dowair 1778131b0c Introduce robustness job pre-test rc (#1770)
Before running a robustness test using tools/robustness_job.sh, it
may be useful to perform some other work or to set some environment
variables to prepare the environment/test repo (for example, to clean
up files, redirect input/output). This commit adds an additional
optional argument TEST_RC, to be consumed as an environment variable.
If TEST_RC is set, then the script sources its value before launching
the robustness job.
2022-02-24 17:36:01 -08:00
Shikhar MallandShikhar Mall 58c28f2205 feat(general): Upgrade lock retries & monitoring during open & write sessions (#1758)
* background upgrade lock monitor

* retry lock forever on connect

* pr feedback

* remove time computations under read lock for efficiency

* extend the unit test to cover lock monitoring with a controlled time function

* more cleanup

Co-authored-by: Shikhar Mall <small@kopia.io>
2022-02-24 15:00:02 -08:00
Jarek Kowalski 9d63e56bb9 feat(cli): improved formatting of 'policy show' outputs (#1767) 2022-02-22 22:21:48 -08:00
Jarek Kowalski c9d9bf2de2 chore(infra): upgraded linter to 1.44.2 (#1768) 2022-02-22 21:46:02 -08:00
Jarek Kowalski 6d29b2a082 feat(cli): added snapshot list flags: --storage-stats and --reverse (#1766)
For #1722
Related #1755
2022-02-21 20:24:13 -08:00
Jarek Kowalski f1d3130351 refactor(repository): expose ContentInfo() on repo.Repository (#1765) 2022-02-21 14:38:59 -08:00
Jarek Kowalski f5cee5ae42 refactor(snapshots): refactored TreeWalker to use workshare (#1762)
Also cleaned up the API and added unit tests.
2022-02-20 16:58:53 -08:00
Jarek Kowalski bf38c7b788 fix(ci): remove changelog type filter (#1761) 2022-02-20 11:01:33 -08:00
xkxx a379302f36 docs(providers): Instructions for how to use the Google Drive backend. (#1760) 2022-02-19 12:04:25 -08:00
Jarek Kowalski 4bc69d05ef feat(snapshots): big upload performance improvements by parallelizing directory traversal (#1752)
* feat(general): added internal/workshare package

This introduces work sharing utility useful when walking trees of
things (such as filesystem), which allows N threads/goroutines to be
used.

Whenever a routine is visiting its children, it can share some of that
work with another idle goroutine in the pool (when available). If
no other goroutine is idle, we are already at capacity and the caller
simply does the work in their own goroutine.

The API introduced here is not the most beautiful, but allows us to
avoid allocations in most cases, which is critical for high-performance
data processing.

* feat(snapshots): speed up uploads by parallelizing directory traversal

Previously directories were walked strictly sequentially which means
we could never be uploading data from multiple directories in parallel,
even if they had just a few files each.

This change switches to using the new `workshare` utility which improves
parallelism. It also reduces memory allocations, goroutine creations
and overall memory usage when taking large snapshots, while increasing
CPU utilization.

Tests on realistic directory structures show huge speed-ups during cold
snapshots (without any metadata caching:)

Photo library - 160GB, files:41717 dirs:1350

    Before: 3m11s
    After: 1m50s
    Total time reduction: 43%

Working code directory - 30.7 GB files:194560 dirs:42455

    Before: 55s
    After: 25s
    Total time reduction: 55%

* do not report multiple cancelation errors during parallel uploads

* do not report multiple cancelation errors during parallel uploads

* pr feedback, clarified usage, added comments

* fixed flaky test
2022-02-18 21:16:30 -08:00
edffa6482e feat(repository): repository writer upgrade lock apis (#1742)
* feat(repo): repository writer upgrade lock apis

* Update repo/upgrade_lock.go

Co-authored-by: Nick <nick@kasten.io>

* pr feedback

* rollback from multiple owners

* more tests to improve coverage

* dropping old format id

Co-authored-by: Shikhar Mall <small@kopia.io>
Co-authored-by: Nick <nick@kasten.io>
2022-02-17 22:07:44 -08:00
a48e24e693 feat(providers): add Google Drive support (#1731)
* feat(provider): Add Google Drive support.

Co-authored-by: xkxx <xkxx@users.noreply.github.com>
Co-authored-by: xkxx <xkxiang@gmail.com>
2022-02-16 22:34:48 -08:00
Jarek Kowalski 512dcebaf4 feat(ui): support for editing pins and description (#1748)
https://user-images.githubusercontent.com/249880/152926565-2d3186d2-7989-4482-bcbc-9b20659745dd.mp4
v0.10.5
2022-02-11 20:34:15 -08:00
Jarek Kowalski 6ae46a4d19 feat(ui): support for defining snapshot and folder actions (#1746)
Thanks @arminus
2022-02-11 18:25:38 -08:00
dependabot[bot] 67e9878074 build(deps): bump follow-redirects from 1.14.6 to 1.14.8 in /app (#1745)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.6 to 1.14.8.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.6...v1.14.8)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-11 18:24:45 -08:00
Jarek Kowalski aa6f9b17df build(ci): auto-cancel previous workflows on the same branch (#1747) 2022-02-11 18:21:11 -08:00
Jarek Kowalski 7b1e652c1b build(deps): update storage/azblob to v0.3.0 (#1743)
This helped remove ugly workaround we had in place due to upstream bug.
2022-02-11 17:29:24 -08:00
Jarek Kowalski a2e4c56978 docs(site): update homebrew installation instructions (#1744)
Since we now have official kopia formula in the `homebrew-core` repo,
dedicated `kopia` TAP is no longer needed (we will keep publishing to
it for a while so it's available as a fallback).

Fixes #1728
2022-02-11 17:28:50 -08:00
Julio Lopez a108469735 docs(general): update compilation from source for Go 1.17 (#1741) 2022-02-10 14:28:39 -08:00
Shikhar MallandShikhar Mall 8ca0061bf6 feat(general): Upgrade Lock intent definitions (#1727)
* feat(general): upgrade lock intent definition

* dropping the constructor

* replace OldFormatVersion with Message field

Co-authored-by: Shikhar Mall <small@kopia.io>
2022-02-08 12:21:31 -08:00
Jarek Kowalski 18ff029dc7 fix(ui): fix parsing of times of day between 00:00 and 00:59 (#1736) 2022-02-07 22:32:37 -08:00
Jarek Kowalski 90df511609 fix(snapshots): treat empty retention policy as retaining ALL, not NONE (#1733)
This is a safety measure which addresses P0 improvement for #1732.

Given that retention policies that retain nothing make no sense, this
is not considered a breaking change.
2022-02-07 11:40:27 -08:00
Jarek Kowalski 04f70f74f3 test(server): fixed flaky TestSnapshotCounters test (#1734) 2022-02-06 21:12:49 -08:00
Jarek Kowalski 4fb473dc77 feat(ui): support for snapshot and source deletion (#1730)
* feat(ui): add support snapshot and source deletion

https://user-images.githubusercontent.com/249880/152661589-51aeaab3-4b14-4c87-84ec-519815b34a89.mp4

* fix(server): fixed ListSnapshots() unique result
2022-02-05 22:43:54 -08:00
Julio Lopez d68cc9f647 refactor(general): misc nits (#1729)
* nit: clarify maxCleanupTime documentation

* refactor: runTaskIndexCompactionQuick helper

Expand one-line wrapper

* nit: improve error message

* nit: clarify error message

* nit: use errors.New for static error messages
2022-02-04 17:38:40 -08:00
dependabot[bot] 3447f1865f build(deps-dev): bump electron from 16.0.5 to 17.0.0 in /app (#1705)
Bumps [electron](https://github.com/electron/electron) from 16.0.5 to 17.0.0.
- [Release notes](https://github.com/electron/electron/releases)
- [Changelog](https://github.com/electron/electron/blob/main/docs/breaking-changes.md)
- [Commits](https://github.com/electron/electron/compare/v16.0.5...v17.0.0)

---
updated-dependencies:
- dependency-name: electron
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
v0.10.4
2022-02-02 22:53:16 -08:00
dependabot[bot] 55f044550a build(deps-dev): bump electron-builder from 22.14.11 to 22.14.13 in /app (#1699)
Bumps [electron-builder](https://github.com/electron-userland/electron-builder/tree/HEAD/packages/electron-builder) from 22.14.11 to 22.14.13.
- [Release notes](https://github.com/electron-userland/electron-builder/releases)
- [Changelog](https://github.com/electron-userland/electron-builder/blob/master/packages/electron-builder/CHANGELOG.md)
- [Commits](https://github.com/electron-userland/electron-builder/commits/v22.14.13/packages/electron-builder)

---
updated-dependencies:
- dependency-name: electron-builder
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-02 21:59:25 -08:00
dependabot[bot] 5785b24b2c build(deps): bump github.com/minio/minio-go/v7 from 7.0.20 to 7.0.21 (#1715)
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.20 to 7.0.21.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.20...v7.0.21)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-02 21:58:03 -08:00
Jarek Kowalski 09c1489be4 fix(ci): add rclone binaries to the list of artifacts (#1719) 2022-02-02 21:53:39 -08:00
dependabot[bot] 77d9bba1a3 build(deps): bump github.com/prometheus/client_golang (#1716)
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.11.0 to 1.12.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.11.0...v1.12.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-02 21:27:21 -08:00
Jarek Kowalski d3edab594e build(ci): added rclone 1.57.0 to Docker container images (#1718)
* build(ci): added rclone 1.57.0 to Docker container images

/bin/rclone is configured to use config in /app/rclone/rclone.conf
so the user can pass -v ~/.config/rclone:/app/rclone

* pr feedback
2022-02-02 21:06:43 -08:00
dependabot[bot] e56cf3ce50 build(deps): bump cloud.google.com/go/storage from 1.18.2 to 1.19.0 (#1714)
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.18.2 to 1.19.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/storage/v1.18.2...spanner/v1.19.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-02 21:06:14 -08:00
dependabot[bot] d6cc776d93 build(deps): bump github.com/google/go-cmp from 0.5.6 to 0.5.7 (#1708)
Bumps [github.com/google/go-cmp](https://github.com/google/go-cmp) from 0.5.6 to 0.5.7.
- [Release notes](https://github.com/google/go-cmp/releases)
- [Commits](https://github.com/google/go-cmp/compare/v0.5.6...v0.5.7)

---
updated-dependencies:
- dependency-name: github.com/google/go-cmp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-02 20:53:27 -08:00
Shikhar MallandShikhar Mall 63bedd3446 feat(cli): allow changing retention parameters from CLI (#1680)
Co-authored-by: Shikhar Mall <small@kopia.io>
2022-02-02 19:04:22 -08:00
Jarek Kowalski e15852af41 test(cli): fixed test flake in TestServerControl (#1717) 2022-02-02 18:44:48 -08:00
Onkar Bhat 6f55e65cbd feat(providers): treat token expiration errors as non-retryable (#1675)
Token expiration errors should be treated as non-retryable errors.
2022-02-02 17:38:32 -08:00
dependabot[bot] 38984b806e build(deps): bump github.com/klauspost/compress from 1.13.6 to 1.14.2 (#1704)
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.13.6 to 1.14.2.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.13.6...v1.14.2)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-02 07:52:27 -08:00
dependabot[bot] bd2143f716 build(deps): bump github.com/sanity-io/litter from 1.5.1 to 1.5.2 (#1709)
Bumps [github.com/sanity-io/litter](https://github.com/sanity-io/litter) from 1.5.1 to 1.5.2.
- [Release notes](https://github.com/sanity-io/litter/releases)
- [Changelog](https://github.com/sanity-io/litter/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sanity-io/litter/compare/v1.5.1...v1.5.2)

---
updated-dependencies:
- dependency-name: github.com/sanity-io/litter
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-02 07:51:54 -08:00