mirror/kopia
1
0
Fork 0
mirror of https://github.com/kopia/kopia.git synced 2026-03-28 02:53:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
2,223 Commits 7 Branches 88 Tags
c7ce87f95b61925fb346cf9b0a7485e700ebddae
Commit Graph

2223 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ali Dowair
c7ce87f95b feat(providers): Implement API to get Storage free space (#1753) 
* Introduce Volume sub-interface
  The Volume interface defines APIs to access a storage provider's
  volume (disk) capacity, usage, etc.. It is inherited by the Storage
  interface, and is at the same hierarchical level as the Reader
  interface.

* Add validations for new Volume method:
  Check that GetCapacity() either returns `ErrNotAVolume`, or that it
  returns a Capacity struct with values that make sense.

* Implement default (passthrough) GetCapacity:
  Cloud providers do not have finite volumes, and WebDAV volumes have no
  notion of volume size and usage. These implementations should just
  return an error (ErrNotAVolume) when their GetCapacity() is called.

* Implement GetCapacity for sftp storage: Uses the sftp.Client interface
* Implement GetCapacity for logging, readonly store
* Implement GetCapacity() for blobtesting implementations

* Implement GetCapacity() for Google Drive:
  Also modifies GetDriveClient to return the entire service instead of just the Files client.

* Implemented GetCapacity() for filesystem storage:
  Implemented the function in a seperate file for each OS/architecture (Unix, OpenBSD, Windows).
 2022-03-16 12:35:33 -07:00
Jarek Kowalski
3ef81134a1 fix(cli): fixing ignoring lines starting with '#' in 'policy edit' (#1830) 
Fixes #1821
 2022-03-16 08:00:50 -07:00
Jarek Kowalski
2b8f7453db fix(cli): fixed and unified help text for policy commands (#1829) 
Fixes #1822
 2022-03-16 00:46:57 -07:00
Jarek Kowalski
c4ab086d5f chore(repository): streamlined internal content cache API (#1828) 
moved from ./repo/content to ./internal/cache
 2022-03-15 23:57:04 -07:00
Jarek Kowalski
1aa3a35518 refactor(repository): moved index blob cache to separate directory (#1827) 
Also cleaned up internal content cache APIs.
 2022-03-13 21:40:15 -07:00
Jarek Kowalski
69dc7ba969 feat(repository): added 'hint' to Prefetch methods. (#1825) 2022-03-12 23:16:39 -08:00
Jarek Kowalski
db7dcac33c feat(repo): exposed PrefetchContents on the repo.Repository() (#1824) 2022-03-12 14:36:54 -08:00
Jarek Kowalski
e591f4b79b chore(ci): skip non-deterministic tests when computing code coverage (#1817) 2022-03-09 07:42:01 -08:00
Jarek Kowalski
68c0d6e278 fix(snapshots): fixed snapshotting of mounted VSS snapshot roots (#1816) 2022-03-08 21:32:11 -08:00
Jarek Kowalski
c6944c70b1 fix(repository): fix deduplication when snapshotting identical files in parallel (#1815) 
This issue was reported on Slack by Ming. Thanks!
This is particularly easy to reproduce when compression is used.
 2022-03-08 20:11:06 -08:00
Jarek Kowalski
766cb57160 feat(ci): automatically bump homebrew version on tag (#1807) v0.10.6 2022-03-06 21:44:13 -08:00
Jarek Kowalski
53bc4d13f4 feat(ui): display snapshot description in snapshot list (#1806) 
Fixes #1769
 2022-03-06 21:09:20 -08:00
Jarek Kowalski
436933caa1 feat(kopiaui): support both left and right click to open tray menu (#1805) 
* feat(kopiaui): support both left and right click to open tray menu

Fixes #1798

* feat(kopiaui): show all repository windows on double click
 2022-03-06 21:09:01 -08:00
Jarek Kowalski
353e8b1ff4 feat(kopiaui): provide feedback for 'Check For Updates Now' (#1804) 
Fixes #1797
 2022-03-06 20:25:27 -08:00
Jarek Kowalski
133513c2cb feat(repository): removed noisy log output (#1803) 2022-03-06 18:51:23 -08:00
Jarek Kowalski
369d304084 refactor(repository): better context cancelation handling (#1802) 
Instead of ignoring context cancelation in Open(), ensure we don't
spawn goroutines that might be canceled.
 2022-03-06 16:56:30 -08:00
dependabot[bot]
a70da61d0e build(deps-dev): bump electron from 17.0.0 to 17.1.0 in /app (#1784) 
Bumps [electron](https://github.com/electron/electron) from 17.0.0 to 17.1.0.
- [Release notes](https://github.com/electron/electron/releases)
- [Changelog](https://github.com/electron/electron/blob/main/docs/breaking-changes.md)
- [Commits](https://github.com/electron/electron/compare/v17.0.0...v17.1.0)

---
updated-dependencies:
- dependency-name: electron
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-06 15:32:29 -08:00
dependabot[bot]
526b1d7377 build(deps): bump go.uber.org/zap from 1.20.0 to 1.21.0 (#1795) 
Bumps [go.uber.org/zap](https://github.com/uber-go/zap) from 1.20.0 to 1.21.0.
- [Release notes](https://github.com/uber-go/zap/releases)
- [Changelog](https://github.com/uber-go/zap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/zap/compare/v1.20.0...v1.21.0)

---
updated-dependencies:
- dependency-name: go.uber.org/zap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-06 14:31:34 -08:00
Jarek Kowalski
926e14aacb feat(repository): added PrefetchObjects() API (#1779) 
* feat(repository): added precaching of data blobs

* feat(repository): added utilities for converting ID slices to strings

* feat(repository): added object.PrefetchBackingContents

* feat(repository): implemented Repository.PrefetchObjects

* feat(cli): added 'cache prefetch' subcommand

* feat(repository): prefetch in parallel

* added tests
 2022-03-06 14:30:58 -08:00
Jarek Kowalski
4962c5f501 chore(repository): added context to beforeop callbacks (#1800) 
This helps eliminate a class of bugs where 'ctx' used in a callbacks
is captured from a closure variable/parameter, which is almost never
correct, because it can be canceled while the beforeop wrapper is
still alive.

Follow-up for #1796.
 2022-03-06 12:02:36 -08:00
Jarek Kowalski
6baf10ad31 fix(repository): fixed 'context canceled' regression (#1799) 
This was broken in #1758 by holding onto a context beyond its intended
lifetime.

Fixes #1796
 2022-03-03 21:02:40 -08:00
dependabot[bot]
192191e056 build(deps): bump github.com/aws/aws-sdk-go from 1.42.44 to 1.43.9 (#1785) 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.44 to 1.43.9.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.44...v1.43.9)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-02 20:12:32 -08:00
dependabot[bot]
5f0dc23d25 build(deps): bump google.golang.org/api from 0.69.0 to 0.70.0 (#1791) 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.69.0 to 0.70.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.69.0...v0.70.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-02 20:12:20 -08:00
dependabot[bot]
5932a660e6 build(deps): bump github.com/minio/minio-go/v7 from 7.0.21 to 7.0.23 (#1794) 
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.21 to 7.0.23.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.21...v7.0.23)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-02 19:31:17 -08:00
dependabot[bot]
8f778ce951 build(deps): bump github.com/chromedp/chromedp from 0.7.6 to 0.7.8 (#1793) 
Bumps [github.com/chromedp/chromedp](https://github.com/chromedp/chromedp) from 0.7.6 to 0.7.8.
- [Release notes](https://github.com/chromedp/chromedp/releases)
- [Commits](https://github.com/chromedp/chromedp/compare/v0.7.6...v0.7.8)

---
updated-dependencies:
- dependency-name: github.com/chromedp/chromedp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 22:25:16 -08:00
dependabot[bot]
1617d51315 build(deps): bump github.com/klauspost/compress from 1.14.2 to 1.14.4 (#1792) 
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.14.2 to 1.14.4.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.14.2...v1.14.4)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 22:03:23 -08:00
dependabot[bot]
1d50a343f7 build(deps): bump nanoid from 3.1.30 to 3.3.1 in /app (#1790) 
Bumps [nanoid](https://github.com/ai/nanoid) from 3.1.30 to 3.3.1.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/3.1.30...3.3.1)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 21:26:00 -08:00
dependabot[bot]
7d5abf87f4 build(deps): bump electron-updater from 4.6.1 to 4.6.5 in /app (#1781) 
Bumps [electron-updater](https://github.com/electron-userland/electron-builder/tree/HEAD/packages/electron-updater) from 4.6.1 to 4.6.5.
- [Release notes](https://github.com/electron-userland/electron-builder/releases)
- [Changelog](https://github.com/electron-userland/electron-builder/blob/master/packages/electron-updater/CHANGELOG.md)
- [Commits](https://github.com/electron-userland/electron-builder/commits/electron-updater@4.6.5/packages/electron-updater)

---
updated-dependencies:
- dependency-name: electron-updater
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 21:23:50 -08:00
dependabot[bot]
dc52d624a4 build(deps): bump go.uber.org/multierr from 1.7.0 to 1.8.0 (#1783) 
Bumps [go.uber.org/multierr](https://github.com/uber-go/multierr) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/uber-go/multierr/releases)
- [Changelog](https://github.com/uber-go/multierr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/multierr/compare/v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: go.uber.org/multierr
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 21:23:17 -08:00
dependabot[bot]
c7ccf4c79d build(deps): bump actions/checkout from 2 to 3 (#1782) 
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 21:22:55 -08:00
dependabot[bot]
1005ba0ec4 build(deps): bump github.com/golang-jwt/jwt/v4 from 4.2.0 to 4.3.0 (#1787) 
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.2.0 to 4.3.0.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.2.0...v4.3.0)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 21:21:29 -08:00
dependabot[bot]
ff8d64c1a9 build(deps): bump electron-log from 4.4.5 to 4.4.6 in /app (#1788) 
Bumps [electron-log](https://github.com/megahertz/electron-log) from 4.4.5 to 4.4.6.
- [Release notes](https://github.com/megahertz/electron-log/releases)
- [Changelog](https://github.com/megahertz/electron-log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/megahertz/electron-log/compare/v4.4.5...v4.4.6)

---
updated-dependencies:
- dependency-name: electron-log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 21:21:11 -08:00
dependabot[bot]
de0f7c4dfb build(deps): bump cloud.google.com/go/storage from 1.19.0 to 1.21.0 (#1789) 
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.19.0 to 1.21.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.19.0...spanner/v1.21.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-01 21:20:58 -08:00
Ali Dowair
7bc32c485a Fix failed robustness job script if TEST_RC unset (#1780) 
If the TEST_RC environment variable is unset, the robustness_job.sh
script exits prematurely due to an unbound variable error, instead of
starting the robustness job without that argument. This commit
evaluates said variable conditionally so that the script doesn't fail.
 2022-03-01 13:32:36 -08:00
Ali Dowair
813927fe7e Add PR Checklist to contribution guidelines (#1775) 
To save reviewers' time and CI/CD cloud spend (if any), this commit adds
a short checklist to the contribution guidelines, outlining some checks
developers can run locally against their code before creating/updating
a pull request on Github.
 2022-02-25 20:55:44 -08:00
Ali Dowair
1778131b0c Introduce robustness job pre-test rc (#1770) 
Before running a robustness test using tools/robustness_job.sh, it
may be useful to perform some other work or to set some environment
variables to prepare the environment/test repo (for example, to clean
up files, redirect input/output). This commit adds an additional
optional argument TEST_RC, to be consumed as an environment variable.
If TEST_RC is set, then the script sources its value before launching
the robustness job.
 2022-02-24 17:36:01 -08:00
Shikhar Mall
58c28f2205 feat(general): Upgrade lock retries & monitoring during open & write sessions (#1758) 
* background upgrade lock monitor

* retry lock forever on connect

* pr feedback

* remove time computations under read lock for efficiency

* extend the unit test to cover lock monitoring with a controlled time function

* more cleanup

Co-authored-by: Shikhar Mall <small@kopia.io>
 2022-02-24 15:00:02 -08:00
Jarek Kowalski
9d63e56bb9 feat(cli): improved formatting of 'policy show' outputs (#1767) 2022-02-22 22:21:48 -08:00
Jarek Kowalski
c9d9bf2de2 chore(infra): upgraded linter to 1.44.2 (#1768) 2022-02-22 21:46:02 -08:00
Jarek Kowalski
6d29b2a082 feat(cli): added snapshot list flags: --storage-stats and --reverse (#1766) 
For #1722
Related #1755
 2022-02-21 20:24:13 -08:00
Jarek Kowalski
f1d3130351 refactor(repository): expose ContentInfo() on repo.Repository (#1765) 2022-02-21 14:38:59 -08:00
Jarek Kowalski
f5cee5ae42 refactor(snapshots): refactored TreeWalker to use workshare (#1762) 
Also cleaned up the API and added unit tests.
 2022-02-20 16:58:53 -08:00
Jarek Kowalski
bf38c7b788 fix(ci): remove changelog type filter (#1761) 2022-02-20 11:01:33 -08:00
xkxx
a379302f36 docs(providers): Instructions for how to use the Google Drive backend. (#1760) 2022-02-19 12:04:25 -08:00
Jarek Kowalski
4bc69d05ef feat(snapshots): big upload performance improvements by parallelizing directory traversal (#1752) 
* feat(general): added internal/workshare package

This introduces work sharing utility useful when walking trees of
things (such as filesystem), which allows N threads/goroutines to be
used.

Whenever a routine is visiting its children, it can share some of that
work with another idle goroutine in the pool (when available). If
no other goroutine is idle, we are already at capacity and the caller
simply does the work in their own goroutine.

The API introduced here is not the most beautiful, but allows us to
avoid allocations in most cases, which is critical for high-performance
data processing.

* feat(snapshots): speed up uploads by parallelizing directory traversal

Previously directories were walked strictly sequentially which means
we could never be uploading data from multiple directories in parallel,
even if they had just a few files each.

This change switches to using the new `workshare` utility which improves
parallelism. It also reduces memory allocations, goroutine creations
and overall memory usage when taking large snapshots, while increasing
CPU utilization.

Tests on realistic directory structures show huge speed-ups during cold
snapshots (without any metadata caching:)

Photo library - 160GB, files:41717 dirs:1350

    Before: 3m11s
    After: 1m50s
    Total time reduction: 43%

Working code directory - 30.7 GB files:194560 dirs:42455

    Before: 55s
    After: 25s
    Total time reduction: 55%

* do not report multiple cancelation errors during parallel uploads

* do not report multiple cancelation errors during parallel uploads

* pr feedback, clarified usage, added comments

* fixed flaky test
 2022-02-18 21:16:30 -08:00
Shikhar Mall
edffa6482e feat(repository): repository writer upgrade lock apis (#1742) 
* feat(repo): repository writer upgrade lock apis

* Update repo/upgrade_lock.go

Co-authored-by: Nick <nick@kasten.io>

* pr feedback

* rollback from multiple owners

* more tests to improve coverage

* dropping old format id

Co-authored-by: Shikhar Mall <small@kopia.io>
Co-authored-by: Nick <nick@kasten.io>
 2022-02-17 22:07:44 -08:00
Jarek Kowalski
a48e24e693 feat(providers): add Google Drive support (#1731) 
* feat(provider): Add Google Drive support.

Co-authored-by: xkxx <xkxx@users.noreply.github.com>
Co-authored-by: xkxx <xkxiang@gmail.com>
 2022-02-16 22:34:48 -08:00
Jarek Kowalski
512dcebaf4 feat(ui): support for editing pins and description (#1748) 
https://user-images.githubusercontent.com/249880/152926565-2d3186d2-7989-4482-bcbc-9b20659745dd.mp4
v0.10.5 		2022-02-11 20:34:15 -08:00
Jarek Kowalski
6ae46a4d19 feat(ui): support for defining snapshot and folder actions (#1746) 
Thanks @arminus
 2022-02-11 18:25:38 -08:00
dependabot[bot]
67e9878074 build(deps): bump follow-redirects from 1.14.6 to 1.14.8 in /app (#1745) 
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.6 to 1.14.8.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.6...v1.14.8)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-11 18:24:45 -08:00