mirror/kopia
1
0 Fork 0
mirror of https://github.com/kopia/kopia.git synced 2026-10-06 00:51:37 -04:00
Code Issues Packages Projects Releases Wiki Activity
1,879 Commits 7 Branches 90 Tags
a203fa414dcc368afd1d7ebd91db45ea79858a37
Commit Graph
1879 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Jarek Kowalski a203fa414d gather: MakeContiguous support for arbitrary chunk sizes (#1247) 2021-08-22 12:33:36 -07:00
Jarek Kowalski 9e182f131a linter: upgraded to 1.42.0 (#1246) 2021-08-20 18:26:45 -07:00
Jarek Kowalski 35d0f31c0d huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244) 
This helps recycle buffers more efficiently during snapshots.
Also, improved memory tracking, enabled profiling flags and added pprof
by default.
 2021-08-20 08:45:10 -07:00
Jarek Kowalski 4676e9e40e content: automatically refresh indexes on read instead of relying on RefreshPeriodically() loop (#1243) 
also force refresh indexes before running maintenance
 2021-08-13 00:06:46 -07:00
Jarek Kowalski bece46d5f6 logging: switched file log timestamps to be UTC by default, otherwise it's hard to correlate logs (#1240) 2021-08-07 20:18:03 -07:00
Jarek Kowalski 1e10e85225 content: ensure that we always consistently pick content.Info amongst entries with identical time and deleted flag (#1239) 
This was reported by Stephan Budach on Slack (thanks!) who provided
the log where two separate write sessions created identical content
at exactly the same time.

2021-07-11T06:49:30.056211Z DEBUG [writer-4:Source Manager Uploader] write-content e5a857d1505308e94a4d495833a278da new
2021-07-11T06:49:30.222525Z DEBUG [writer-4:Source Manager Uploader] add-to-pack pe5873427a445f07f92f66837bfbb5752-sfa2a7c24c8d31100106 e5a857d1505308e94a4d495833a278da p:pe5873427a445f07f92f66837bfbb5752-sfa2a7c24c8d31100106 3461377 d:false
2021-07-11T06:49:30.857310Z DEBUG [writer-2:Source Manager Uploader] write-content e5a857d1505308e94a4d495833a278da new
2021-07-11T06:49:31.982930Z DEBUG [writer-2:Source Manager Uploader] add-to-pack p24b9a682047d63b19614827b653e54eb-s583f9f99a3312721106 e5a857d1505308e94a4d495833a278da p:p24b9a682047d63b19614827b653e54eb-s583f9f99a3312721106 3461377 d:false

Because we made different decision w.r.t which one is active at two
separate times, maintenance ended up deleting the wrong blob.

Introduced contentInfoGreaterThan(a, b) which is used in all 3 places
where we do content ID merges.
 2021-08-07 16:10:29 -07:00
dependabot[bot] 0830de9c16 build(deps): bump github.com/klauspost/compress from 1.12.2 to 1.13.3 (#1237) 
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.12.2 to 1.13.3.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.12.2...v1.13.3)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-07 14:33:50 -07:00
dependabot[bot] ff19caab95 build(deps): bump github.com/aws/aws-sdk-go from 1.40.12 to 1.40.17 (#1236) 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.12 to 1.40.17.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.12...v1.40.17)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-07 11:30:33 -07:00
Jarek Kowalski 4aacad25f5 server: switched to github.com/golang-jwt/jwt/v4 to fix upstream security issue (#1235) 2021-08-07 09:18:29 -07:00
Jarek Kowalski e3e7cae4fe content: removed EpochMarkerIndexBlobPrefix from allIndexBlobPrefixes (#1232) 2021-08-06 23:13:48 -07:00
Jarek Kowalski 1238a0db60 cli: additional 'index inspect' flags (#1231) 
* cli: fixed output of 'index inspect' to go to stdout

* cli: additional flags for 'index inspect'
 2021-08-06 21:54:38 -07:00
Jarek Kowalski b96623f8a7 epoch: simplified TestIndexEpochManager_DeletionFailing test (#1230) 2021-08-02 22:24:33 -07:00
dependabot[bot] fcd79b40af build(deps): bump electron-log from 4.3.5 to 4.4.1 in /app (#1220) 
Bumps [electron-log](https://github.com/megahertz/electron-log) from 4.3.5 to 4.4.1.
- [Release notes](https://github.com/megahertz/electron-log/releases)
- [Changelog](https://github.com/megahertz/electron-log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/megahertz/electron-log/compare/v4.3.5...v4.4.1)

---
updated-dependencies:
- dependency-name: electron-log
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-02 21:38:46 -07:00
dependabot[bot] eae75218bd build(deps): bump google.golang.org/api from 0.50.0 to 0.52.0 (#1229) 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.50.0 to 0.52.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.50.0...v0.52.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-02 21:38:13 -07:00
Jarek Kowalskianddependabot[bot] 67165cae5f build(deps): bump github.com/prometheus/client_golang (#1226) 
Includes manual change to fix linter deprecation warning.

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.10.0 to 1.11.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/master/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.10.0...v1.11.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-02 21:36:15 -07:00
dependabot[bot] 17b430603d build(deps): bump postcss from 8.3.5 to 8.3.6 in /htmlui (#1217) 
Bumps [postcss](https://github.com/postcss/postcss) from 8.3.5 to 8.3.6.
- [Release notes](https://github.com/postcss/postcss/releases)
- [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/postcss/compare/8.3.5...8.3.6)

---
updated-dependencies:
- dependency-name: postcss
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-02 21:12:21 -07:00
dependabot[bot] c04e3be510 build(deps-dev): bump electron-notarize from 1.0.0 to 1.0.1 in /app (#1219) 
Bumps [electron-notarize](https://github.com/electron/electron-notarize) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/electron/electron-notarize/releases)
- [Changelog](https://github.com/electron/electron-notarize/blob/master/.releaserc.json)
- [Commits](https://github.com/electron/electron-notarize/compare/v1.0.0...v1.0.1)

---
updated-dependencies:
- dependency-name: electron-notarize
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-02 21:11:52 -07:00
dependabot[bot] ecd9b0214e build(deps): bump github.com/zeebo/blake3 from 0.1.1 to 0.2.0 (#1223) 
Bumps [github.com/zeebo/blake3](https://github.com/zeebo/blake3) from 0.1.1 to 0.2.0.
- [Release notes](https://github.com/zeebo/blake3/releases)
- [Commits](https://github.com/zeebo/blake3/compare/v0.1.1...v0.2.0)

---
updated-dependencies:
- dependency-name: github.com/zeebo/blake3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-02 21:10:55 -07:00
dependabot[bot] 140a86ba72 build(deps): bump github.com/google/uuid from 1.2.0 to 1.3.0 (#1222) 
Bumps [github.com/google/uuid](https://github.com/google/uuid) from 1.2.0 to 1.3.0.
- [Release notes](https://github.com/google/uuid/releases)
- [Commits](https://github.com/google/uuid/compare/v1.2.0...v1.3.0)

---
updated-dependencies:
- dependency-name: github.com/google/uuid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-02 21:10:37 -07:00
dependabot[bot] 5b8485b8ee build(deps): bump github.com/aws/aws-sdk-go from 1.39.4 to 1.40.12 (#1224) 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.39.4 to 1.40.12.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.39.4...v1.40.12)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-02 21:09:37 -07:00
Jarek Kowalski d6d9a1fb5f Maintenance improvements for epoch-based index structures (#1225) 
* testing: KOPIA_TEST_LOG_OUTPUT logs subcommand outputs

* cli: additional flags for 'blob list'

* Makefile: run all tests against epoch-based index manager

* epoch: added support for deletion watermark, which keeps track of latest maintenance which dropped index entries

* content: added deletion watermark to content manager

* maintenance: improved maintenance without safety to force rewrites

* maintenance: skip quick maintenance when epoch manager is enabled

* maintenance: do not enable quick maintenance when epoch manager is used

* testing: skip TestIndexOptimize when running against epoch manager-backed index strutures
 2021-08-02 21:08:54 -07:00
Jarek Kowalski d77e771df7 provider test: fixed typo 2021-07-29 06:26:06 -07:00
Jarek Kowalski cfdbc40756 cli: Fixed ETA estimation of 'snapshot verify' (#1213) 2021-07-26 18:57:58 -07:00
abba23 6205b55601 Fix weekly retention labels not being assigned correctly (#1211) 
* Fix weekly retention labels not being assigned correctly

This should resolve #814.

* Add test for weekly retention label assignment
 2021-07-26 07:19:59 -07:00
Jarek Kowalski 6c5bf42d23 cli: changed 'kopia snapshot verify --verify-files-percent' to float (#1210) 
* cli: changed 'kopia snapshot verify --verify-files-percent' to float

* cli: added 'kopia content verify --download-percent'
 2021-07-24 13:04:50 -07:00
Benedikt McMullinandJarek Kowalski 317d5c3419 force-hash: Allow fractional percentages (#1209) 
* force-hash: Allow fractional percentages

Co-authored-by: Jarek Kowalski <jaak@jkowalski.net>
 2021-07-24 09:45:46 -07:00
Jarek Kowalski 4f93a17782 WebDAV and Rclone atomic writes (#1206) 
* webdav: added --atomic-writes (defaults to false) to assume WebDAV supports atomic writes

* rclone: added --atomic-writes (defaults to true)

This allows the provider to assume rclone backend supports atomic writes (most of them do).
 2021-07-19 21:55:56 -07:00
Jarek Kowalski e42cc6ccce Added 'kopia repository validate-provider` (#1205) 
* cli: added 'repository validate-provider' which runs a set of tests against blob storage provider to validate it

This implements a provider tests which exercises subtle behaviors which are not always correctly implemented by providers claiming compatibility with S3, for example.

The test checks:

- not found behavior
- prefix scans
- timestamps
- write atomicity

* retry: improved error message on failure

* rclone: fixed stats reporting and awaiting for completion

* webdav: prevent panic when attempting to mkdir with empty name

* testing: run providervalidation.ValidateProvider as part of regular provider tests

* cli: print a recommendation to validate provider after repository creation
 2021-07-19 21:42:24 -07:00
Jarek Kowalski 7dd9e332ad ci: fixed syntax 2021-07-18 09:33:38 -07:00
Jarek Kowalski e2e0cb37d7 ci: changed continue-on-error: true to if: succeeded() || failed() 2021-07-18 09:31:23 -07:00
Jarek Kowalski bdd53f05d5 ci: install provider test dependencies separately 2021-07-18 09:18:14 -07:00
Jarek Kowalski 49d8a32783 testing: split provider tests into separate steps 2021-07-18 09:15:40 -07:00
Jarek Kowalski 33cf322941 ci: run provider test more often to get better stability signal 2021-07-18 09:09:43 -07:00
dependabot[bot] adbf2131bb build(deps): bump @testing-library/react in /htmlui (#1169) 
Bumps [@testing-library/react](https://github.com/testing-library/react-testing-library) from 11.2.7 to 12.0.0.
- [Release notes](https://github.com/testing-library/react-testing-library/releases)
- [Changelog](https://github.com/testing-library/react-testing-library/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testing-library/react-testing-library/compare/v11.2.7...v12.0.0)

---
updated-dependencies:
- dependency-name: "@testing-library/react"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-17 20:37:08 -07:00
Jarek Kowalski 9df5e5f5e4 rclone: updated wait logic 2021-07-17 20:13:29 -07:00
Jarek Kowalski 9fd6a89f43 testing: possible fix for TestS3StorageMinioSTS 2021-07-17 20:10:20 -07:00
Jarek Kowalski 8c2e4ff799 testing: simplified S3 test to fix error caused by minio behavior change 2021-07-17 20:01:00 -07:00
Jarek Kowalski 1a6f0ba49a testing: possible fix for TestS3StorageMinioSTS 2021-07-17 19:06:24 -07:00
Jarek Kowalski 4c9b9e523a Test cleanup improvements (#1203) 
* testing: removed testutil.Retry because all providers now have internal retries

* testing: simplified and unified cleanup for all cloud providers using shared buckets
 2021-07-17 18:54:37 -07:00
Jarek Kowalski 42f0278074 s3: cleaned up test 2021-07-17 16:29:06 -07:00
Jarek Kowalski 4c2f52a2e3 Rclone and testing improvements (#1202) 
* sharded: added parallel iteration of blobs to improve performance

* retry: reduce first retry delay 1s->100ms

* testing: additional assertions for blob storage testing

* rclone: testing cleanup improvements, re-enabled OneDrive

* cli: added --list-parallelism parameter to fs,webdav,sftp and rclone

* sharded: added dedicated test
 2021-07-17 16:04:51 -07:00
Jarek Kowalski 928e1d3071 rclone: wait for rclone transfers to finish when closing (#1201) 
added --rclone-debug flag to enable better logging from rclone
 2021-07-17 12:55:03 -07:00
Jarek Kowalski 7b70446b74 ci: switch license NPM check to 'onlyAllow' instead of 'failOn' 2021-07-17 10:31:21 -07:00
Jarek Kowalski 42e5661bf4 ci: added license check using github.com/frapposelli/wwhrd (#1200) 
* ci: added license check using github.com/frapposelli/wwhrd

* ci: also check NPM licenses

* readme: removed FOSSA badge

* Makefile: install NPM dependencies before license check
 2021-07-17 09:58:25 -07:00
dependabot[bot] b2cc840f2c build(deps): bump github.com/pierrec/lz4 (#1188) 
Bumps [github.com/pierrec/lz4](https://github.com/pierrec/lz4) from 2.6.0+incompatible to 2.6.1+incompatible.
- [Release notes](https://github.com/pierrec/lz4/releases)
- [Commits](https://github.com/pierrec/lz4/compare/v2.6.0...v2.6.1)

---
updated-dependencies:
- dependency-name: github.com/pierrec/lz4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-17 07:59:53 -07:00
dependabot[bot] 51c54033b5 build(deps): bump github.com/Azure/azure-storage-blob-go (#1194) 
Bumps [github.com/Azure/azure-storage-blob-go](https://github.com/Azure/azure-storage-blob-go) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/Azure/azure-storage-blob-go/releases)
- [Changelog](https://github.com/Azure/azure-storage-blob-go/blob/master/ChangeLog.md)
- [Commits](https://github.com/Azure/azure-storage-blob-go/compare/v0.13.0...v0.14.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-storage-blob-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-17 07:59:29 -07:00
Jarek KowalskiandJulio Lopez 730ba7b94a Repository password change support (#1197) 
* repo: added 'enable password change' flag (defaults to true for new repositories), which prevents embedding replicas of kopia.repository in pack blobs

* cli: added 'repo change-password' which can change the password of a connected repository

* repo: nit - renamed variables and functions dealing with key derivation

* repo: fixed cache validation HMAC secret to use stored HMAC secret instead of password-derived one

* cli: added test for repo change-password

* repo: negative cases for attempting to change password in an old repository

* Update cli/command_repository_change_password.go

Co-authored-by: Julio Lopez <julio+gh@kasten.io>

Co-authored-by: Julio Lopez <julio+gh@kasten.io>
 2021-07-17 07:58:02 -07:00
dependabot[bot] f15c76ef6e build(deps): bump google.golang.org/api from 0.48.0 to 0.50.0 (#1191) 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.48.0 to 0.50.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.48.0...v0.50.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-15 19:11:06 -07:00
Jarek Kowalski bffe8b37da repo: limit the duration of kopia.repository caching to 15 minutes (#1196) 
added flags to specify kopia.repository cache duration
 2021-07-15 11:32:55 -07:00
Steve Joachim 83b4dee349 serialize client auth (#232) (#1198) 2021-07-15 11:27:13 -07:00