mirror/kopia
1
0
Fork 0
mirror of https://github.com/kopia/kopia.git synced 2026-05-15 02:05:39 -04:00
Code Issues Packages Projects Releases Wiki Activity
3,660 Commits 6 Branches 89 Tags
1c5c4e256839dabec00eb86ee4801dfd198efe4c
Commit Graph

3660 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
dependabot[bot]
532cb63f28 build(deps): bump uuid from 8.3.2 to 9.0.0 in /app (#2478) 
Bumps [uuid](https://github.com/uuidjs/uuid) from 8.3.2 to 9.0.0.
- [Release notes](https://github.com/uuidjs/uuid/releases)
- [Changelog](https://github.com/uuidjs/uuid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/uuidjs/uuid/compare/v8.3.2...v9.0.0)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-21 08:02:35 -08:00
Jarek Kowalski
f0d237fadb chore(ci): auto-merge dependabot PRs for golang.org/x/* (#2699) 2023-01-21 07:58:33 -08:00
dependabot[bot]
aa6422d347 build(deps): bump json5 from 2.2.1 to 2.2.3 in /app (#2687) 
Bumps [json5](https://github.com/json5/json5) from 2.2.1 to 2.2.3.
- [Release notes](https://github.com/json5/json5/releases)
- [Changelog](https://github.com/json5/json5/blob/main/CHANGELOG.md)
- [Commits](https://github.com/json5/json5/compare/v2.2.1...v2.2.3)

---
updated-dependencies:
- dependency-name: json5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-21 07:39:05 -08:00
Edward Betts
1e97574391 fix(general): correct spelling mistakes (#2684) 2023-01-21 07:37:15 -08:00
ashmrtn
3bb7879a8c fix(snapshots): cached stream file size (#2690) 
* Properly populate file size for cached entries

StreamingFiles don't know their size until they are read. This leads to
entries marked with size 0 because size is not compared when determining
if something is cached or not as the size is sourced from the current
(unread) entry.

Instead, create the dir entry from the previous entry. As StreamingFile
do not allow setting file mode, permissions, owner, etc using the old
entry suffices.

* Use cached entry size for stats

Also use the cached entry size when calculating stats since
StreamingFile has 0 size.

* Update tests for cached files to check size

Check at least the total size of the snapshot is updated when handling
StreamingFiles.
 2023-01-20 14:07:39 -08:00
dependabot[bot]
ba938cf58c build(deps): bump go.opentelemetry.io/otel/sdk from 1.11.1 to 1.11.2 (#2664) 2022-12-29 23:25:24 +00:00
dependabot[bot]
1b781f1482 build(deps): bump go.opentelemetry.io/otel from 1.11.1 to 1.11.2 (#2639) 
Bumps [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) from 1.11.1 to 1.11.2.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.11.1...v1.11.2)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-29 19:52:47 +00:00
Bishop Clark
37d0e56e64 docs(site): Update README.md (#2657) 
'storages' isn't really a word.  It's like 'fetch' isn't an adjective.
 2022-12-28 20:47:01 +00:00
Jarek Kowalski
f8be8f6a56 refactor(repository): extract parts repo/content into packages (#2651) 
- repolog package
- blobcrypto package
- indexblob package

Minor cleanups:

- removed dead code
- introduced New*() methods for object construction
 2022-12-17 16:19:12 +00:00
Aaron Alpar
4c0fe390e1 feat(repository): new repository upgrade sub-command to validate indices. (#2561) 
* remove unused context param

* add index validation

* add test

* add command stub

* add comments. fixup.

* Update cli/command_repository_upgrade.go

Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>

* fixup bad merge

* add validation to command line

* set-parameter fixup

* move poison blob to commit

* fixups from PR comments

* fix bug for set-parameter

* robust commit-mode. add upgrade test

* fixup incorrect methods

* fixup for lint

* fixup for lint

* fixup named return values

* fixup failing test

* fixup lint

* fixup for lint

* Update repo/content/committed_read_manager.go

Co-authored-by: Nick <nick@kasten.io>

* Update repo/content/committed_read_manager.go

Co-authored-by: Nick <nick@kasten.io>

* use xor instead of fixed value

* lint fixups

* fixup for lint cyclomatic complexity

* fixup bad merge

* fixup set-parameters downgrade

* Update cli/command_repository_upgrade.go

Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>

* Update cli/command_repository_upgrade.go

Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>

* fixup set-parameters downgrade

* cleanup set-parameters test

* fixup tests, refactor for index verification

* fixups and clarity

* remove set-parameter downgrade check

* index on feat-k10-12441-B-validate-index: b63879cd Merge 'master' into feat-k10-12441-B-validate-index

* trigger build

* remove needless blank line

Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
Co-authored-by: Nick <nick@kasten.io>
 2022-12-13 09:42:09 -08:00
dependabot[bot]
064f3a4648 build(deps): bump github.com/aws/aws-sdk-go from 1.44.155 to 1.44.157 (#2643) 2022-12-09 23:25:14 +00:00
dependabot[bot]
4173feeb9e build(deps): bump google.golang.org/api from 0.103.0 to 0.104.0 (#2641) 2022-12-08 23:27:00 +00:00
dependabot[bot]
bca66b1ffd build(deps): bump github.com/aws/aws-sdk-go from 1.44.143 to 1.44.155 (#2638) 2022-12-07 23:38:23 +00:00
Shikhar Mall
c5b8510ba3 fix(general): prevent storage retries on ESTALE errors (#2628) 
* fix(general): prevent retries on ESTALE errors

* adding missing imports

* silent the linter for platform specific code

Co-authored-by: Shikhar Mall <small@kopia.io>
 2022-12-06 21:49:35 -08:00
dependabot[bot]
bdb91a40e4 build(deps): bump github.com/pkg/profile from 1.6.0 to 1.7.0 (#2555) 
Bumps [github.com/pkg/profile](https://github.com/pkg/profile) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/pkg/profile/releases)
- [Commits](https://github.com/pkg/profile/compare/v1.6.0...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/pkg/profile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-06 20:01:07 -08:00
dependabot[bot]
dba94ebf6e build(deps): bump golang.org/x/sys from 0.2.0 to 0.3.0 (#2630) 
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.2.0 to 0.3.0.
- [Release notes](https://github.com/golang/sys/releases)
- [Commits](https://github.com/golang/sys/compare/v0.2.0...v0.3.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-06 20:00:44 -08:00
lciti
37d64afc21 fix(snapshots): Partly fix #544 by supporting setuid/setgid/sticky bits when snapshotting, restoring and fuse-mounting. (#2597) 
Also modified an end-to-end test to also check that these extra mode flags work when snapshotting+restoring.
Manually tested fuse-mount.

Co-authored-by: Luca Citi <lciti@ieee.org>
 2022-12-07 03:26:29 +00:00
Muescha
0e09e1a13f add homebrew installation for kopiaui (#2627) 2022-12-06 19:05:59 -08:00
dependabot[bot]
759b8a3480 build(deps): bump go.uber.org/zap from 1.23.0 to 1.24.0 (#2631) 
Bumps [go.uber.org/zap](https://github.com/uber-go/zap) from 1.23.0 to 1.24.0.
- [Release notes](https://github.com/uber-go/zap/releases)
- [Changelog](https://github.com/uber-go/zap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/zap/compare/v1.23.0...v1.24.0)

---
updated-dependencies:
- dependency-name: go.uber.org/zap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-06 19:05:43 -08:00
Aaron Alpar
ca7df80e64 fix(cli): do not allow index downgrade by using set-parameter (#2629) 
* disallow downgrades

* fixup merge
 2022-12-06 18:33:55 -08:00
dependabot[bot]
04fcedad12 build(deps): bump github.com/klauspost/reedsolomon from 1.11.2 to 1.11.3 (#2610) 
Bumps [github.com/klauspost/reedsolomon](https://github.com/klauspost/reedsolomon) from 1.11.2 to 1.11.3.
- [Release notes](https://github.com/klauspost/reedsolomon/releases)
- [Commits](https://github.com/klauspost/reedsolomon/compare/v1.11.2...v1.11.3)

---
updated-dependencies:
- dependency-name: github.com/klauspost/reedsolomon
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-04 20:57:07 -08:00
Jarek Kowalski
c2cc25c504 feat(repository): added internal/metricid package (#2626) 
This manages mapping of metric names to IDs which allows efficient JSON
representation of counter values for each set of metrics where
only values are in the index order.

This will be used in the telemetry protocol and for storing counters in
the repository.

Added test that ensures all metrics registered in a repository have
the corresponding mapping.
 2022-12-05 04:41:42 +00:00
Jarek Kowalski
0828eb2900 feat(repository): added OnSuccessfulFlush() (#2623) 2022-12-03 23:28:29 +00:00
dependabot[bot]
5463fb7e1b build(deps-dev): bump playwright from 1.26.1 to 1.28.1 in /app (#2615) 2022-12-03 20:17:45 +00:00
dependabot[bot]
3b7fbf294d build(deps): bump cloud.google.com/go/storage from 1.28.0 to 1.28.1 (#2619) 2022-12-03 20:15:58 +00:00
Jarek Kowalski
4c385c4f93 feat(repository): added before and after callbacks for RepositoryWriter.Flush() (#2622) 2022-12-03 19:41:00 +00:00
Jarek Kowalski
7cc93265ef refactor(repository): moved cache protection to separate package (#2621) 2022-12-03 18:47:01 +00:00
dependabot[bot]
c44c60d376 build(deps-dev): bump playwright-core from 1.26.1 to 1.28.1 in /app (#2614) 2022-12-03 12:51:26 +00:00
dependabot[bot]
89787057de build(deps): bump github.com/minio/minio-go/v7 from 7.0.44 to 7.0.45 (#2609) 2022-12-02 12:51:36 +00:00
Jarek Kowalski
e57020fb70 test(repository): server testability refactoring (#2612) 
- removed repo.OpenAPIServer() which was only needed for testability
- introduced servertesting package to replace it
 2022-12-01 06:27:52 +00:00
ashmrtn
fa5aaa40ac feat(snapshots): Add labels to snapshot checkpoint manifests (#2548) 
* Apply labels to checkpoint snapshot manifests

This allows looking up checkpoint snapshot manifests by label if
desired.

* Fixup other calls to Upload function

* Fix missed Upload call

* Revert code to pass checkpoint labels as arg

This reverts commits
* bf844ff068
* 76438c9bc1
* d748b6361b

* Add checkpoint labels to checkpoint manifests

Allow finer-grained lookups on checkpoints by adding labels to them.
Labels are specified in the Uploader struct and apply to all checkpoints
for a snapshot.

* Test checkpoint labels.

Be strict about testing and have a separate copy of the labels.
 2022-11-22 18:23:58 -08:00
dependabot[bot]
a7d7418111 build(deps): bump github.com/minio/minio-go/v7 from 7.0.43 to 7.0.44 (#2592) 2022-11-22 06:39:45 +00:00
Jarek Kowalski
82b3f1c648 refactor(repository): refactored server repository implementations (#2595) 
This extracts common bits shared by HTTP and GRPC implementations into
one.
 2022-11-22 04:29:05 +00:00
Jarek Kowalski
f50779241e refactor(repository): added metrics aggregation, removed gauges (#2594) 2022-11-22 04:18:21 +00:00
dependabot[bot]
525f754a2c build(deps): bump github.com/aws/aws-sdk-go from 1.44.139 to 1.44.143 (#2593) 2022-11-21 23:51:01 +00:00
dependabot[bot]
4051630f3a build(deps): bump google.golang.org/grpc from 1.50.1 to 1.51.0 (#2591) 2022-11-21 23:38:35 +00:00
dependabot[bot]
b5dca25d52 build(deps): bump actions/dependency-review-action from 2 to 3 (#2574) 
Bumps [actions/dependency-review-action](https://github.com/actions/dependency-review-action) from 2 to 3.
- [Release notes](https://github.com/actions/dependency-review-action/releases)
- [Commits](https://github.com/actions/dependency-review-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/dependency-review-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-11-19 10:41:30 -08:00
Jarek Kowalski
65f295ed79 refactor(repository): replaced atomic values with Go 1.19 atomic wrappers (#2590) 
Almost all were easy to replace, except ones exposed via JSON which
have been left as-is.

The linter has a cool behavior where it flags attempts to pass
`atomic.Int32` for example by value , which is always a mistake,
say as an argument to `fmt.Sprintf()`
 2022-11-19 18:39:04 +00:00
dependabot[bot]
d0b8f5f7c0 build(deps): bump golang.org/x/crypto from 0.1.0 to 0.3.0 (#2580) 
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.1.0 to 0.3.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/compare/v0.1.0...v0.3.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-11-19 10:29:39 -08:00
dependabot[bot]
f41ecf4df8 build(deps): bump github.com/klauspost/reedsolomon from 1.11.1 to 1.11.2 (#2586) 
Bumps [github.com/klauspost/reedsolomon](https://github.com/klauspost/reedsolomon) from 1.11.1 to 1.11.2.
- [Release notes](https://github.com/klauspost/reedsolomon/releases)
- [Commits](https://github.com/klauspost/reedsolomon/compare/v1.11.1...v1.11.2)

---
updated-dependencies:
- dependency-name: github.com/klauspost/reedsolomon
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-11-19 10:29:11 -08:00
Jarek Kowalski
d782dbe550 refactor(repository): refactored Repository close logic (#2589) 
This ensures metric registry is always closed on last Close().
 2022-11-19 06:52:03 +00:00
Jarek Kowalski
3d38ec8405 feat(repository): added Snapshot(bool) option to reset metrics (#2588) 2022-11-19 05:29:24 +00:00
dependabot[bot]
6c0b01b551 build(deps): bump golang.org/x/mod from 0.6.0 to 0.7.0 (#2570) 
Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/golang/mod/releases)
- [Commits](https://github.com/golang/mod/compare/v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/mod
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-11-16 20:08:54 -08:00
dependabot[bot]
ce96853302 build(deps): bump github.com/aws/aws-sdk-go from 1.44.136 to 1.44.139 (#2581) 2022-11-16 23:29:28 +00:00
dependabot[bot]
b4c8844ce1 build(deps): bump github.com/aws/aws-sdk-go from 1.44.135 to 1.44.136 (#2569) 2022-11-11 23:31:23 +00:00
dependabot[bot]
3a3e2b9512 build(deps): bump golang.org/x/net from 0.1.0 to 0.2.0 (#2566) 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.1.0 to 0.2.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/compare/v0.1.0...v0.2.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-11-10 16:15:23 -08:00
dependabot[bot]
91dc0f7175 build(deps): bump github.com/aws/aws-sdk-go from 1.44.129 to 1.44.135 (#2567) 2022-11-10 23:39:25 +00:00
dependabot[bot]
3ab930a1c0 build(deps): bump minimist from 1.2.6 to 1.2.7 in /app (#2536) 
Bumps [minimist](https://github.com/minimistjs/minimist) from 1.2.6 to 1.2.7.
- [Release notes](https://github.com/minimistjs/minimist/releases)
- [Changelog](https://github.com/minimistjs/minimist/blob/main/CHANGELOG.md)
- [Commits](https://github.com/minimistjs/minimist/compare/v1.2.6...v1.2.7)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-11-09 22:22:21 -08:00
Jarek Kowalski
78edd92692 refactor(repository): refactored Prometheus metrics (#2532) 
This may be a breaking change for users who rely on particular kopia metrics (unlikely):

- introduced blob-level metrics:

* `kopia_blob_download_full_blob_bytes_total`
* `kopia_blob_download_partial_blob_bytes_total`
* `kopia_blob_upload_bytes_total`
* `kopia_blob_storage_latency_ms` - per-method latency distribution
* `kopia_blob_errors_total` - per-method error counter

- updated cache metrics to indicate particular cache

* `kopia_cache_hit_bytes_total{cache="CACHE_TYPE"}`
* `kopia_cache_hit_total{cache="CACHE_TYPE"}`
* `kopia_cache_malformed_total{cache="CACHE_TYPE"}`
* `kopia_cache_miss_total{cache="CACHE_TYPE"}`
* `kopia_cache_miss_errors_total{cache="CACHE_TYPE"}`
* `kopia_cache_miss_bytes_total{cache="CACHE_TYPE"}`
* `kopia_cache_store_errors_total{cache="CACHE_TYPE"}`

where `CACHE_TYPE` is one of `contents`, `metadata` or `index-blobs`

- reorganized and unified content-level metrics:

* `kopia_content_write_bytes_total`
* `kopia_content_write_duration_nanos_total`

* `kopia_content_compression_attempted_bytes_total`
* `kopia_content_compression_attempted_duration_nanos_total`
* `kopia_content_compression_savings_bytes_total`
* `kopia_content_compressible_bytes_total`
* `kopia_content_non_compressible_bytes_total`
* `kopia_content_after_compression_bytes_total`

* `kopia_content_decompressed_bytes_total`
* `kopia_content_decompressed_duration_nanos_total`

* `kopia_content_encrypted_bytes_total`
* `kopia_content_encrypted_duration_nanos_total`

* `kopia_content_hashed_bytes_total`
* `kopia_content_hashed_duration_nanos_total`

* `kopia_content_deduplicated_bytes_total`

* `kopia_content_read_bytes_total`
* `kopia_content_read_duration_nanos_total`

* `kopia_content_decrypted_bytes_total`
* `kopia_content_decrypted_duration_nanos_total`

* `kopia_content_uploaded_bytes_total`

Also introduced `internal/metrics` framework which constructs Prometheus metrics in a uniform way and will allow us to include some of these metrics in telemetry report in future PRs.
 2022-11-10 05:30:06 +00:00
dependabot[bot]
3ff7b5a840 build(deps): bump go.opentelemetry.io/otel/exporters/jaeger (#2557) 2022-11-09 00:05:31 +00:00