Commit Graph

18 Commits

Author SHA1 Message Date
Shikhar Mall
2857c4831a storage api put-blob retention options (#1511)
* storage api put-blob retention options

Co-authored-by: Shikhar Mall <shikhar@kasten.io>
2021-11-15 19:46:42 -08:00
Jarek Kowalski
669b8515e1 epoch: eliminate redundant ListBlobs() calls in epoch.Manager (#1492) 2021-11-06 16:44:45 -07:00
Jarek Kowalski
a0cfa2556f introduced structural debug logging and optional JSON output (#1475)
* logging: added Logger.Debugw(message, key1, value1, ..., keyN, valueN)

This is based on ZAP and allows structural logs to be emitted.

* cli: added --json-log-console and --json-log-file flags

* logging: updated storage logging wrapper to use structural logging

* pr feedback
2021-11-03 21:57:37 -07:00
Jarek Kowalski
3a40cc621f epoch: do not fail on moving deletion watermark backwards (#1422)
This is a legitimate possiblity if someone uses `--safety=none` followed
by regular maintenance.

Fixes #1312
2021-10-19 08:17:58 -07:00
Jarek Kowalski
d5c619f98d epoch: additional logging to get to the bottom of #1402 (#1417) 2021-10-18 22:39:18 -07:00
Jarek Kowalski
24d8eb75da epoch: increase epoch time and cleanup safety margin (#1407)
Reduces likelihood of #1402 until we fix the root cause.
2021-10-17 22:31:56 -07:00
Jarek Kowalski
7f98c49edc epoch: when doing background compaction and cleanup work, ignore cancelation signal (#1406)
This would frequently prevent the actual compaction from being done
leading to performance degradation.
2021-10-17 20:14:25 -07:00
Jarek Kowalski
4a47bc3210 logging: switched from go-logging to zap (#1376)
This is much more efficient in terms of memory allocations
and speeds up backup due to less GC pressure.

Fixes #1345
2021-10-12 22:52:24 -07:00
Jarek Kowalski
7e68d8e4c1 Consolidated format version flags (#1284) 2021-09-08 18:44:03 -07:00
Jarek Kowalski
35d0f31c0d huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)
This helps recycle buffers more efficiently during snapshots.
Also, improved memory tracking, enabled profiling flags and added pprof
by default.
2021-08-20 08:45:10 -07:00
Jarek Kowalski
b96623f8a7 epoch: simplified TestIndexEpochManager_DeletionFailing test (#1230) 2021-08-02 22:24:33 -07:00
Jarek Kowalski
d6d9a1fb5f Maintenance improvements for epoch-based index structures (#1225)
* testing: KOPIA_TEST_LOG_OUTPUT logs subcommand outputs

* cli: additional flags for 'blob list'

* Makefile: run all tests against epoch-based index manager

* epoch: added support for deletion watermark, which keeps track of latest maintenance which dropped index entries

* content: added deletion watermark to content manager

* maintenance: improved maintenance without safety to force rewrites

* maintenance: skip quick maintenance when epoch manager is enabled

* maintenance: do not enable quick maintenance when epoch manager is used

* testing: skip TestIndexOptimize when running against epoch manager-backed index strutures
2021-08-02 21:08:54 -07:00
Jarek Kowalski
47bc640c00 epoch: handling of canceled context on refresh (#1178) 2021-07-08 08:38:31 -07:00
Jarek Kowalski
62ad437bb6 Implemented epoch-based index manager (#1174)
* epoch: misc fixes and logging

* blob: misc helpers

* cli: removed useless 'repository upgrade', replaced by 'repository set-parameters'

* content: implemented indexBlobManagerV1 which uses epoch manager

* cli: commands to manipulate repository epoch parameters

* cli: commands to examine epoch-based indexes

* content: added test suite that uses epoch-based index manager

* content: fixed a ton of test data races caused by sharing blobtesting.DataMap

* cli: additional tests and validation for 'repository set-params'

* testing: replaced the use of suite with our own, since suite is not parallelizable
2021-07-06 21:38:08 -07:00
Jarek Kowalski
5642a8a521 refactor: extracted complete blob set functions to separate package (#1175)
* refactor: extracted complete blob set functions to separate package

* completeset: added more functions

Also treat malformed blob IDs as their own sets for backwards compat.
2021-07-05 17:08:40 -07:00
Jarek Kowalski
0756dee6d5 More epoch manager work (#1147)
* content: added packIndexBuilder sharding

* epoch manager improvements
2021-06-19 16:48:45 -07:00
Jarek Kowalski
8b0296cdf2 Misc index blob manager refactorings (#1138)
* content: extracted encryptedBlobMgr component

* content: renamed files

* content: refactored ParseIndexBlob

* epoch: adjusted API to return blob.Metadata

* content: removed IndexBlobReader interface

* content: cleaned up indexBlobManager API
2021-06-13 18:52:49 -07:00
Jarek Kowalski
7735fcc525 epoch: added epoch manager + unit tests (#1128) 2021-06-11 18:57:21 -07:00