Jarek Kowalski
35d0f31c0d
huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository ( #1244 )
...
This helps recycle buffers more efficiently during snapshots.
Also, improved memory tracking, enabled profiling flags and added pprof
by default.
2021-08-20 08:45:10 -07:00
Jarek Kowalski
8c2e4ff799
testing: simplified S3 test to fix error caused by minio behavior change
2021-07-17 20:01:00 -07:00
Jarek Kowalski
4c2f52a2e3
Rclone and testing improvements ( #1202 )
...
* sharded: added parallel iteration of blobs to improve performance
* retry: reduce first retry delay 1s->100ms
* testing: additional assertions for blob storage testing
* rclone: testing cleanup improvements, re-enabled OneDrive
* cli: added --list-parallelism parameter to fs,webdav,sftp and rclone
* sharded: added dedicated test
2021-07-17 16:04:51 -07:00
Jarek Kowalski
6277fa27c9
content: refactored own writes cache and list cache into blob.Storage wrappers ( #1133 )
...
added blob.Storage.FlushCaches method.
2021-06-12 19:22:25 -07:00
Jarek Kowalski
a5fc1f57e0
testing: allow bigger time drift between GetMetadata and ListBlobs - appears to be needed by Wasabi
2021-04-06 08:57:52 -07:00
Jarek Kowalski
d0f2ef53d7
blob: improved startup error handling of rclone and webdav PutBlob race ( #915 )
...
* added framework for unit testing against remote real rclone remotes,
added google drive backend
* added parallelism to blobtesting which revealed some races during
PutBlob with WebDAV.
2021-03-28 08:26:35 -07:00
Jarek Kowalski
b8e89c2808
s3: refactored testing to support multiple S3-compatible providers ( #914 )
...
The credentials are passed via JSON matching s3.Options struct
Fixed some verification issue with Wasabi.
2021-03-26 14:33:20 -07:00
Jarek Kowalski
b6e68fa28a
Fixed few coverage flakes ( #872 )
...
* blobtesting: coverage for GetMetadata() returning ErrNotFound
* content: additional direct coverage for diskCommittedContentIndexCache
2021-03-07 00:03:20 -08:00
Jarek Kowalski
9a6dea898b
Linter upgrade to v1.30.0 ( #526 )
...
* fixed godot linter errors
* reformatted source with gofumpt
* disabled some linters
* fixed nolintlint warnings
* fixed gci warnings
* lint: fixed 'nestif' warnings
* lint: fixed 'exhaustive' warnings
* lint: fixed 'gocritic' warnings
* lint: fixed 'noctx' warnings
* lint: fixed 'wsl' warnings
* lint: fixed 'goerr113' warnings
* lint: fixed 'gosec' warnings
* lint: upgraded linter to 1.30.0
* lint: more 'exhaustive' warnings
Co-authored-by: Nick <nick@kasten.io >
2020-08-12 19:28:53 -07:00
Jarek Kowalski
8c4fb53c96
blob: support for GetMetadata() to get server-side timestamp and blob length ( #440 )
2020-05-18 11:06:34 -07:00
Jarek Kowalski
5f96b0240a
testing: added retry helper
2020-03-09 21:34:10 -07:00
Jarek Kowalski
8e812b76c0
blob: added retries to Filesystem provider, fixes #249 ( #251 )
...
Wrote a test first which failed 100% on Windows. After adding retries it passed 20 times in a row, execution time is ~10s.
Fixes #249
2020-02-19 13:17:47 -08:00
Jarek Kowalski
6217df1a87
lint: switched to 1.21 and fixed a ton of whitespace issues discovered
...
by new wsl linter
2019-11-26 06:49:49 -08:00
Jarek Kowalski
54edb97b3a
refactoring: renamed repo/block to repo/content
...
Also introduced strongly typed content.ID and manifest.ID (instead of string)
This aligns identifiers across all layers of repository:
blob.ID
content.ID
object.ID
manifest.ID
2019-06-01 22:24:19 -07:00
Jarek Kowalski
9e5d0beccd
refactoring: renamed storage.Storage to blob.Storage
...
This updates the terminology everywhere - blocks become blobs and
`storage.Storage` becomes `blob.Storage`.
Also introduced blob.ID which is a specialized string type, that's
different from CABS block ID.
Also renamed CLI subcommands from `kopia storage` to `kopia blob`.
While at it introduced `block.ErrBlockNotFound` and
`object.ErrObjectNotFound` that do not leak from lower layers.
2019-06-01 14:10:35 -07:00
