* object: added AsyncWrites to ObjectWriter, which improves performance of uploading of a single file
Fixes#351
Co-Authored-By: Julio López <julio+gh@kasten.io>
* performance: added buf.Pool which can be used to manage ephemeral buffers for encryption and compression
* repo: switched object writer to buf.Pool
* content: switched encryption to use buf.Pool
* object: switched compression to use buf.Pool
* testing: added missing content manager Close()
Previously we would store special field Payload for contents
that were added but never flushed to the store and it was not
encrypted. This required special handling different for pending
vs flushed contents.
This change maintains pending pack buffer ready to be flushed
and appends encrypted contents to it, which avoids data copying.
The buffers are pooled to avoid allocations.
Add an implementation of the metadata store using kopia snapshots and restores to manage persistence of walk metadata. Metadata are stored to and retrieved from the store, and a mechanism for persisting the store is implemented using kopia snapshots.
- added pooled splitters and ability to reset them without having to recreate
- added support for caller-provided compressor output to be able to pool it
- added pooling of compressor instances, since those are costly
non-optimized (0.5.0)
0. BLAKE2B-256-128 AES256-GCM-HMAC-SHA256 644.9 MiB / second
before this change:
0. BLAKE2B-256-128 AES256-GCM-HMAC-SHA256 655.9 MiB / second
after (this change):
0. BLAKE2B-256-128 AES256-GCM-HMAC-SHA256 781.5 MiB / second
* performance: plumbed through output buffer to encryption and hashing, so that the caller can pre-allocate/reuse it
* testing: fixed how we do comparison of byte slices to account for possible nils, which can be returned from encryption
Upload: reduced memory usage during uploads
Replaced custom work item management with much simpler solution.
Upload: added pooling of upload buffers
This reduces a lot of allocations and GC pressure.
In a test of 2 dirs x 100K files x 100 bytes each, allocated bytes
went from 27 GB to 20 GB.
This improves #93
* repo: added some initial metrics using OpenCensus
* cli: added flags to expose Prometheus metrics on a local endpoint
`--metrics-listen-addr=localhost:X` exposes prometheus metrics on
http://localhost:X/metrics
Also, kopia server will automatically expose /metrics endpoint on the
same port it runs as, without authentication.
Snapshotter interface describes an entity that can create,
restore, and delete snapshots, as well as manage a repository.
Add kopia implementation of the snapshotter interface.
Motivation: Allow time injection for (unit) tests, to more easily test and
verify time-dependent invariants.
Add time injection support for:
* repo.Manager
* manifest.Manager
* snapshot.Uploader
Then, wire up to these components. The content.Manager already had support for
time injection, but was not wired up from the time function passed to repo creation.
Add an internal/faketime package for testing. Mainly code movement from testing
code in the repo/content package. Motivation: make it available to other packages
outside content Also, add simple tests for faketime functions.
