* feat(snapshots): improved performance when uploading huge files
This is controlled by an upload policy which specifies the size
threshold above which indvidual files are uploaded in parts
and concatenated.
This allows multiple threads to run splitting, hashing, compression
and encryption in parallel, which was previously only possible across
multiple files, but not when a single file was being uploaded.
The default is 2GiB for now, so this feature only kicks in for very
larger files. In the future we may lower this.
Benchmark involved uploading a single 42.1 GB file which was a VM disk
snapshot of fresh Ubuntu installation (fresh EXT4 partition with lots
of zero bytes) to a brand-new filesystem repository on local SSD of
M1 Pro Macbook Pro 2021.
* before: 59-63s (~700 MB/s)
* after: 15-17s (~2.6 GB/s)
* additional test to ensure files are really e2e readable
* object: refactored Open() and VerifyObject() to be stateless
(no code movement yet to facilitate review)
* mechanical: moved function more appropriate files
* object: remove object manager tracing which was unused
* linter: upgraded to 1.33, disabled some linters
* lint: fixed 'errorlint' errors
This ensures that all error comparisons use errors.Is() or errors.As().
We will be wrapping more errors going forward so it's important that
error checks are not strict everywhere.
Verified that there are no exceptions for errorlint linter which
guarantees that.
* lint: fixed or suppressed wrapcheck errors
* lint: nolintlint and misc cleanups
Co-authored-by: Julio López <julio+gh@kasten.io>
Also introduced strongly typed content.ID and manifest.ID (instead of string)
This aligns identifiers across all layers of repository:
blob.ID
content.ID
object.ID
manifest.ID
