Commit Graph

3 Commits

Author SHA1 Message Date
atom
c5efed01f4 feat(cli): Support displaying storage values in base-2 [#2492] (#2502)
* Update display on repository summary

* Apply throughout app

* Situate units_test

* Update Command Line documentation

* Envar cleanup

* Rename to BytesString

* Restore envar string available for test

* Remove extraneous empty check and restore UIPreferences field for frontend

* PR: config bool cleanup and missed `BaseEnv`s

* Fix lint and test
2022-10-24 19:00:36 -07:00
Jarek Kowalski
68b8afd43f feat(snapshots): improved performance when uploading huge files (#2064)
* feat(snapshots): improved performance when uploading huge files

This is controlled by an upload policy which specifies the size
threshold above which indvidual files are uploaded in parts
and concatenated.

This allows multiple threads to run splitting, hashing, compression
and encryption in parallel, which was previously only possible across
multiple files, but not when a single file was being uploaded.

The default is 2GiB for now, so this feature only kicks in for very
larger files. In the future we may lower this.

Benchmark involved uploading a single 42.1 GB file which was a VM disk
snapshot of fresh Ubuntu installation (fresh EXT4 partition with lots
of zero bytes) to a brand-new filesystem repository on local SSD of
M1 Pro Macbook Pro 2021.

* before: 59-63s (~700 MB/s)
* after: 15-17s  (~2.6 GB/s)

* additional test to ensure files are really e2e readable
2022-06-24 07:38:07 +00:00
Jarek Kowalski
92683c5a5e feat(snapshots): support for controlling upload parallelism via policies (#1850) 2022-03-26 19:20:49 +00:00