mirror/kopia
1
0
Fork 0
mirror of https://github.com/kopia/kopia.git synced 2026-05-11 00:04:46 -04:00
Code Issues Packages Projects Releases Wiki Activity
3,660 Commits 6 Branches 88 Tags
1c5c4e256839dabec00eb86ee4801dfd198efe4c
Commit Graph

18 Commits

Author SHA1 Message Date
Jarek Kowalski
09415e0c7d chore(ci): upgraded to go 1.22 (#3746) 
Upgrades go to 1.22 and switches to new-style for loops

---------

Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
 2024-04-08 09:52:47 -07:00
Jarek Kowalski
bbbf65de8c feat(repository): big reduction in memory usage during decompression (#3780) 
This has huge performance benefits for `zstd` and `pgzip`

BEFORE:

```
  0. s2-parallel-4              35.1 MB      8.9 GB/s     688      4.6 GB
  1. s2-parallel-8              35.1 MB      8.1 GB/s     688      4.6 GB
  2. s2-default                 35.1 MB      8 GB/s       689      4.6 GB
  3. s2-better                  33.7 MB      7.2 GB/s     689      4.6 GB
  4. zstd-fastest               18.4 MB      6.3 GB/s     22281    1.2 GB
  5. zstd-better-compression    16.4 MB      5.3 GB/s     17957    2 GB
  6. zstd                       18.1 MB      4.8 GB/s     17711    2.1 GB
  7. deflate-best-speed         20.6 MB      3.7 GB/s     4672     6.4 MB
  8. deflate-best-compression   18.4 MB      3.5 GB/s     8352     6.5 MB
  9. pgzip-best-speed           20.6 MB      3.5 GB/s     27871    679.1 MB
 10. deflate-default            19.7 MB      3.5 GB/s     9152     6.4 MB
 11. pgzip-best-compression     18.4 MB      3.3 GB/s     29494    679.1 MB
 12. pgzip                      19.7 MB      3.3 GB/s     32835    679.2 MB
 13. gzip                       19.3 MB      2.9 GB/s     1427888  4.4 GB
 14. gzip-best-compression      18.5 MB      2.7 GB/s     1489968  4.4 GB
 15. gzip-best-speed            21.6 MB      2.3 GB/s     1373330  4.4 GB
 ```

AFTER:

```
  0. lz4                        33.9 MB      13.4 GB/s    261      4.4 GB (deprecated)
  1. s2-default                 35.1 MB      10.2 GB/s    275      4.3 GB
  2. s2-better                  33.7 MB      9.7 GB/s     274      4.3 GB
  3. s2-parallel-8              35.1 MB      9.7 GB/s     278      4.3 GB
  4. s2-parallel-4              35.1 MB      9.4 GB/s     288      4.3 GB
  5. zstd-fastest               18.4 MB      6.9 GB/s     1192     90.1 MB
  6. zstd-better-compression    16.4 MB      5.8 GB/s     1162     158.8 MB
  7. zstd-best-compression      16.2 MB      5.7 GB/s     1183     159.3 MB (deprecated)
  8. zstd                       18.1 MB      5.2 GB/s     1103     158 MB
  9. deflate-best-speed         20.6 MB      3.8 GB/s     4672     6.4 MB
 10. deflate-best-compression   18.4 MB      3.6 GB/s     8352     6.5 MB
 11. pgzip-best-compression     18.4 MB      3.2 GB/s     28489    75 MB
 12. pgzip-best-speed           20.6 MB      3.2 GB/s     26725    74.9 MB
 13. pgzip                      19.7 MB      3.1 GB/s     31889    75.1 MB
 14. gzip-best-compression      18.5 MB      3.1 GB/s     1489269  4.4 GB
 15. deflate-default            19.7 MB      2.6 GB/s     9164     6.4 MB
 16. gzip-best-speed            21.6 MB      2.4 GB/s     1372639  4.4 GB
 17. gzip                       19.3 MB      2.2 GB/s     1427205  4.4 GB
```
 2024-04-04 18:42:55 -07:00
Jarek Kowalski
51dcaa985d chore(ci): upgraded linter to 1.48.0 (#2294) 
Mechanically fixed all issues, added `lint-fix` make target.
 2022-08-09 06:07:54 +00:00
Jarek Kowalski
0985b80488 feat(ui): support for deprecation of certain algorithms (#2122) 
Some compression algorithms are not recommended because they
allocate disproportionate amounts of memory. They are still
possible to use, just marked as NOT RECOMMENDED in the UI.
 2022-07-03 19:06:14 +00:00
Jarek Kowalski
4548160508 feat(repository): automatically compress all kopia internal metadata using zstd-fastest (#1880) 
Fixes #1541
 2022-04-04 01:43:53 +00:00
Jarek Kowalski
32ed220a6c build(lint): enabled gochecknoglobals and tagged existing globals (#1664) 2022-01-15 12:54:56 -08:00
Jarek Kowalski
35d0f31c0d huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244) 
This helps recycle buffers more efficiently during snapshots.
Also, improved memory tracking, enabled profiling flags and added pprof
by default.
 2021-08-20 08:45:10 -07:00
Jarek Kowalski
9e059a1277 upgraded linter to 1.41.0 (#1144) 2021-06-16 19:44:55 -07:00
Jarek Kowalski
38d01d4c4d compression: added 'lz4' and 'deflate' methods (#1077) 
* compression: add mapping from HeaderID to Name

* compression: minor refactoring

* compression: added 'lz4' based on 'github.com/pierrec/lz4'

* compression: added 'deflate' support

* cli: 'kopia benchmark compression' improvements
 2021-05-16 14:42:56 -07:00
Jarek Kowalski
30ca3e2e6c Upgraded linter to 1.40.1 (#1072) 
* tools: upgraded linter to 1.40.1

* lint: fixed nolintlint vionlations

* lint: disabled tagliatele linter

* lint: fixed remaining warnings
 2021-05-15 12:12:34 -07:00
Jarek Kowalski
7c108930ef testing: ensure tests are releasing all buffer pools to reduce memory usage, we had huge leaks (#895) 
* testing: ensure tests are releasing all buffer pools to reduce memory usage, we had huge leaks

* object: reduced complexity and memory usage of TestEndToEndReadAndSeekWithCompression

* manifest: more test fixes

* trivial: update comment

Co-authored-by: Julio López <julio+gh@kasten.io>
 2021-03-18 06:40:33 -07:00
Jarek Kowalski
388fc32a65 compression: added benchmark to measure impact of library upgrades (#892) 
Usage:

$ go test -run NONE -bench=. ./repo/compression/... | tee file.txt

To compare: https://pkg.go.dev/golang.org/x/tools/cmd/benchcmp
 2021-03-16 21:47:14 -07:00
Jarek Kowalski
1f3b8d4da4 upgrade linter to 1.35 (#786) 
* lint: added test that enforces Makefile and GH action linter versions are in sync
* workaround for linter gomnd problem - https://github.com/golangci/golangci-lint/issues/1653
 2021-01-16 18:21:16 -08:00
Jarek Kowalski
9a6dea898b Linter upgrade to v1.30.0 (#526) 
* fixed godot linter errors
* reformatted source with gofumpt
* disabled some linters
* fixed nolintlint warnings
* fixed gci warnings
* lint: fixed 'nestif' warnings
* lint: fixed 'exhaustive' warnings
* lint: fixed 'gocritic' warnings
* lint: fixed 'noctx' warnings
* lint: fixed 'wsl' warnings
* lint: fixed 'goerr113' warnings
* lint: fixed 'gosec' warnings
* lint: upgraded linter to 1.30.0
* lint: more 'exhaustive' warnings

Co-authored-by: Nick <nick@kasten.io>
 2020-08-12 19:28:53 -07:00
Jarek Kowalski
8d452a8285 performance: improvements to object manager (#336) 
- added pooled splitters and ability to reset them without having to recreate
- added support for caller-provided compressor output to be able to pool it
- added pooling of compressor instances, since those are costly
 2020-03-13 08:56:18 -07:00
Jarek Kowalski
514df69afa performance: added wrapper around io.Copy() 
this pools copy buffers so they can be reused instead of throwing away
after each io.Copy()
 2020-03-10 21:52:30 -07:00
Jarek Kowalski
ac70a38101 lint: upgraded to 1.22.2 and make lint issues a build failure 
fixed or silenced linter warnings, mostly due to magic numeric constants
 2020-01-03 16:39:30 -08:00
Jarek Kowalski
2ba4e83cef moved all compression to separate package and sanitized identifiers 2019-12-10 23:25:28 -08:00