* refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* chore: remove //nolint:gosec for os.ReadFile
At the time of this commit, the G304 rule of gosec does not include the
`os.ReadFile` function. We remove `//nolint:gosec` temporarily until
https://github.com/securego/gosec/pull/706 is merged.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* upload: do not launch error group when doing non-parallel upload
* encryption: refactored code to reduce slice allocation, this uncovered a minor subtle bug in index recovery due to manual memory management
* content: fixed use-after-free bug during index recovery that started failing test after encryption memory management improvements
* gather: tweaked retention and chunk sizes
* gather: added typicalContiguousAllocator with 8MB chunks for default configuration
* content: fixed data race in IterateUnreferencedBlobs
* upload: fixed data race between uploader and estimator
* testing: fixed data race in repo/blob/logging test
* makefile: run tests on CI/linux/amd64 with -race
* robustness: fixed test race
* content: fixed data race getContentDataUnlocked that triggers TestParallelWrites - looks scary but in practice very hard to trigger in real life and does not cause data corruption
* testing: reduce test complexity under race detector
* server: fixed minor race in refreshStatus()
* testing: reduced depth of sharedTestDataDir2
* ci: run race detector in separate job
* ci: run unit test race detector in parallel to integration tests
* linter: upgraded to 1.33, disabled some linters
* lint: fixed 'errorlint' errors
This ensures that all error comparisons use errors.Is() or errors.As().
We will be wrapping more errors going forward so it's important that
error checks are not strict everywhere.
Verified that there are no exceptions for errorlint linter which
guarantees that.
* lint: fixed or suppressed wrapcheck errors
* lint: nolintlint and misc cleanups
Co-authored-by: Julio López <julio+gh@kasten.io>
, where blob.Storage.PutBlob gets a list of slices and writes them sequentially
* performance: added gather.Bytes and gather.WriteBuffer
They are similar to bytes.Buffer but instead of managing a single
byte slice, they maintain a list of slices that and when they run out of
space they allocate new fixed-size slice from a free list.
This helps keep memory allocations completely under control regardless
of the size of data written.
* switch from byte slices and bytes.Buffer to gather.Bytes.
This is mostly mechanical, the only cases where it's not involve blob
storage providers, where we leverage the fact that we don't need to
ever concatenate the slices into one and instead we can do gather
writes.
* PR feedback
