* chore(ci): upgraded linter to 1.53.3
This flagged a bunch of unused parameters, so the PR is larger than
usual, but 99% mechanical.
* separate lint CI task
* run Lint in separate CI
We can't enable checklocks on CI yet until
https://github.com/google/gvisor/pull/8807 is merged upstream.
This was tested with private build of checklocks with this patch
applied and the results were clean.
* feat(repository): live cache eviction for persistent lru content cache
* Update internal/cache/persistent_lru_cache.go
Co-authored-by: Ali Dowair <adowair@umich.edu>
* merge the mutex cache into list cache
---------
Co-authored-by: Shikhar Mall <small@kopia.io>
Co-authored-by: Ali Dowair <adowair@umich.edu>
The special case (Windows) has to be checked before "filepath.IsAbs".
The test for internal\ospath now return success.
Co-authored-by: lupusA <lupuapps@gmail.com>
Lint fixes in preparation for moving to Go 1.20
Remove deprecated calls to `rand.Seed`
In Go 1.20 the default generator is seeded randomly at program startup,
which is the desired behavior for these tests.
Remove uses of deprecated rand.Read: replace with calls to rand.Uint64()
Remove deprecated uses of rand.Read in content manager tests and
S3 versioned tests.
Adds a concurrency-safe helpers to provide functionality similar to that
provided by `rand.Read(b []byte) (int, error)`
This simplifies use cases where we intend to replace a manifest uniquely
identified by a set of labels with another one as is the case for
policies.
This helped fix annoying test flake on Windows where the clock is not
guaranteed to move forward when read in quick succession.
This is now passing on Windows:
```
$ go test -timeout 1000s ./internal/server -run TestSourceRefreshesAfterPolicy -count=1000
```
This change adds a new streaming response to the FindManifests API. The
server will deliver the response in chunks of N manifests where N is
requested by the client. This allows the client to process the response
in chunks and improves pipelining of responses.
For now client will hold the entire response in memory since this
is what FindManifests() API currently does. This will be fixed in a
follow up change.
Replaces #2713Fixes#2660
* feat(server): improved server shutdown and integration tests
Added `--shutdown-grace-period` flag to `kopia server start` command
which can be used to specify how long the server will wait for active
connections to finish before forcibly shutting down.
This allowed removal of final out-of-process execution of
during integration tests and the need for `integration-tests` target
which was running the same tests as `tests` but in out-of-process mode.
We thus now have all the test coverage in-process without having to
build and launch `kopia` binary.
* fixed logging
* increase test timeout
* speed up and/or parallelize longest-running tests
Also modified an end-to-end test to also check that these extra mode flags work when snapshotting+restoring.
Manually tested fuse-mount.
Co-authored-by: Luca Citi <lciti@ieee.org>
This manages mapping of metric names to IDs which allows efficient JSON
representation of counter values for each set of metrics where
only values are in the index order.
This will be used in the telemetry protocol and for storing counters in
the repository.
Added test that ensures all metrics registered in a repository have
the corresponding mapping.
Almost all were easy to replace, except ones exposed via JSON which
have been left as-is.
The linter has a cool behavior where it flags attempts to pass
`atomic.Int32` for example by value , which is always a mistake,
say as an argument to `fmt.Sprintf()`
This may be a breaking change for users who rely on particular kopia metrics (unlikely):
- introduced blob-level metrics:
* `kopia_blob_download_full_blob_bytes_total`
* `kopia_blob_download_partial_blob_bytes_total`
* `kopia_blob_upload_bytes_total`
* `kopia_blob_storage_latency_ms` - per-method latency distribution
* `kopia_blob_errors_total` - per-method error counter
- updated cache metrics to indicate particular cache
* `kopia_cache_hit_bytes_total{cache="CACHE_TYPE"}`
* `kopia_cache_hit_total{cache="CACHE_TYPE"}`
* `kopia_cache_malformed_total{cache="CACHE_TYPE"}`
* `kopia_cache_miss_total{cache="CACHE_TYPE"}`
* `kopia_cache_miss_errors_total{cache="CACHE_TYPE"}`
* `kopia_cache_miss_bytes_total{cache="CACHE_TYPE"}`
* `kopia_cache_store_errors_total{cache="CACHE_TYPE"}`
where `CACHE_TYPE` is one of `contents`, `metadata` or `index-blobs`
- reorganized and unified content-level metrics:
* `kopia_content_write_bytes_total`
* `kopia_content_write_duration_nanos_total`
* `kopia_content_compression_attempted_bytes_total`
* `kopia_content_compression_attempted_duration_nanos_total`
* `kopia_content_compression_savings_bytes_total`
* `kopia_content_compressible_bytes_total`
* `kopia_content_non_compressible_bytes_total`
* `kopia_content_after_compression_bytes_total`
* `kopia_content_decompressed_bytes_total`
* `kopia_content_decompressed_duration_nanos_total`
* `kopia_content_encrypted_bytes_total`
* `kopia_content_encrypted_duration_nanos_total`
* `kopia_content_hashed_bytes_total`
* `kopia_content_hashed_duration_nanos_total`
* `kopia_content_deduplicated_bytes_total`
* `kopia_content_read_bytes_total`
* `kopia_content_read_duration_nanos_total`
* `kopia_content_decrypted_bytes_total`
* `kopia_content_decrypted_duration_nanos_total`
* `kopia_content_uploaded_bytes_total`
Also introduced `internal/metrics` framework which constructs Prometheus metrics in a uniform way and will allow us to include some of these metrics in telemetry report in future PRs.
This removes tons of boilerplate code around:
- retry loop
- connection management
- storage registration
* used generics in runInParallel
* introduced generics in freepool
* introduced strong typing for workshare.Pool and workshare.AsyncGroup
* fixed linter error on openbsd
Lack of generics support is blocking various dependency upgrades,
so this unblocks that.
Temporarily disabled `checklocks` linter until it is fixed upstream.
* Update display on repository summary
* Apply throughout app
* Situate units_test
* Update Command Line documentation
* Envar cleanup
* Rename to BytesString
* Restore envar string available for test
* Remove extraneous empty check and restore UIPreferences field for frontend
* PR: config bool cleanup and missed `BaseEnv`s
* Fix lint and test
Benchmarked from macOS client to a Linux server over Wifi connection:
(2-5ms latency)
linux 5.14.8 (1.1 GB) to a clean repository:
Before: 240s After: 27s (90% faster)
Fixes#2372
* feat(repository): added bigmap Set and Map backed by custom on-disk hashtable
* additional test coverage for corner cases
* profile flags
* exclude bigmapbench from code coverage
* refactored based on conversation with Julio
The concern was that values stored on disk are not encrypted.
* bigmap.Map - implements encryption of values
* bigmap.Set - stores keys only, so no encryption necessary
* bigmap.internalMap - used as backing structure for Map and Set
* implemented benchmark with values
* Make callback for upload file completion
Callback does not indicate that a file will be reachable immediately in
the resulting snapshot, but does indicate that the uploader is done
processing the file in some way (either via uploading data or finding a
previous version in the repo) and whether there was an error processing
the file.
* Tests for new FinishedFile callback
Ensure hadErr is properly populated and FinishedFile is called even if
the file was considered cached.
* Refine comment on interface function slightly
* Give callback error instead of bool about error
* Add locks around concurrent accesses in test
* refactor(repository): moved format blob management to separate package
This is completely mechanical, no behavior changes, only:
- moved types and functions to a new package
- adjusted visibility where needed
- added missing godoc
- renamed some identifiers to align with current usage
- mechanically converted some top-level functions into member functions
- fixed some mis-named variables
* refactor(repository): moved content.FormatingOptions to format.ContentFormat
* feat(repository): added `required features` to the repository
This is intended for future compatibility to be able to reliably
stop old kopia client from being able to open a repository when
the old code does not understand new `required feature`.
Required features are checked on startup and periodically using the
same method as upgrade lock, where they will return errors during blob
operations.
* pr feedback
Instead of passing static content.FormattingOptions (and caching it)
we now introduce an interface to provide its values.
This will allow the values to dynamically change at runtime in the
future to support cases like live migration.
