- expand command flag description for clarification
- include blob id in blob get error in the cache
- nit: remove unused BOTO_PATH
- nit: fix comment
- cleanup: remove unnecessary function declaration in interface
- leverage 'testify' to simplify test
We were incorrectly assigning the same content cache key regardless of
the storage format of the content (compression format, etc.)
This means that if a content has both compressed and non-compressed
entry in the index, it would sometimes get cached incorrectly
and the subsequent reads would fail.
Clearing the cache would fix the issue - this change appends
format-specific suffix to cache keys, so that clearing the cache is not
needed.
Huge thanks to Mark Derricutt who helped get to the bottom of this
on Slack.
Fix#1843
From https://github.com/google/gvisor/tree/master/tools/checklocks
This will perform static verification that we're using
`sync.Mutex`, `sync.RWMutex` and `atomic` correctly to guard access
to certain fields.
This was mostly just a matter of adding annotations to indicate which
fields are guarded by which mutex.
In a handful of places the code had to be refactored to allow static
analyzer to do its job better or to not be confused by some
constructs.
In one place this actually uncovered a bug where a function was not
releasing a lock properly in an error case.
The check is part of `make lint` but can also be invoked by
`make check-locks`.
* feat(repository): ensure we don't run parallel fetches for the same blob IDs
Fixed a bunch of test flakes.
* fixed race condition, limit the size of mutex cache by using LRU
This allows KopiaUI server to start when the repository directory
is not mounted or otherwise unavailable. Connection attempts will
be retried indefinitely and user will see new `Initializing` page.
This also exposes `Open` and `Connect` as tasks allowing the user to see
logs directly in the UI and cancel the operation.
This is done by protecting newly added cache items from being swept for
X amount of time where X defaults to:
* `metadata` - 24 hours (new)
* `data` - 10 min (new)
* `indexes` - 1 hours (same as today)
Fixes#1540
* blob: changed default shards from {3,3} to {1,3}
Turns out for very large repository around 100TB (5M blobs),
we end up creating max ~16M directories which is way too much
and slows down listing. Currently each leaf directory only has a handful
of files.
Simple sharding of {3} should work much better and will end up creating
directories with meaningful shard sizes - 12 K files per directory
should not be too slow and will reduce the overhead of listing by
4096 times.
The change is done in a backwards-compatible way and will respect
custom sharding (.shards) file written by previous 0.9 builds
as well as older repositories that don't have the .shards file (which
we assume to be {3,3}).
* fixed compat tests
The dual time measurement is described in
https://go.googlesource.com/proposal/+/master/design/12914-monotonic.md
The fix is to discard hidden monotonic time component of time.Time
by converting to unix time and back.
Reviewed usage of clock.Now() and replaced with timetrack.StartTimer()
when measuring time.
The problem in #1402 was that passage of time was measured using
the monotonic time and not wall clock time. When the computer goes
to sleep, monotonic time is still monotonic while wall clock time makes
a leap when the computer wakes up. This is the behavior that
epoch manager (and most other compontents in Kopia) rely upon.
Fixes#1402
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
This was caused by additional resolution of path names only done in UI,
which caused \\hostname\share to be treated as relative and resolved
against the home directory.
Fixes#1385Fixes#1362
Removed Warning, Notify and Fatal:
* `Warning` => `Error` or `Info`
* `Notify` => `Info`
* `Fatal` was never used.
Note that --log-level=warning is still supported for backwards
compatibility, but it is the same as --log-level=error.
Co-authored-by: Julio López <julio+gh@kasten.io>
* cache: refactored reusable portion of cache into separate package
* repo: plumbed through caching for remote repository clients
* repo: plumb through cache in the unit tests
* cache: ensure we only allow absolute cache paths, fixed cache path resolution for remote repositories
