This is enabled by `kopia server --ui` and can be viewed in a browser
at http://localhost:51515/
Right now it can only list snapshots and policies (barely).
Motivation:
- Consistent locking/unlocking `c.mu` in `Cache.getEntries`
- Fix CI failure
`Cache.getEntriesFromCache()` was releasing `c.mu` when returning an
entry and not releasing it when returning nil. Previously, the caller
(`Cache.getEntries`) was accordingly not releasing `c.mu` when
`getEntriesFromCache()` was returning an entry, and unlocking `c.mu`
otherwise.
`Cache.getEntries` is the only caller for `getEntriesFromCache`.
After this change c.mu is consistently locked and unlocked in
`getEntries`
* Fixed how blob storage PutBlob errors are handled in content.Manager
In order to guarantee that all index entries have corresponding
pack blobs, we must ensure that `content.Manager.Flush` will
not succeed unless all pending writes have completed.
Added test that simulates various patterns of PutBlock failures and
ensures that data remains durable despite those, assuming all calls
to `WriteContent()` and `Flush()` are retried.
* addressed review feedback
`snapshot gc` marks contents not reachable from the root of any snapshot
as soft-deleted
The algorithm is a mark-and-sweep with parallel iteration of objects.
Currently it stores content IDs and object IDs in a map, so won't scale
to huge repositories, but this can be fixed in the future.
This fixes#110 at least for reasonable repository sizes.
Previously, it was possible for Flush() to miss in-flight writes,
but only when using repository manually since Uploader guarantees
there are no in-flight writes when it completes.
With this change Flush() will guarantee that any pending writes
completed before Flush() has started are guaranteed to be committed
to the repository before Flush() returns.
This was actually a regression introduced in #105.
Added regression test to prevent it from reoccurring.
Previously 'packIndexBuilder' contained both contents that have been
written to packs and the ones that have not.
This change makes it so that 'packIndexBuilder' only contains contents
from flushed packs, but non pending ones. It will help parallelize
writes later.
- separated portions that don't require locking into separate struct
to make it easier to reason about state
- moved iteration-related content to separate file
- parallelized os.Lstat() x 16 (dramatically improves speed)
- discarded unused portions of os.FileInfo (uses 60% less RAM on macOS)
BEFORE:
10:47:03.670 [kopia/localfs] listed 200000 entries in 43.871211686s using 79126528 bytes of heap
After:
10:49:12.439 [kopia/localfs] listed 200000 entries in 1.953018184s using 30515200 bytes of heap
