* refactor: extracted complete blob set functions to separate package
* completeset: added more functions
Also treat malformed blob IDs as their own sets for backwards compat.
* testing: refactored logs directory management
* content: fixed index mutex to be shared across all write sessions
added mutex protection during writecontent/refresh race
* testing: upload log artifacts
* content: bump revision number after index has been added
This fixes a bug where manifest manager in another session for
the same open repository may not see a content added, because they
will prematurely cache the incomplete set of contents.
This took 2 weeks to find.
* manifest: improved log output, fixed unnecessary mutex release
* testing: rewrote stress test to be model-based and more precise
Detailed changes:
* Implement Kopia API client
* Implement kopia client
* Implement new persister
* Separate storeLoad and Delete concurrency tests
* Update Store interface to return error for Delete
* Return errors from os.RemoveAll
* Robustness test updates
* Push robustness metadata without writing to fs
* Fix testenv.AssertNoError references
* Update tests to use new kopia persister
* Minor updates to multiclient test cases
* Use require.NoError instead of assertNoError
* Add context to store interface
* Update logging to be less verbose
* Use io instead of ioutil package
* Simplify restore by using object ID
* Accommodate repository.NewWriter signature change
* Improve tests to increase code coverage
* Spelling and error string fixes
* Address lint errors
Co-authored-by: Nick <nick@kasten.io>
* Multiple placeholder expansion
It's inconvenient to only be able to expand a single placeholder file
at a time. Add support to the restore command to expand any number of
shallow placeholder arguments.
* Add placeholder looping
Loop over placeholder targets to expand from within the restore
command.
* Added multiple placeholder expansion test
Added a test to exercise the new code capable of expanding multiple
placeholder arguments to the restore command.
This fixes race condition between index compaction and index refresh
which was causing recent test flakes. This is not likely to happen in
real world as compaction is super rare but test was triggering it very
frequentl.
I was able to reliably repro this with a nearly complete rewrite
of a stress test (coming in separate PR). Without the fix it would fail
in 1-3 attempts, with the fix it succeeded 100 attempts of the new
stress test.
Also improved logging and fixed old terminology leftovers.
When doing a shallow restore, small files might take up less size than
storing the DirectoryEntry metadata. Add a minimum file size flag that
where files below that size will be written directly instead being
represented with shallow placeholders. This improves on #710.
* trivial: refactored indexBlobManager interface
* testlogging: added NewLogger()
* blob: added helper to extract blob IDs
* content: added packIndexBuilder.BuildStable()
This uncovered a bug where we were failing to return the name of
last pack object ID in an index file. Because of padding this was never
possible, but with stable indexes will. Fixed that.
* nit: dead code cleanup
* content: cleaned up committedContextIndex usage, removed unused code
* content: moved fetching index blobs into committedContentIndex
* logging: added logger wrappers for Broadcast and Prefix
* nit: moved max hash size to a named constant
* content: added internal logger
* content: replaced context-based logging with explicit Loggers
This will capture the logger.Logger associated with the context when
the repository is opened and will reuse it for all logs instead of
creating new logger for each log message.
The new logger will also write logs to the internal logger in addition
to writing to a log file/console.
* cli: allow decrypting all blobs whose names start with _
* maintenance: added logs cleanup
* cli: commands to view logs
* cli: log selected command on each write session
* cli: added a flag to create repository with v2 index features
* content: plumb through compression.ID parameter to content.Manager.WriteContent()
* content: expose content.Manager.SupportsContentCompression
This allows object manager to decide whether to create compressed object
or let the content manager do it.
* object: if compression is requested and the repo supports it, pass compression ID to the content manager
* cli: show compression status in 'repository status'
* cli: output compression information in 'content list' and 'content stats'
* content: compression and decompression support
* content: unit tests for compression
* object: compression tests
* testing: added integration tests against v2 index
* testing: run all e2e tests with and without content-level compression
* htmlui: added UI for specifying index format on creation
* cli: additional tests for 'content ls' and 'content stats'
* applied pr suggestions
* cli: Added --max-examples-per-bucket flag to 'kopia snapshot estimate'
Added and cleaned up a bunch of unit tests.
Fixes#1054
* cli: misc tests to increase code coverage of the cli package
* ci: move code coverage run into separate GH job
