* fix(repository): fixed handling of content.Info
Previously content.Info was an interface which was implemented by:
* index.InfoStruct
* index.indexEntryInfoV1
* index.indexEntryInfoV2
The last 2 implementations were relying on memory-mapped files
which in rare cases could be closed while Kopia was still processing
them leading to #2599.
This changes fixes the bug and strictly separates content.Info (which
is now always a struct) from the other two (which were renamed as
index.InfoReader and only used inside repo/content/...).
In addition to being safer, this _should_ reduce memory allocations.
* reduce the size of content.Info with proper alignment.
* pr feedback
* renamed index.InfoStruct to index.Info
* refactor(repository): ensure we always parse content.ID and object.ID
This changes the types to be incompatible with string to prevent direct
conversion to and from string.
This has the additional benefit of reducing number of memory allocations
and bytes for all IDs.
content.ID went from 2 allocations to 1:
typical case 32 characters + 16 bytes per-string overhead
worst-case 65 characters + 16 bytes per-string overhead
now: 34 bytes
object.ID went from 2 allocations to 1:
typical case 32 characters + 16 bytes per-string overhead
worst-case 65 characters + 16 bytes per-string overhead
now: 36 bytes
* move index.{ID,IDRange} methods to separate files
* replaced index.IDFromHash with content.IDFromHash externally
* minor tweaks and additional tests
* Update repo/content/index/id_test.go
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* Update repo/content/index/id_test.go
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* pr feedback
* post-merge fixes
* pr feedback
* pr feedback
* fixed subtle regression in sortedContents()
This was actually not producing invalid results because of how base36
works, just not sorting as efficiently as it could.
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* feat(cli): implementation for 'kopia snapshot fix'
This allows modifications and fixes to the snapshots after they have
been taken.
Supported are:
* `kopia snapshot fix remove-invalid-files [--verify-files-percent=X]`
Removes all directory entries where the underlying files cannot be
read based on index analysis (this does not read the files, only index
structures so is reasonably quick).
`--verify-files-percent=100` can be used to trigger full read for
all files.
* `kopia snapshot fix remove-files --object-id=<object-id>`
Removes the object with a given ID from the entire snapshot tree.
Useful when you accidentally snapshot a sensitive file.
* `kopia snapshot fix remove-files --filename=<wildcard>`
Removes the files with a given name from the entire snapshot tree.
Useful when you accidentally snapshot a sensitive file.
By default all snapshots are analyzed and rewritten. To limit the scope
use:
--source=user@host:/path
--manifest-id=manifestID
By default the rewrite operation writes new directory entries but
does not replace the manifests. To do that pass `--commit`.
Related #1906Fixes#799
reorganized CLI per PR suggestion
* additional logging for diff command
* added Clone() method to snapshot manifst and directory entry
* added a comprehensive test, moved DirRewriter to separate file
* pr feedback
* more pr feedback
* improved logging output
* disable test in -race configuration since it's way to slow
* pr feedback
