`snapshot gc` marks contents not reachable from the root of any snapshot
as soft-deleted
The algorithm is a mark-and-sweep with parallel iteration of objects.
Currently it stores content IDs and object IDs in a map, so won't scale
to huge repositories, but this can be fixed in the future.
This fixes#110 at least for reasonable repository sizes.
This puts all content blocks with non-empty prefix into starting with
`q` instead of `p`. This neatly separates all data (p) from metadata
(q) at the storage level and allows different storage policies, since
most data is not going to be ever accessed ever, but metadata is going
to be read a lot..
We can more aggressively cache contents from `q`.
Repository.Token() generates a base64-encoded token that can
be stored in password manager that fully describes repository connection
information (blob.ConnectionInfo) and optionally a password.
Use `kopia repo status -t` to print the token.
Use `kopia repo status -t -s` to print the token that also includes
repository password.
Use `kopia repo connect from-config --token T` to reconnect using the
token.
Uses go/ssh and pkg/sftp as building blocks and implements the common
sharded.Storage interface, shared between the filesystem and webdav
providers.
A couple of notes:
- The provider assumes the user has a working public/private key
connection to the ssh server.
No other authentication method is supported
- The repository path must exist on the server
- (testing related) The pkg/sftp server doesn't offer a way to set a
server filesystem root, so, during testing, it runs from the local
directory which is repo/blob/sftp. So the tests leave some debris
behind. Additionally, that's the reason why id_rsa and known_hosts
are there at all.
- Encrypted keyfiles are currently not supported (but it could be done)
This cleans up the code a lot and removes many ugly hacks.
The performance is pretty reasonable and with separate metadata cache it's likely to stay that way.
Also introduced strongly typed content.ID and manifest.ID (instead of string)
This aligns identifiers across all layers of repository:
blob.ID
content.ID
object.ID
manifest.ID
This updates the terminology everywhere - blocks become blobs and
`storage.Storage` becomes `blob.Storage`.
Also introduced blob.ID which is a specialized string type, that's
different from CABS block ID.
Also renamed CLI subcommands from `kopia storage` to `kopia blob`.
While at it introduced `block.ErrBlockNotFound` and
`object.ErrObjectNotFound` that do not leak from lower layers.
The splitter in question was depending on
github.com/silvasur/buzhash which is not licensed according to FOSSA bot
Switched to new faster implementation of buzhash, which is
unfortunately incompatible and will split the objects in different
places.
This change is be semi-breaking - old repositories can be read, but
when uploading large objects they will be re-uploaded where previously
they would be de-duped.
Also added 'benchmark splitters' subcommand and moved 'block cryptobenchmark'
subcommand to 'benchmark crypto'.
Julio Lopez