Add comparer interface which gathers data on a path and
compares that data to a new path, returning error if the path
differs in any way from the input data. The details of what
constitutes a difference is left to the implementation.
FSWalker implementation uses Walk and Report to do the data
gathering and comparison. Filters are applied to sort out any
differences that might be expected (e.g. ctime, atime, mtime,
rename of root directory after restore).
* Ignore read errors based on policy settings
Added an error handling policy section. Can independently control error handling for directory and file read errors, toggle-able from the `policy set` command to either "true", "false", or "inherit". If any read error is hit, the error handling will check the effective policy on whether to ignore it or not. Currently there is no differentiation between read error types, though in the future we may want to add the `errors.Is(err, os.ErrPermission)` conditional.
Fix was implemented such that the policy ignores read errors ONLY on child entries of the source. So a snapshot will still fail if the source root directory itself can't be read, but you can ignore the error if a file or a subdirectory in the snapshot source root can't be read. I did this to address some otherwise strange behavior where you would successfully snapshot (because you ignored the error), but couldn't restore that snapshot because nothing really happened during the operation.
Temporary workaround for compile issues on MacOS and Windows due
to upstream fswalker bug. Only build the reporter and walker
packages as GOOS=linux for now.
Adds a wrapper around `Walk` that takes a Policy (protobuf definition) and performs a walk using it as configuration. The resulting Walk struct pointer is returned. The only exported functionality is unfortunately to read the Policy as a protobuf text file, so the implementation creates a temporary policy file whose lifetime is the duration of the call.
Adds a wrapper around the the FSWalker reporter `Compare` functionality. Takes a config file and two Walk pointers and compares the walks, returning the pb-defined Report struct. Again, the only exported functionality for reading config information is to read it as a protobuf text file. Creates a temporary config file, whose lifetime is the duration of the call, to pass in to the fswalker function.
There's now one target, `travis-release` that can be run
locally, or on Travis CI except for:
- code signing (Travis on non-PR runs)
- publishing artifacts to GH releases (Travis on tagged releases)
- creating long-term repository for testing (Travis on tagged releases)
Adding a helper library that wraps fio execution. This is the basic initial check-in that implements the runners, configs, and a single WriteFiles helper. It should be enough to unblock subsequent tasks that will use fio to generate data sets for kopia snapshot verification. More helper workloads can be added as needed.
In this implementation the tests will all skip from test main if the `FIO_EXE` env variable is not set. Adding fio to the CI environment will be addressed as a separate PR.
Tracking progress in issue https://github.com/kopia/kopia/issues/179
Those will make it possible to securely host 'kopia server' embedded
in a desktop app that runs in the background and can access UI.
- added support for using and generating TLS certificates
- added /api/v1/shutdown API to remotely trigger server shutdown
- added support for automatically shutting down server if no requests
arrive in certain amount of time
- added support for generating and printing random password to STDERR
TLS supports 3 modes:
1. serve TLS using externally-provided cert/key PEM files
2. generate & write PEM files, then serve TLS using them
3. generate and use emphemeral cert/key (prints SHA256 fingerprint)
Adding test to probe snapshot failure. Tests snapshot of non-existent source directory, and then iterates through file permissions for files, directories, and issues snapshot to the source. Permission combinations are applied to a parent directory of the source, source itself, contents of source root when the source is a directory, and contents of a subdirectory in that source root.
Fixing kopia executable panic when a directory's contents can't be read. Previously the only Lstat error responded to was not-exist, letting other errors fall through and passing a nil `os.FileInfo` to the following function call, resulting in panic.
```
make lint
kopia/tools/.tools/bin/golangci-lint --deadline 180s run | tee .linterr.txt
cli/command_snapshot_delete.go:5: File is not `goimports`-ed with -local github.com/kopia/kopia (goimports)
tests/end_to_end_test/end_to_end_test.go:303: Function 'TestSnapshotDelete' is too long (187 > 100) (funlen)
func TestSnapshotDelete(t *testing.T) {
tests/end_to_end_test/end_to_end_test.go:306:2: only one cuddle assignment allowed before range statement (wsl)
for _, tc := range []struct {
^
tests/end_to_end_test/end_to_end_test.go:517:4: only one cuddle assignment allowed before if statement (wsl)
if expectDeleteSucceeds {
^
tests/end_to_end_test/end_to_end_test.go:537:2: assignments should only be cuddled with other assignments (wsl)
line := lines[0]
^
tests/end_to_end_test/end_to_end_test.go:542:2: only one cuddle assignment allowed before if statement (wsl)
if typeVal != "policy" {
^
tests/end_to_end_test/end_to_end_test.go:558:2: assignments should only be cuddled with other assignments (wsl)
restoreDir := filepath.Join(e.dataDir, "restored")
^
tests/end_to_end_test/end_to_end_test.go:568:2: if statements should only be cuddled with assignments (wsl)
if got, want := len(si[0].snapshots), 1; got != want {
^
tests/end_to_end_test/end_to_end_test.go:571:2: assignments should only be cuddled with other assignments (wsl)
snapID := si[0].snapshots[0].snapshotID
^
tests/end_to_end_test/end_to_end_test.go:604:2: if statements should only be cuddled with assignments (wsl)
if len(fileInfo) != 0 {
^
cli/command_snapshot_delete.go:25:2: only one cuddle assignment allowed before if statement (wsl)
if err != nil {
^
cli/command_snapshot_delete.go:36:3: if statements should only be cuddled with assignments (wsl)
if labels["username"] != getUserName() {
^
```
Snapshot restore will take a snapshot ID and restore the
associated snapshot to the target path.
- Looks up the manifest with the snapshot ID
- Gets the snapshot root entry
- Copies the snapshot from the root entry to the target path
Because it uses the parent manifest with the copied permissions,
the restored directory will have the permissions of the original
source directory.
Implemented snapshot delete command. Behaves similarly to manifest rm, but with extra verification steps.
- Checks that the referenced manifest is of type "snapshot"
- Checks that the ID points to a snapshot, checks that the host name, user name, and path provided by flag or defaults match the source of the snapshot ID. Command will fail if they do not match, except if given --unsafe-ignore-source, which will bypass the associated safety requirement and delete anyway.
Added end to end tests for input combinations, restore in conjunction with delete, and trying to snapshot delete a manifest by ID of a non-snapshot manifest.
`snapshot gc` marks contents not reachable from the root of any snapshot
as soft-deleted
The algorithm is a mark-and-sweep with parallel iteration of objects.
Currently it stores content IDs and object IDs in a map, so won't scale
to huge repositories, but this can be fixed in the future.
This fixes#110 at least for reasonable repository sizes.
Previously, it was possible for Flush() to miss in-flight writes,
but only when using repository manually since Uploader guarantees
there are no in-flight writes when it completes.
With this change Flush() will guarantee that any pending writes
completed before Flush() has started are guaranteed to be committed
to the repository before Flush() returns.
This was actually a regression introduced in #105.
Added regression test to prevent it from reoccurring.
Repository.Token() generates a base64-encoded token that can
be stored in password manager that fully describes repository connection
information (blob.ConnectionInfo) and optionally a password.
Use `kopia repo status -t` to print the token.
Use `kopia repo status -t -s` to print the token that also includes
repository password.
Use `kopia repo connect from-config --token T` to reconnect using the
token.
Also introduced strongly typed content.ID and manifest.ID (instead of string)
This aligns identifiers across all layers of repository:
blob.ID
content.ID
object.ID
manifest.ID
This updates the terminology everywhere - blocks become blobs and
`storage.Storage` becomes `blob.Storage`.
Also introduced blob.ID which is a specialized string type, that's
different from CABS block ID.
Also renamed CLI subcommands from `kopia storage` to `kopia blob`.
While at it introduced `block.ErrBlockNotFound` and
`object.ErrObjectNotFound` that do not leak from lower layers.
