Commit Graph

65 Commits

Author SHA1 Message Date
Julio Lopez
c8360ce6ca cleanup: fix linter errors
```
make lint
kopia/tools/.tools/bin/golangci-lint --deadline 180s run | tee .linterr.txt
cli/command_snapshot_delete.go:5: File is not `goimports`-ed with -local github.com/kopia/kopia (goimports)

tests/end_to_end_test/end_to_end_test.go:303: Function 'TestSnapshotDelete' is too long (187 > 100) (funlen)
func TestSnapshotDelete(t *testing.T) {
tests/end_to_end_test/end_to_end_test.go:306:2: only one cuddle assignment allowed before range statement (wsl)
	for _, tc := range []struct {
	^
tests/end_to_end_test/end_to_end_test.go:517:4: only one cuddle assignment allowed before if statement (wsl)
			if expectDeleteSucceeds {
			^
tests/end_to_end_test/end_to_end_test.go:537:2: assignments should only be cuddled with other assignments (wsl)
	line := lines[0]
	^
tests/end_to_end_test/end_to_end_test.go:542:2: only one cuddle assignment allowed before if statement (wsl)
	if typeVal != "policy" {
	^
tests/end_to_end_test/end_to_end_test.go:558:2: assignments should only be cuddled with other assignments (wsl)
	restoreDir := filepath.Join(e.dataDir, "restored")
	^
tests/end_to_end_test/end_to_end_test.go:568:2: if statements should only be cuddled with assignments (wsl)
	if got, want := len(si[0].snapshots), 1; got != want {
	^
tests/end_to_end_test/end_to_end_test.go:571:2: assignments should only be cuddled with other assignments (wsl)
	snapID := si[0].snapshots[0].snapshotID
	^
tests/end_to_end_test/end_to_end_test.go:604:2: if statements should only be cuddled with assignments (wsl)
	if len(fileInfo) != 0 {
	^
cli/command_snapshot_delete.go:25:2: only one cuddle assignment allowed before if statement (wsl)
	if err != nil {
	^
cli/command_snapshot_delete.go:36:3: if statements should only be cuddled with assignments (wsl)
		if labels["username"] != getUserName() {
		^
```
2019-12-18 18:17:15 -08:00
Nick
3f721aaf01 Snapshot restore command
Snapshot restore will take a snapshot ID and restore the
associated snapshot to the target path.
- Looks up the manifest with the snapshot ID
- Gets the snapshot root entry
- Copies the snapshot from the root entry to the target path

Because it uses the parent manifest with the copied permissions,
the restored directory will have the permissions of the original
source directory.
2019-12-13 06:18:50 -08:00
Nick
ae3d5610bf Dedicated snapshot delete command to delete by snap ID (#8)
Implemented snapshot delete command. Behaves similarly to manifest rm, but with extra verification steps. 
- Checks that the referenced manifest is of type "snapshot"
- Checks that the ID points to a snapshot, checks that the host name, user name, and path provided by flag or defaults match the source of the snapshot ID. Command will fail if they do not match, except if given --unsafe-ignore-source, which will bypass the associated safety requirement and delete anyway.

Added end to end tests for input combinations, restore in conjunction with delete, and trying to snapshot delete a manifest by ID of a non-snapshot manifest.
2019-12-12 18:02:48 -08:00
Jarek Kowalski
8ab93e0e2f addressed PR comments 2019-12-10 23:25:28 -08:00
Jarek Kowalski
9dd4e2aaf2 compression: added e2e test 2019-12-10 23:25:28 -08:00
Julio Lopez
5396166505 Display diff when restored dir does not match source 2019-12-10 22:22:32 -08:00
Julio Lopez
5c58b306a4 E2E test for snapshot restore command 2019-12-10 22:22:32 -08:00
Jarek Kowalski
6217df1a87 lint: switched to 1.21 and fixed a ton of whitespace issues discovered
by new wsl linter
2019-11-26 06:49:49 -08:00
Pavan Navarathna
829742599c [#124] Add optional flags to override hostname and username 2019-11-22 15:15:21 +09:00
Jarek Kowalski
3b1177cc4e cli: implemented 'snapshot gc' and moved verify to be snapshot verify
`snapshot gc` marks contents not reachable from the root of any snapshot
as soft-deleted

The algorithm is a mark-and-sweep with parallel iteration of objects.
Currently it stores content IDs and object IDs in a map, so won't scale
to huge repositories, but this can be fixed in the future.

This fixes #110 at least for reasonable repository sizes.
2019-09-15 21:25:48 -07:00
Jarek Kowalski
0654e30b26 linter: upgraded to golangci-lint 1.18.0, fixed or suppressed warnings 2019-09-11 06:42:08 -07:00
Jarek Kowalski
4c3272dd94 content: fixed content.Manager.Flush()
Previously, it was possible for Flush() to miss in-flight writes,
but only when using repository manually since Uploader guarantees
there are no in-flight writes when it completes.

With this change Flush() will guarantee that any pending writes
completed before Flush() has started are guaranteed to be committed
to the repository before Flush() returns.

This was actually a regression introduced in #105.
Added regression test to prevent it from reoccurring.
2019-09-02 19:13:36 -07:00
Jarek Kowalski
9893a552e2 metrics: disable google analytics as it's not providing any useful data, need to rethink the story here 2019-08-17 16:13:30 -07:00
Jarek Kowalski
671a63aa24 travis: generate repository snapshot for long-term storage
This is triggered for tagged releases and will be used later for
long-term repository compatibility testing.
2019-08-12 19:32:25 -07:00
Jarek Kowalski
b365d3414c removed content.Manager.ListContents*() API 2019-07-27 19:10:58 -07:00
Jarek Kowalski
8ada9587f3 travis: added encrypted 'known_hosts' file for sftp testing 2019-07-16 20:29:39 -10:00
Jarek Kowalski
59f0231f4c minor testability tweaks 2019-07-16 15:48:00 -10:00
Jarek Kowalski
769b3c5f12 travis: reencrypted credentials with fixed key/iv 2019-07-14 19:30:22 -07:00
Jarek Kowalski
975824768e sftp: added encrypted e2e test credentials 2019-07-14 10:25:10 -07:00
Jarek Kowalski
e414e7a4d1 Added support for reconnect tokens
Repository.Token() generates a base64-encoded token that can
be stored in password manager that fully describes repository connection
information (blob.ConnectionInfo) and optionally a password.

Use `kopia repo status -t` to print the token.
Use `kopia repo status -t -s` to print the token that also includes
repository password.

Use `kopia repo connect from-config --token T` to reconnect using the
token.
2019-07-10 21:03:40 -07:00
Jarek Kowalski
b16e19dc64 fixing gcs credentials, try #3 - regenerated new SA JSON key 2019-07-04 10:56:50 -07:00
Jarek Kowalski
cb7127ac4a fixed credentials path, try #2 2019-07-04 10:20:25 -07:00
Jarek Kowalski
27f5b0540d fixed installation of GCS credentials for testing, which was silently being ignored 2019-07-04 10:13:21 -07:00
Jarek Kowalski
72520029b0 golangci-lint: added more linters
Also fixed pre-existing lint errors.
2019-06-02 22:56:57 -07:00
Jarek Kowalski
54edb97b3a refactoring: renamed repo/block to repo/content
Also introduced strongly typed content.ID and manifest.ID (instead of string)

This aligns identifiers across all layers of repository:

blob.ID
content.ID
object.ID
manifest.ID
2019-06-01 22:24:19 -07:00
Jarek Kowalski
916da07e0f deprecate block format v0 2019-06-01 16:40:51 -07:00
Jarek Kowalski
9e5d0beccd refactoring: renamed storage.Storage to blob.Storage
This updates the terminology everywhere - blocks become blobs and
`storage.Storage` becomes `blob.Storage`.

Also introduced blob.ID which is a specialized string type, that's
different from CABS block ID.

Also renamed CLI subcommands from `kopia storage` to `kopia blob`.

While at it introduced `block.ErrBlockNotFound` and
`object.ErrObjectNotFound` that do not leak from lower layers.
2019-06-01 14:10:35 -07:00
Jarek Kowalski
1a7a02ddbe cleanup imports by grouping all local imports together 2019-06-01 10:57:55 -07:00
Jarek Kowalski
63303904e1 switched remaining fmt.Errorf to errors.Wrap() 2019-06-01 10:57:05 -07:00
Jarek Kowalski
0c41d41276 Fixed up paths after merge 2019-05-27 15:48:39 -07:00
Jarek Kowalski
b6abef96e6 Merge github.com:kopia/repo into import-repo 2019-05-27 15:41:55 -07:00
Jarek Kowalski
bdafe117d9 Makefile: switched linter to golangci-lint and updated goveralls setup
fixed lint errors & removed .gometalinter config
2019-04-01 19:22:01 -07:00
Jarek Kowalski
fdfd336d6e lint: fixed various linter issues 2019-04-01 18:59:48 -07:00
Jarek Kowalski
913f3392e3 cli: renamed 'repo migration' to 'snapshot migrate' and improved caching 2019-01-19 10:52:03 -08:00
Jarek Kowalski
b37c29513b cli: added 'repo repair' subcommand which operates on the same arguments as 'repo connect', but it fixes contents of the repository by restoring missing kopia.repository file
tests: reorganized end-to-end tests
2019-01-19 05:48:40 -05:00
Jarek Kowalski
c35037601d repo: embed format block bytes in all pack indexes
This allows format block to be recovered if any block
indexes survives. The block is not encrypted, but has HMAC is
followed by 2-byte size.
2019-01-12 14:24:09 -08:00
Jarek Kowalski
2e297dc60d repo: upgraded to latest version, removed format block settings from repository create, instead --encryption=NONE disables encryption of both blocks and format 2019-01-12 09:24:12 -08:00
Jarek Kowalski
000ca67f25 test: fixed end_to_end_test 2019-01-02 19:59:21 -08:00
Jarek Kowalski
840d5ab749 removed support for legacy block format, to migrate sync to previous commit and run 'kopia repo upgrade' 2018-12-31 19:01:08 -08:00
Jarek Kowalski
6e8a48ed4f Revamped hash and encryption algorithm handling.
This enables many more combinations of hash and encryption algorithm
to be selected, some with dramatically better performance.
2018-12-29 15:08:29 -08:00
Jarek Kowalski
afad2ad237 repo: moved repository_test to top level 2018-12-23 20:53:44 -08:00
Jarek Kowalski
3487cc191e removed asynchronous mode for object.Manager 2018-12-22 19:58:00 -08:00
Jarek Kowalski
e458ee24d8 imported github.com/kopia/kopia/repo and renamed package path to github.com/kopia/repo/ 2018-10-26 17:33:58 -07:00
Jarek Kowalski
bcbab79dd0 coverage: updated 'go test' command to include all packages
previously the coverage for each package only included lines covered by tests in the same package,
excluding end-to-end tests
now, we get very different coverage numbers...
2018-10-20 09:20:08 -07:00
Jarek Kowalski
3d61b4b39b cli: fixed some 'diff' issues and added integration test 2018-10-06 11:32:54 -07:00
Jarek Kowalski
fbffab8bc9 cli: create global policy when repository is created 2018-09-29 09:51:15 +02:00
Jarek Kowalski
1bbc169c0d added missing package godoc, fixed test paths 2018-08-31 19:12:58 -07:00
Jarek Kowalski
91066f2469 reorganized low-level repository packages by moving them all under kopia/kopia/repo/ 2018-08-30 22:01:05 -07:00
Jarek Kowalski
0f20965143 added 'blockindex recover' tests 2018-08-25 16:30:11 -07:00
Jarek Kowalski
be32f6b717 enabled more comprehensive repository stress test after resolving remaining races:
skipped stress tests during short testing
changed default make target to run short tests
2018-07-20 18:48:00 -08:00