Commit Graph

1030 Commits

Author SHA1 Message Date
Nick
08519b4ab0 Add test to induce a failure during a restore using blob delete
Add a test that will delete a kopia pack blob, expecting a subsequent restore issued from the snapshot that referenced that blob to fail.
2020-01-28 19:36:00 -08:00
Jarek Kowalski
9680dc376b cli: improvements for 'kopia server' and client
Those will make it possible to securely host 'kopia server' embedded
in a desktop app that runs in the background and can access UI.

- added support for using and generating TLS certificates
- added /api/v1/shutdown API to remotely trigger server shutdown
- added support for automatically shutting down server if no requests
  arrive in certain amount of time
- added support for generating and printing random password to STDERR

TLS supports 3 modes:

1. serve TLS using externally-provided cert/key PEM files
2. generate & write PEM files, then serve TLS using them
3. generate and use emphemeral cert/key (prints SHA256 fingerprint)
2020-01-24 17:25:45 -08:00
Nick
7367f85afe Snapshot failure test and fix kopia panic for non-executable directories
Adding test to probe snapshot failure. Tests snapshot of non-existent source directory, and then iterates through file permissions for files, directories, and issues snapshot to the source. Permission combinations are applied to a parent directory of the source, source itself, contents of source root when the source is a directory, and contents of a subdirectory in that source root.

Fixing kopia executable panic when a directory's contents can't be read. Previously the only Lstat error responded to was not-exist, letting other errors fall through and passing a nil `os.FileInfo` to the following function call, resulting in panic.
2020-01-20 21:29:55 -08:00
Nick
3913241159 [Trivial] Fix command description for blob delete
Fix probable copy paste error on "blob delete" subcommand from
the "blob show" description.
2020-01-20 21:28:44 -08:00
Jarek Kowalski
2b475565bb testing: marked all remaining tests as parallel 2020-01-16 19:40:34 -08:00
Jarek Kowalski
2bc8383e47 testing: customized e2e test directory tree shapes
based on PR feedback, instead of 3 uniform directories,
have 3 different shapes.
2020-01-16 19:40:34 -08:00
Jarek Kowalski
7d39be976f testing: split end_to_end_test into separate files.
refactored test helpers to separate package
made all tests parallel and improved the code structure
2020-01-16 19:40:34 -08:00
Jarek Kowalski
644ef93d45 s3: added optional --region parameter
Fixes #168
v0.4.0
2020-01-09 16:37:36 -08:00
Jarek Kowalski
0b8c4d0ef9 object: fixed compression bug where we were not clearing the buffer
this effectively defeated the purpose of compression, caused high
memory usage and other kinds of bad behavior.

refactored the code to prevent this issue by resetting the buffer
at the caller not callee.

fixed previous e2e test to catch the issue mentioned in #166,
verified it fails against master and passes with this change.
2020-01-09 16:36:57 -08:00
Jarek Kowalski
de71f0f057 fs: added functions Update() and Remove() on fs.Entries which return a copy of entries with specified entry updated or removed 2020-01-05 09:13:41 -08:00
Jarek Kowalski
aa1b24762d lint: added retry loop around linter invocations 2020-01-03 16:39:30 -08:00
Jarek Kowalski
1bb7eeebe7 fixed inadvertent behavior change 2020-01-03 16:39:30 -08:00
Jarek Kowalski
ac70a38101 lint: upgraded to 1.22.2 and make lint issues a build failure
fixed or silenced linter warnings, mostly due to magic numeric constants
2020-01-03 16:39:30 -08:00
Julio Lopez
b72965444c Update e2e restore test 2020-01-02 13:49:28 -08:00
Julio Lopez
1249a255d3 Allow copying to an existing directory
Add some logging
2020-01-02 13:49:28 -08:00
Julio Lopez
ac6e84107e Add CopyOptions and copier structs 2020-01-02 13:49:28 -08:00
Julio Lopez
20b896da18 Add restore command flags 2020-01-02 13:49:28 -08:00
Julio Lopez
7361d277e0 Remove special-case handling when target is root directory
No longer needed since directories can be overwritten by default
2020-01-02 13:49:28 -08:00
Julio Lopez
99bca0218d Add comment to exported constant 2020-01-02 09:50:45 -08:00
Julio Lopez
8526376e08 Close channel in the error case as well 2020-01-02 09:50:45 -08:00
Julio Lopez
2389abb465 manifest.TypeLabelKey const 2020-01-02 09:50:45 -08:00
Julio Lopez
5c62bce5eb snapshot.ManifestType const 2020-01-02 09:50:45 -08:00
Julio Lopez
c0b1345d9a Trivial: add clarifying comment 2020-01-02 09:50:45 -08:00
Julio Lopez
8e9d56fcf2 Trivial: simplify localfs.Directory 2020-01-02 09:50:45 -08:00
Julio Lopez
a7ee7d57b6 Install go-bindata in /bin 2020-01-02 09:50:45 -08:00
Julio Lopez
c03a143ae9 Fix flag description 2020-01-02 09:50:45 -08:00
Jarek Kowalski
5d0167e2bb snapshotfs: added HasDirEntry and type assertions 2020-01-01 15:10:43 -08:00
Jarek Kowalski
8ff8ec0a36 snapshotfs: expose functions to round-trip between fs.Entry and DirEntry 2020-01-01 15:10:43 -08:00
Jarek Kowalski
503fa74be5 fs: added Directory.Child() method to look up children by name without necessarily loading all of them 2019-12-30 19:21:17 -08:00
Julio Lopez
194680adfc Refactor: relocate user/host helpers to userhost.go 2019-12-18 23:11:38 -08:00
Julio Lopez
0b7f30b6b4 rename cli/snapshot_utils.go -> cli/userhost.go 2019-12-18 23:11:38 -08:00
Jarek Kowalski
771fec1077 build: added simple Windows-based build script
This orchestrates building HTMLUI and main binary, only uses built-in shell script and avoids having any dependencies on Unix tools. Latest go and npm are required.
2019-12-18 19:35:40 -08:00
Julio Lopez
c8360ce6ca cleanup: fix linter errors
```
make lint
kopia/tools/.tools/bin/golangci-lint --deadline 180s run | tee .linterr.txt
cli/command_snapshot_delete.go:5: File is not `goimports`-ed with -local github.com/kopia/kopia (goimports)

tests/end_to_end_test/end_to_end_test.go:303: Function 'TestSnapshotDelete' is too long (187 > 100) (funlen)
func TestSnapshotDelete(t *testing.T) {
tests/end_to_end_test/end_to_end_test.go:306:2: only one cuddle assignment allowed before range statement (wsl)
	for _, tc := range []struct {
	^
tests/end_to_end_test/end_to_end_test.go:517:4: only one cuddle assignment allowed before if statement (wsl)
			if expectDeleteSucceeds {
			^
tests/end_to_end_test/end_to_end_test.go:537:2: assignments should only be cuddled with other assignments (wsl)
	line := lines[0]
	^
tests/end_to_end_test/end_to_end_test.go:542:2: only one cuddle assignment allowed before if statement (wsl)
	if typeVal != "policy" {
	^
tests/end_to_end_test/end_to_end_test.go:558:2: assignments should only be cuddled with other assignments (wsl)
	restoreDir := filepath.Join(e.dataDir, "restored")
	^
tests/end_to_end_test/end_to_end_test.go:568:2: if statements should only be cuddled with assignments (wsl)
	if got, want := len(si[0].snapshots), 1; got != want {
	^
tests/end_to_end_test/end_to_end_test.go:571:2: assignments should only be cuddled with other assignments (wsl)
	snapID := si[0].snapshots[0].snapshotID
	^
tests/end_to_end_test/end_to_end_test.go:604:2: if statements should only be cuddled with assignments (wsl)
	if len(fileInfo) != 0 {
	^
cli/command_snapshot_delete.go:25:2: only one cuddle assignment allowed before if statement (wsl)
	if err != nil {
	^
cli/command_snapshot_delete.go:36:3: if statements should only be cuddled with assignments (wsl)
		if labels["username"] != getUserName() {
		^
```
2019-12-18 18:17:15 -08:00
Jarek Kowalski
fc4675e6b6 htmlui: fixed sorting by path in snapshots view 2019-12-18 18:13:14 -08:00
Jarek Kowalski
313d326b45 htmlui: added elipsis in pagination bar for super long tables 2019-12-18 18:13:14 -08:00
Jarek Kowalski
495fc6deb1 htmlui: various UI tweaks
unified column ordering and sorting
switched to react-table v7 which allows seamless integration with bootstrap
2019-12-14 19:30:09 -08:00
Jarek Kowalski
c1454db2a9 makefile: added quick-install target which only rebuilds the binary and embeds pre-built HTMLUI 2019-12-14 19:30:09 -08:00
Jarek Kowalski
d9d4804dcb server: handle known UI routes specially by serving root index file 2019-12-14 13:10:32 -08:00
Jarek Kowalski
bd7a82ab94 makefile: added clean target 2019-12-14 10:48:31 -08:00
Jarek Kowalski
dbc820e972 htmlui: fixed npm vulnerability warnings 2019-12-14 10:19:08 -08:00
Jarek Kowalski
3a809f01e8 htmlui: added UI to browse contents of directories and download files 2019-12-14 10:19:08 -08:00
Jarek Kowalski
b93e58760a makefile: install and install-noui target 2019-12-14 10:19:08 -08:00
Jarek Kowalski
5d55568767 server: added /api/v1/objects/:id that retrieves contents of object given its ID 2019-12-14 10:19:08 -08:00
Nick
3f721aaf01 Snapshot restore command
Snapshot restore will take a snapshot ID and restore the
associated snapshot to the target path.
- Looks up the manifest with the snapshot ID
- Gets the snapshot root entry
- Copies the snapshot from the root entry to the target path

Because it uses the parent manifest with the copied permissions,
the restored directory will have the permissions of the original
source directory.
2019-12-13 06:18:50 -08:00
Nick
ae3d5610bf Dedicated snapshot delete command to delete by snap ID (#8)
Implemented snapshot delete command. Behaves similarly to manifest rm, but with extra verification steps. 
- Checks that the referenced manifest is of type "snapshot"
- Checks that the ID points to a snapshot, checks that the host name, user name, and path provided by flag or defaults match the source of the snapshot ID. Command will fail if they do not match, except if given --unsafe-ignore-source, which will bypass the associated safety requirement and delete anyway.

Added end to end tests for input combinations, restore in conjunction with delete, and trying to snapshot delete a manifest by ID of a non-snapshot manifest.
2019-12-12 18:02:48 -08:00
Julio Lopez
39a2cad2d9 Trivial/cleanup: remove unused type 2019-12-11 06:46:41 -08:00
Julio Lopez
9550c11aeb Add .gitignore for htmlui_bindata.go 2019-12-11 06:46:41 -08:00
Julio Lopez
7a34cb0a44 lint cleanup: add comments to exported symbols 2019-12-11 06:46:41 -08:00
Jarek Kowalski
2ba4e83cef moved all compression to separate package and sanitized identifiers 2019-12-10 23:25:28 -08:00
Jarek Kowalski
8ab93e0e2f addressed PR comments 2019-12-10 23:25:28 -08:00