Commit Graph

186 Commits

Author SHA1 Message Date
Jarek Kowalski
9680dc376b cli: improvements for 'kopia server' and client
Those will make it possible to securely host 'kopia server' embedded
in a desktop app that runs in the background and can access UI.

- added support for using and generating TLS certificates
- added /api/v1/shutdown API to remotely trigger server shutdown
- added support for automatically shutting down server if no requests
  arrive in certain amount of time
- added support for generating and printing random password to STDERR

TLS supports 3 modes:

1. serve TLS using externally-provided cert/key PEM files
2. generate & write PEM files, then serve TLS using them
3. generate and use emphemeral cert/key (prints SHA256 fingerprint)
2020-01-24 17:25:45 -08:00
Jarek Kowalski
1bb7eeebe7 fixed inadvertent behavior change 2020-01-03 16:39:30 -08:00
Jarek Kowalski
ac70a38101 lint: upgraded to 1.22.2 and make lint issues a build failure
fixed or silenced linter warnings, mostly due to magic numeric constants
2020-01-03 16:39:30 -08:00
Jarek Kowalski
503fa74be5 fs: added Directory.Child() method to look up children by name without necessarily loading all of them 2019-12-30 19:21:17 -08:00
Jarek Kowalski
d9d4804dcb server: handle known UI routes specially by serving root index file 2019-12-14 13:10:32 -08:00
Jarek Kowalski
5d55568767 server: added /api/v1/objects/:id that retrieves contents of object given its ID 2019-12-14 10:19:08 -08:00
Julio Lopez
9550c11aeb Add .gitignore for htmlui_bindata.go 2019-12-11 06:46:41 -08:00
Julio Lopez
d3f324f59b Add diff.compareEntry helper to compare attribute entries
- Use it to compare the entry attributes for all entry types.
  Allows comparing differences in file attributes among the
  contents of two directories. This is useful for verifying
  restored contents in end-to-end tests.

- Print message about modified entries only when both entries
  are files and they are being compared.
2019-12-10 22:22:32 -08:00
Julio Lopez
820b29cbf2 FS hasher testing helper 2019-12-10 22:22:32 -08:00
Jarek Kowalski
f6d59608ab upload: require *policy.Tree for each upload 2019-12-07 14:24:57 -08:00
Jarek Kowalski
b68ef7d781 refactored policy types and policy.Getter 2019-12-06 23:53:22 -08:00
Jarek Kowalski
6217df1a87 lint: switched to 1.21 and fixed a ton of whitespace issues discovered
by new wsl linter
2019-11-26 06:49:49 -08:00
Julio Lopez
24f292b719 Cleanup lint: empty lines (via whitespace and wsl linters) 2019-11-22 15:17:40 +09:00
Jarek Kowalski
36859bdf8b server: removed dependency on pat 2019-11-20 15:49:40 -08:00
Jarek Kowalski
e8a1eaa9ee htmlui: added experimental HTML-based UI
This is enabled by `kopia server --ui` and can be viewed in a browser
at http://localhost:51515/

Right now it can only list snapshots and policies (barely).
2019-11-20 14:41:38 -08:00
Julio Lopez
945708c5c5 Add test for snapshot.Stats
Also, set directory mode and default permissions in mockfs.NewDirectory
2019-11-20 04:27:56 -08:00
Jarek Kowalski
3b1177cc4e cli: implemented 'snapshot gc' and moved verify to be snapshot verify
`snapshot gc` marks contents not reachable from the root of any snapshot
as soft-deleted

The algorithm is a mark-and-sweep with parallel iteration of objects.
Currently it stores content IDs and object IDs in a map, so won't scale
to huge repositories, but this can be fixed in the future.

This fixes #110 at least for reasonable repository sizes.
2019-09-15 21:25:48 -07:00
Jarek Kowalski
4c3272dd94 content: fixed content.Manager.Flush()
Previously, it was possible for Flush() to miss in-flight writes,
but only when using repository manually since Uploader guarantees
there are no in-flight writes when it completes.

With this change Flush() will guarantee that any pending writes
completed before Flush() has started are guaranteed to be committed
to the repository before Flush() returns.

This was actually a regression introduced in #105.
Added regression test to prevent it from reoccurring.
2019-09-02 19:13:36 -07:00
Niraj Tolia
f32f0ece2f Switch file log level to info
Reduces multi-GB log files. Main culprits were:

- [upload.go:385] could not find cache entry for...
- [content_manager.go:790] WriteContent("...") - new
- [content_manager.go:396] adding ... length=... deleted=false
2019-08-24 21:09:39 -07:00
Jarek Kowalski
399164b3be webdav: improved error handling
- improved HTTP error handling
- added exponential back-off around gowebdav calls
- fixed blob.ListAllBlobsConsistent which was never finishing for WebDAV

Fixes #88 - 429 Too Many Requests - PROPFIND /n0a #88
Fixex #89 - Debug log clashes with interactive repo password request
2019-08-18 10:49:11 -07:00
Jarek Kowalski
3a80d945d9 lint: upgraded to 1.17.1 2019-07-19 20:00:23 -10:00
Jarek Kowalski
22170b4832 cli: changed 'cache set' subcommand to support changing individual parameters 2019-06-11 22:08:52 -07:00
Jarek Kowalski
23d979f290 deleted jsonstream 2019-06-08 22:47:05 -07:00
Jarek Kowalski
ae1510b65c Deprecate hashcache and replace it with simply walking previous snapshot directory tree.
This cleans up the code a lot and removes many ugly hacks.
The performance is pretty reasonable and with separate metadata cache it's likely to stay that way.
2019-06-08 18:01:47 -07:00
Jarek Kowalski
72520029b0 golangci-lint: added more linters
Also fixed pre-existing lint errors.
2019-06-02 22:56:57 -07:00
Jarek Kowalski
54edb97b3a refactoring: renamed repo/block to repo/content
Also introduced strongly typed content.ID and manifest.ID (instead of string)

This aligns identifiers across all layers of repository:

blob.ID
content.ID
object.ID
manifest.ID
2019-06-01 22:24:19 -07:00
Jarek Kowalski
9e5d0beccd refactoring: renamed storage.Storage to blob.Storage
This updates the terminology everywhere - blocks become blobs and
`storage.Storage` becomes `blob.Storage`.

Also introduced blob.ID which is a specialized string type, that's
different from CABS block ID.

Also renamed CLI subcommands from `kopia storage` to `kopia blob`.

While at it introduced `block.ErrBlockNotFound` and
`object.ErrObjectNotFound` that do not leak from lower layers.
2019-06-01 14:10:35 -07:00
Jarek Kowalski
1a7a02ddbe cleanup imports by grouping all local imports together 2019-06-01 10:57:55 -07:00
Jarek Kowalski
63303904e1 switched remaining fmt.Errorf to errors.Wrap() 2019-06-01 10:57:05 -07:00
Jarek Kowalski
03339c18af [breaking change] deprecated DYNAMIC splitter due to license issue
The splitter in question was depending on
github.com/silvasur/buzhash which is not licensed according to FOSSA bot

Switched to new faster implementation of buzhash, which is
unfortunately incompatible and will split the objects in different
places.

This change is be semi-breaking - old repositories can be read, but
when uploading large objects they will be re-uploaded where previously
they would be de-duped.

Also added 'benchmark splitters' subcommand and moved 'block cryptobenchmark'
subcommand to 'benchmark crypto'.
2019-05-30 22:20:45 -07:00
Jarek Kowalski
0c41d41276 Fixed up paths after merge 2019-05-27 15:48:39 -07:00
Jarek Kowalski
b6abef96e6 Merge github.com:kopia/repo into import-repo 2019-05-27 15:41:55 -07:00
Jarek Kowalski
a6a153b22e switched fmt.Errorf() to errors.Wrap() 2019-05-11 12:34:14 -07:00
Jarek Kowalski
142bfa3622 site: included logging-related flags, cleanup 2019-05-09 23:00:07 -07:00
Jarek Kowalski
bdafe117d9 Makefile: switched linter to golangci-lint and updated goveralls setup
fixed lint errors & removed .gometalinter config
2019-04-01 19:22:01 -07:00
Jarek Kowalski
fdfd336d6e lint: fixed various linter issues 2019-04-01 18:59:48 -07:00
Jarek Kowalski
2e297dc60d repo: upgraded to latest version, removed format block settings from repository create, instead --encryption=NONE disables encryption of both blocks and format 2019-01-12 09:24:12 -08:00
Jarek Kowalski
766f574973 repo: removed controls for setting format block encryption, instead when block encryption is enabled, the format block is automatically encrypted using default encryption algorithm 2019-01-12 09:14:34 -08:00
Jarek Kowalski
db9759bf3a upgraded repo 2018-12-31 19:10:21 -08:00
Jarek Kowalski
840d5ab749 removed support for legacy block format, to migrate sync to previous commit and run 'kopia repo upgrade' 2018-12-31 19:01:08 -08:00
Jarek Kowalski
0b89cbef3b cli: added 'repo upgrade' command that invokes Repository.Upgrade() 2018-12-31 17:12:30 -08:00
Jarek Kowalski
24bd5bbe1f repo: added Repository.Upgrade() API 2018-12-31 17:08:41 -08:00
Jarek Kowalski
b4c80348f8 object: removed dependency on jsonstream for parsing indirect objects 2018-12-31 14:21:47 -08:00
Jarek Kowalski
a6a9729f71 upgraded repo to v0.2.0 2018-12-29 15:14:29 -08:00
Jarek Kowalski
6e8a48ed4f Revamped hash and encryption algorithm handling.
This enables many more combinations of hash and encryption algorithm
to be selected, some with dramatically better performance.
2018-12-29 15:08:29 -08:00
Jarek Kowalski
7d03d82cd1 retry: added tests for retry loop 2018-12-23 23:20:29 -08:00
Jarek Kowalski
53b4a9abd2 jsonstream: added test cases 2018-12-23 23:04:12 -08:00
Jarek Kowalski
641b25a508 repo: added tests for Disconnect() 2018-12-23 21:12:53 -08:00
Jarek Kowalski
160d02a6eb block cache: fixed a bug where we remove wrong item from the cache, added tests 2018-12-23 11:59:17 -08:00
Jarek Kowalski
a673d36b1d lint: ran gofmt -s and fixed spelling
as recommended by goreportcard.com
2018-11-07 17:46:57 -08:00