Commit Graph

40 Commits

Author SHA1 Message Date
Jarek Kowalski
c8fcae93aa logging: refactored logging
This is mostly mechanical and changes how loggers are instantiated.

Logger is now associated with a context, passed around all methods,
(most methods had ctx, but had to add it in a few missing places).

By default Kopia does not produce any logs, but it can be overridden,
either locally for a nested context, by calling

ctx = logging.WithLogger(ctx, newLoggerFunc)

To override logs globally, call logging.SetDefaultLogger(newLoggerFunc)

This refactoring allowed removing dependency from Kopia repo
and go-logging library (the CLI still uses it, though).

It is now also possible to have all test methods emit logs using
t.Logf() so that they show up in failure reports, which should make
debugging of test failures suck less.
2020-02-25 17:24:44 -08:00
Jarek Kowalski
a21da7b960 cli: fixed double-close of repository during 'server start' 2020-02-16 22:43:36 -08:00
Jarek Kowalski
0f79279f5e server: added support for new verbs in the API
/api/v1/repo/create
/api/v1/repo/connect
/api/v1/repo/disconnect

Refactored server code and fixed a number of outstanding robustness
issues. Tweaked the API responses a bit to make more sense when consumed
by the UI.
2020-02-13 17:23:50 -08:00
Jarek Kowalski
4736e9037e revamped progress output and cleaned up logging
See https://asciinema.org/a/ykx6uzEhKY3451fWEnX9nm9uo
2020-02-10 19:08:35 -08:00
Jarek Kowalski
9680dc376b cli: improvements for 'kopia server' and client
Those will make it possible to securely host 'kopia server' embedded
in a desktop app that runs in the background and can access UI.

- added support for using and generating TLS certificates
- added /api/v1/shutdown API to remotely trigger server shutdown
- added support for automatically shutting down server if no requests
  arrive in certain amount of time
- added support for generating and printing random password to STDERR

TLS supports 3 modes:

1. serve TLS using externally-provided cert/key PEM files
2. generate & write PEM files, then serve TLS using them
3. generate and use emphemeral cert/key (prints SHA256 fingerprint)
2020-01-24 17:25:45 -08:00
Julio Lopez
887a9180fa Trivial: fix imports grouping 2019-12-10 22:22:32 -08:00
Jarek Kowalski
6217df1a87 lint: switched to 1.21 and fixed a ton of whitespace issues discovered
by new wsl linter
2019-11-26 06:49:49 -08:00
Jarek Kowalski
09a31fa910 cli: added -–track-memory-usage flag that periodically logs memory usage 2019-08-24 13:18:31 -07:00
Jarek Kowalski
81ea751c0c cli: added profile flags, built only with 'profiling' tag 2019-08-24 10:42:12 -07:00
Jarek Kowalski
9893a552e2 metrics: disable google analytics as it's not providing any useful data, need to rethink the story here 2019-08-17 16:13:30 -07:00
Jarek Kowalski
6ef696d97a cli: resolve symlinks for snapshot roots
also - error handling improvements in the CLI
2019-07-18 08:40:44 -10:00
Jarek Kowalski
54edb97b3a refactoring: renamed repo/block to repo/content
Also introduced strongly typed content.ID and manifest.ID (instead of string)

This aligns identifiers across all layers of repository:

blob.ID
content.ID
object.ID
manifest.ID
2019-06-01 22:24:19 -07:00
Jarek Kowalski
9e5d0beccd refactoring: renamed storage.Storage to blob.Storage
This updates the terminology everywhere - blocks become blobs and
`storage.Storage` becomes `blob.Storage`.

Also introduced blob.ID which is a specialized string type, that's
different from CABS block ID.

Also renamed CLI subcommands from `kopia storage` to `kopia blob`.

While at it introduced `block.ErrBlockNotFound` and
`object.ErrObjectNotFound` that do not leak from lower layers.
2019-06-01 14:10:35 -07:00
Jarek Kowalski
1a7a02ddbe cleanup imports by grouping all local imports together 2019-06-01 10:57:55 -07:00
Jarek Kowalski
03339c18af [breaking change] deprecated DYNAMIC splitter due to license issue
The splitter in question was depending on
github.com/silvasur/buzhash which is not licensed according to FOSSA bot

Switched to new faster implementation of buzhash, which is
unfortunately incompatible and will split the objects in different
places.

This change is be semi-breaking - old repositories can be read, but
when uploading large objects they will be re-uploaded where previously
they would be de-duped.

Also added 'benchmark splitters' subcommand and moved 'block cryptobenchmark'
subcommand to 'benchmark crypto'.
2019-05-30 22:20:45 -07:00
Jarek Kowalski
0c41d41276 Fixed up paths after merge 2019-05-27 15:48:39 -07:00
Jarek Kowalski
a6a153b22e switched fmt.Errorf() to errors.Wrap() 2019-05-11 12:34:14 -07:00
Jarek Kowalski
327d8317d8 refactored repo/ into separate github.com/kopia/repo/ git repository 2018-10-26 20:40:57 -07:00
Jarek Kowalski
f4573f089e removed 'object' subcommand and flattened children directly under app 2018-10-06 09:55:12 -07:00
Jarek Kowalski
08d62d60f1 revamped progress bar by including hashing and uploading part separately 2018-09-14 11:56:48 -07:00
Jarek Kowalski
69782d2abe refactored cache commands 2018-09-13 19:33:12 -07:00
Jarek Kowalski
1bbc169c0d added missing package godoc, fixed test paths 2018-08-31 19:12:58 -07:00
Jarek Kowalski
91066f2469 reorganized low-level repository packages by moving them all under kopia/kopia/repo/ 2018-08-30 22:01:05 -07:00
Jarek Kowalski
c57531b751 switched Go logging library to github.com/op/go-logging which lets us override log levels per module 2018-07-09 21:28:12 -07:00
Jarek Kowalski
83036f1a0d upload progress 2018-06-23 13:10:54 -07:00
Jarek Kowalski
2f1cf8129b added skeleton of CLI actions for manipulating server sources 2018-06-18 18:02:36 -07:00
Jarek Kowalski
846a46b879 changed how caching is controlled in block manager by using context instead of options 2018-06-10 10:20:52 -07:00
Jarek Kowalski
53db414ff7 Added simple analytics mechanism based on Google Analytics for tracking features usage and latency.
Controlled on command line via --analytics-consent (defaults to asking user)
2018-05-30 21:22:07 -07:00
Jarek Kowalski
d8201229d8 plumbed through ctx in storage.Storage APIs and all uses 2018-04-03 17:39:54 -07:00
Jarek Kowalski
453bab3560 fixed some lint errors, mostly dead code and missing error checks 2018-03-19 12:26:28 -07:00
Jarek Kowalski
faa2625a5f revamped CLI help to hide most commands 2018-02-15 19:49:51 -08:00
Jarek Kowalski
4564bc704f breaking format change: removed support for groups in block manager, this produces bigger, tightly packed blocks
align pack blocks to 4096 bytes and insert random preamble
2018-02-10 13:07:10 -08:00
Jarek Kowalski
df35ebad73 added low-level storage viewing subcommands 2018-02-03 22:11:53 -08:00
Jarek Kowalski
090d97ba78 cli: reorganized all repo commands (connect/create/status) to top level 2018-01-10 19:13:09 -08:00
Jarek Kowalski
10ca234633 switched to using zerolog for logging, by default log level=info, can be set to debug on commandline 2018-01-01 19:12:35 -08:00
Jarek Kowalski
eebb04c0cb added 'manifest ls' subcommand 2017-11-27 18:05:44 -08:00
Jarek Kowalski
e76f1f8a72 refactored various show command into one 2017-10-24 21:59:40 -07:00
Jarek Kowalski
444b6306ca refactored object manager to use block manager, exposed on repository
added CLI commands to manipulate blocks
2017-10-08 12:44:53 -07:00
Jarek Kowalski
f606ab4347 reorganized more top-level CLI commands into subcommands 2017-08-20 07:51:24 -07:00
Jarek Kowalski
4a878738a0 moved main executable package to kopia/kopia, moved cli support to kopia/kopia/cli 2017-07-17 15:20:32 +02:00