* logging: cleaned up stderr logging
- do not show module
- do not show timestamps by default (enable with --console-timestamps)
* logging: replaced most printStderr() with log.Info
* cli: additional logging cleanup
* restore: improved user experience
* 'snapshot restore' is now the same as 'restore' and both will
support restoring by manifest ID, root ID or root ID + subdirectory
* added support for restoring individual files
* implemented PR feedback and refactored object ID parsing
Moving helpers inside the snapshot/ package helped clean up the code
a lot.
* repo: refactored client-specific options (hostname,username,description,readonly) into new struct that is JSON-compatible with current config
* cli: added 'repository set-client' to configure parameters of connected repository
* cli: cleaned up 'repository status' output
* snapshot: support for periodic checkpointing of snapshots in progress
For each snapshot that takes longer than 45 minutes, we trigger
internal cancellation, save the manifest and restart the snapshot
at which point all files will be cached.
This helps ensure the property that no file or directory objects
in the repository remain unreachable from a snapshot root for more than
one hour, which is important from GC perspective.
* nit: unified spelling 'cancelled' => 'canceled'
* This is 99% mechanical:
Extracted repo.Repository interface that only exposes high-level object and manifest management methods, but not blob nor content management.
Renamed old *repo.Repository to *repo.DirectRepository
Reviewed codebase to only depend on repo.Repository as much as possible, but added way for low-level CLI commands to use DirectRepository.
* PR fixes
- cleaned up migration progress output
- fixed migration idempotency
- added migration of policies
- renamed --parallelism to --parallel
- improved e2e test
- do not prompt for password to source repository if persisted
The hostname/username are now persisted when connecting to repository
in a local config file.
This prevents weird behavior changes when hostname is suddenly changed,
such as when moving between networks.
repo.Repository will now expose Hostname/Username properties which
are always guarnateed to be set, and are used throughout.
Removed --hostname/--username overrides when taking snapshot et.al.
This is mostly mechanical and changes how loggers are instantiated.
Logger is now associated with a context, passed around all methods,
(most methods had ctx, but had to add it in a few missing places).
By default Kopia does not produce any logs, but it can be overridden,
either locally for a nested context, by calling
ctx = logging.WithLogger(ctx, newLoggerFunc)
To override logs globally, call logging.SetDefaultLogger(newLoggerFunc)
This refactoring allowed removing dependency from Kopia repo
and go-logging library (the CLI still uses it, though).
It is now also possible to have all test methods emit logs using
t.Logf() so that they show up in failure reports, which should make
debugging of test failures suck less.
* Ignore read errors based on policy settings
Added an error handling policy section. Can independently control error handling for directory and file read errors, toggle-able from the `policy set` command to either "true", "false", or "inherit". If any read error is hit, the error handling will check the effective policy on whether to ignore it or not. Currently there is no differentiation between read error types, though in the future we may want to add the `errors.Is(err, os.ErrPermission)` conditional.
Fix was implemented such that the policy ignores read errors ONLY on child entries of the source. So a snapshot will still fail if the source root directory itself can't be read, but you can ignore the error if a file or a subdirectory in the snapshot source root can't be read. I did this to address some otherwise strange behavior where you would successfully snapshot (because you ignored the error), but couldn't restore that snapshot because nothing really happened during the operation.
This cleans up the code a lot and removes many ugly hacks.
The performance is pretty reasonable and with separate metadata cache it's likely to stay that way.
