* chore(ci): upgraded linter to 1.53.3
This flagged a bunch of unused parameters, so the PR is larger than
usual, but 99% mechanical.
* separate lint CI task
* run Lint in separate CI
* Update display on repository summary
* Apply throughout app
* Situate units_test
* Update Command Line documentation
* Envar cleanup
* Rename to BytesString
* Restore envar string available for test
* Remove extraneous empty check and restore UIPreferences field for frontend
* PR: config bool cleanup and missed `BaseEnv`s
* Fix lint and test
* Allow dynamic directory entries with virtualfs
* Tests for new virtualfs implementation
* Add escape hatch for estimator during upload
Some virtualfs.StreamingDirectory-s may not be able to (efficiently)
support iterating through entries multiple times. Make a way for the
estimator to ask if they support multiple iterations and skip the
directory if they do not.
* Exapand Directory interface
Expand the Directory interface instead of making a new interface as it's
error-prone to ensure all wrapper types properly handle types that use
the new interface.
* Post-rebase fixes
* Make StreamingDirectory single iteration only
Simplify code and test slightly by not allowing users to declare a
StreamingDirectory that can be iterated through multiple times.
* Add better test for estimator ignoring stream dir
Previous test in uploader had a race condition, meaning it may not catch
all cases.
* Ignore atomic access in checklocks
Comparisons known to be done after all additions to the variables in
question.
* Implement reviewer feedback
* Remove unused function parameter
* cli: Added --max-examples-per-bucket flag to 'kopia snapshot estimate'
Added and cleaned up a bunch of unit tests.
Fixes#1054
* cli: misc tests to increase code coverage of the cli package
* ci: move code coverage run into separate GH job
cli: major refactoring of how CLI commands are registered
The goal is to eliminate flags as global variables to allow for better
testing. Each command and subcommand and most sets of flags are now
their own struct with 'setup()' methods that attached the flags or
subcommand to the provided parent.
This change is 94.3% mechanical, but is fully organic and hand-made.
* introduced cli.appServices interface which provides the environment in which commands run
* remove auto-maintenance global flag
* removed globals in memory_tracking.go
* removed globals from cli_progress.go
* removed globals from the update_check.go
* moved configPath into TheApp
* removed remaining globals from config.go
* refactored logfile to get rid of global variables
* removed 'app' global variable
* linter fixes
* fixed password_*.go build
* fixed BSD build
Removed Warning, Notify and Fatal:
* `Warning` => `Error` or `Info`
* `Notify` => `Info`
* `Fatal` was never used.
Note that --log-level=warning is still supported for backwards
compatibility, but it is the same as --log-level=error.
Co-authored-by: Julio López <julio+gh@kasten.io>
* uitask: added support for reporting string progress info
* server: report current directory as task progress
* snapshot: created reusable Estimate() method to be used during upload, cli estimate and via API
* cli: switched to snapshotfs.Estimate()
* server: added API to estimate snapshot size
* kopia-ui: fixed directory selector
* htmlui: streamlined new snapshot flow and cleaned up policy setting
See https://youtu.be/8p6csuoB3kg
- `repo.Repository` is now read-only and only has methods that can be supported over kopia server
- `repo.RepositoryWriter` has read-write methods that can be supported over kopia server
- `repo.DirectRepository` is read-only and contains all methods of `repo.Repository` plus some low-level methods for data inspection
- `repo.DirectRepositoryWriter` contains write methods for `repo.DirectRepository`
- `repo.Reader` removed and merged with `repo.Repository`
- `repo.Writer` became `repo.RepositoryWriter`
- `*repo.DirectRepository` struct became `repo.DirectRepository`
interface
Getting `{Direct}RepositoryWriter` requires using `NewWriter()` or `NewDirectWriter()` on a read-only repository and multiple simultaneous writers are supported at the same time, each writing to their own indexes and pack blobs.
`repo.Open` returns `repo.Repository` (which is also `repo.RepositoryWriter`).
* content: removed implicit flush on content manager close
* repo: added tests for WriteSession() and implicit flush behavior
* invalidate manifest manager after write session
* cli: disable maintenance in 'kopia server start'
Server will close the repository before completing.
* repo: unconditionally close RepositoryWriter in {Direct,}WriteSession
* repo: added panic in case somebody tries to create RepositoryWriter after closing repository
- used atomic to manage SharedManager.closed
* removed stale example
* linter: fixed spurious failures
Co-authored-by: Julio López <julio+gh@kasten.io>
* linter: upgraded to 1.33, disabled some linters
* lint: fixed 'errorlint' errors
This ensures that all error comparisons use errors.Is() or errors.As().
We will be wrapping more errors going forward so it's important that
error checks are not strict everywhere.
Verified that there are no exceptions for errorlint linter which
guarantees that.
* lint: fixed or suppressed wrapcheck errors
* lint: nolintlint and misc cleanups
Co-authored-by: Julio López <julio+gh@kasten.io>
* logging: cleaned up stderr logging
- do not show module
- do not show timestamps by default (enable with --console-timestamps)
* logging: replaced most printStderr() with log.Info
* cli: additional logging cleanup
* repo: refactored client-specific options (hostname,username,description,readonly) into new struct that is JSON-compatible with current config
* cli: added 'repository set-client' to configure parameters of connected repository
* cli: cleaned up 'repository status' output
Fixes#564
cli: added 'kopia policy set --ignore-cache-dirs' option to control
whether to ignore caches (global default=true)
ui: added checkbox to control 'Ignore Cache Dirs' in policy editor
ignorefs: moved ignoring cache directories to ignorefs layer
Co-authored-by: Julio López <julio+gh@kasten.io>
* This is 99% mechanical:
Extracted repo.Repository interface that only exposes high-level object and manifest management methods, but not blob nor content management.
Renamed old *repo.Repository to *repo.DirectRepository
Reviewed codebase to only depend on repo.Repository as much as possible, but added way for low-level CLI commands to use DirectRepository.
* PR fixes
The hostname/username are now persisted when connecting to repository
in a local config file.
This prevents weird behavior changes when hostname is suddenly changed,
such as when moving between networks.
repo.Repository will now expose Hostname/Username properties which
are always guarnateed to be set, and are used throughout.
Removed --hostname/--username overrides when taking snapshot et.al.
This is mostly mechanical and changes how loggers are instantiated.
Logger is now associated with a context, passed around all methods,
(most methods had ctx, but had to add it in a few missing places).
By default Kopia does not produce any logs, but it can be overridden,
either locally for a nested context, by calling
ctx = logging.WithLogger(ctx, newLoggerFunc)
To override logs globally, call logging.SetDefaultLogger(newLoggerFunc)
This refactoring allowed removing dependency from Kopia repo
and go-logging library (the CLI still uses it, though).
It is now also possible to have all test methods emit logs using
t.Logf() so that they show up in failure reports, which should make
debugging of test failures suck less.
