Generalize a couple of functions in the units package using generics.
This allows removing duplicate code and simplifying callers by removing unnecessary integer conversions.
Additional cleanups:
- make "/s" part of the Printf format string ;
- simplify setSizeMBParameter;
- generalize cli.maybeHumanReadable*` helpers;
- remove unneeded receiver in commandRepositorySetParameters helpers.
This removes tons of boilerplate code around:
- retry loop
- connection management
- storage registration
* used generics in runInParallel
* introduced generics in freepool
* introduced strong typing for workshare.Pool and workshare.AsyncGroup
* fixed linter error on openbsd
* Update display on repository summary
* Apply throughout app
* Situate units_test
* Update Command Line documentation
* Envar cleanup
* Rename to BytesString
* Restore envar string available for test
* Remove extraneous empty check and restore UIPreferences field for frontend
* PR: config bool cleanup and missed `BaseEnv`s
* Fix lint and test
cli: major refactoring of how CLI commands are registered
The goal is to eliminate flags as global variables to allow for better
testing. Each command and subcommand and most sets of flags are now
their own struct with 'setup()' methods that attached the flags or
subcommand to the provided parent.
This change is 94.3% mechanical, but is fully organic and hand-made.
* introduced cli.appServices interface which provides the environment in which commands run
* remove auto-maintenance global flag
* removed globals in memory_tracking.go
* removed globals from cli_progress.go
* removed globals from the update_check.go
* moved configPath into TheApp
* removed remaining globals from config.go
* refactored logfile to get rid of global variables
* removed 'app' global variable
* linter fixes
* fixed password_*.go build
* fixed BSD build
* Option to print out the commands for using crypto, splitter and compression
Co-authored-by: Janne Johansson <janne.johansson@safespring.com>
Co-authored-by: Jarek Kowalski <jaak@jkowalski.net>
* linter: upgraded to 1.33, disabled some linters
* lint: fixed 'errorlint' errors
This ensures that all error comparisons use errors.Is() or errors.As().
We will be wrapping more errors going forward so it's important that
error checks are not strict everywhere.
Verified that there are no exceptions for errorlint linter which
guarantees that.
* lint: fixed or suppressed wrapcheck errors
* lint: nolintlint and misc cleanups
Co-authored-by: Julio López <julio+gh@kasten.io>
* logging: cleaned up stderr logging
- do not show module
- do not show timestamps by default (enable with --console-timestamps)
* logging: replaced most printStderr() with log.Info
* cli: additional logging cleanup
* object: refactored writer to detect split points before writing
This introduces new primitive that will be moved into splitters
themselves in subsequent commits. I'm doing this in small steps to
ensure we don't regress at any time.
* splitter: refactored TestSplitters test
This is use slow (byte-by-byte) and fast (nextSplitPoint) methods of
determining split points.
Note nextSplitPoint is not implemented by splitters yet, but this
verifies that the test is expecting the right thing.
* object: splitter refactoring - replaced ShouldSplit() with NextSplitPoint() everywhere, still not optimized
* splitter: added additional dimension to splitter_test
We split either in large chunks or one byte at a time to catch
the corner cases in the splitter implementation.
* splitter: optimized splitters using NextSplitPoint primitive
This improves splitter performance by about 40% (buzhash) and makes
it virtually free for FIXED splitter.
Globally replaced all use of time with internal 'clock' package
which provides indirection to time.Now()
Added support for faking clock in Kopia via KOPIA_FAKE_CLOCK_ENDPOINT
logfile: squelch annoying log message
testenv: added faketimeserver which serves time over HTTP
testing: added endurance test which tests kopia over long time scale
This creates kopia repository and simulates usage of Kopia over multiple
months (using accelerated fake time) to trigger effects that are only
visible after long time passage (maintenance, compactions, expirations).
The test is not used part of any test suite yet but will run in
post-submit mode only, preferably 24/7.
testing: refactored internal/clock to only support injection when
'testing' build tag is present
This is mostly mechanical and changes how loggers are instantiated.
Logger is now associated with a context, passed around all methods,
(most methods had ctx, but had to add it in a few missing places).
By default Kopia does not produce any logs, but it can be overridden,
either locally for a nested context, by calling
ctx = logging.WithLogger(ctx, newLoggerFunc)
To override logs globally, call logging.SetDefaultLogger(newLoggerFunc)
This refactoring allowed removing dependency from Kopia repo
and go-logging library (the CLI still uses it, though).
It is now also possible to have all test methods emit logs using
t.Logf() so that they show up in failure reports, which should make
debugging of test failures suck less.
The splitter in question was depending on
github.com/silvasur/buzhash which is not licensed according to FOSSA bot
Switched to new faster implementation of buzhash, which is
unfortunately incompatible and will split the objects in different
places.
This change is be semi-breaking - old repositories can be read, but
when uploading large objects they will be re-uploaded where previously
they would be de-duped.
Also added 'benchmark splitters' subcommand and moved 'block cryptobenchmark'
subcommand to 'benchmark crypto'.
