Commit Graph

15 Commits

Author SHA1 Message Date
Eng Zer Jun
73e492c9db refactor: move from io/ioutil to io and os package (#1360)
* refactor: move from io/ioutil to io and os package

The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* chore: remove //nolint:gosec for os.ReadFile

At the time of this commit, the G304 rule of gosec does not include the
`os.ReadFile` function. We remove `//nolint:gosec` temporarily until
https://github.com/securego/gosec/pull/706 is merged.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-06 08:39:10 -07:00
Jarek Kowalski
35d0f31c0d huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)
This helps recycle buffers more efficiently during snapshots.
Also, improved memory tracking, enabled profiling flags and added pprof
by default.
2021-08-20 08:45:10 -07:00
Jarek Kowalski
40510c043d Support for content-level compression (#1076)
* cli: added a flag to create repository with v2 index features

* content: plumb through compression.ID parameter to content.Manager.WriteContent()

* content: expose content.Manager.SupportsContentCompression

This allows object manager to decide whether to create compressed object
or let the content manager do it.

* object: if compression is requested and the repo supports it, pass compression ID to the content manager

* cli: show compression status in 'repository status'

* cli: output compression information in 'content list' and 'content stats'

* content: compression and decompression support

* content: unit tests for compression

* object: compression tests

* testing: added integration tests against v2 index

* testing: run all e2e tests with and without content-level compression

* htmlui: added UI for specifying index format on creation

* cli: additional tests for 'content ls' and 'content stats'

* applied pr suggestions
2021-05-22 05:35:27 -07:00
Jarek Kowalski
fcd507a56d Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)
* cli: fixed remaining testability indirections for output and logging

* cli: added cli.RunSubcommand() which is used in testing to execute a subcommand in the same process

* tests: refactored most e2e tests to invoke kopia subcommands in-process

* Makefile: enable code coverage for cli/ and internal/

* testing: pass 'testing' tag to unit tests which uses much faster (insecure) password hashing scheme

* Makefile: push coverage from PRs again

* tests: disable buffer management to reduce memory usage on ARM

* cli: fixed misaligned atomic field on ARMHF

also temporarily fixed statup-time benign race condition when setting
default on the timeZone variable, which is the last global variable.
2021-05-11 22:26:28 -07:00
Jarek Kowalski
d2288c443f cli: major refactoring (#1046)
cli: major refactoring of how CLI commands are registered

The goal is to eliminate flags as global variables to allow for better
testing. Each command and subcommand and most sets of flags are now
their own struct with 'setup()' methods that attached the flags or
subcommand to the provided parent.

This change is 94.3% mechanical, but is fully organic and hand-made.

* introduced cli.appServices interface which provides the environment in which commands run
* remove auto-maintenance global flag
* removed globals in memory_tracking.go
* removed globals from cli_progress.go
* removed globals from the update_check.go
* moved configPath into TheApp
* removed remaining globals from config.go
* refactored logfile to get rid of global variables
* removed 'app' global variable
* linter fixes
* fixed password_*.go build
* fixed BSD build
2021-05-03 10:28:00 -07:00
Jarek Kowalski
e03971fc59 Upgraded linter to v1.33.0 (#734)
* linter: upgraded to 1.33, disabled some linters

* lint: fixed 'errorlint' errors

This ensures that all error comparisons use errors.Is() or errors.As().
We will be wrapping more errors going forward so it's important that
error checks are not strict everywhere.

Verified that there are no exceptions for errorlint linter which
guarantees that.

* lint: fixed or suppressed wrapcheck errors

* lint: nolintlint and misc cleanups

Co-authored-by: Julio López <julio+gh@kasten.io>
2020-12-21 22:39:22 -08:00
Jarek Kowalski
ad4b222939 cli: added support for copying (or moving) snapshot history (#703)
Both source and destination can be specified using user@host,
@host or user@host:/path where destination values override the
corresponding parts of the source, so both targeted
and mass copying is supported.

Supported combinations are:

Source:             Destination         Behavior
---------------------------------------------------
@host1              @host2              copy snapshots from all users of host1
user1@host1         @host2              copy all snapshots to user1@host2
user1@host1         user2@host2         copy all snapshots to user2@host2
user1@host1:/path1  @host2              copy to user1@host2:/path1
user1@host1:/path1  user2@host2         copy to user2@host2:/path1
user1@host1:/path1  user2@host2:/path2  copy snapshots from single path

When --move is specified, the matching source snapshots are also deleted.

* cli: upgraded kingpin to latest version (not tagged)

This allows using `EnableFileExpansion` to disable treating
arguments prefixed with "@" as file includes.
2020-12-04 16:34:55 -08:00
Jarek Kowalski
514df69afa performance: added wrapper around io.Copy()
this pools copy buffers so they can be reused instead of throwing away
after each io.Copy()
2020-03-10 21:52:30 -07:00
Jarek Kowalski
6217df1a87 lint: switched to 1.21 and fixed a ton of whitespace issues discovered
by new wsl linter
2019-11-26 06:49:49 -08:00
Jarek Kowalski
1a7a02ddbe cleanup imports by grouping all local imports together 2019-06-01 10:57:55 -07:00
Jarek Kowalski
a6a153b22e switched fmt.Errorf() to errors.Wrap() 2019-05-11 12:34:14 -07:00
Jarek Kowalski
3c26be391a added global hidden --timezone flag for changing the timezone in which the timestamps are displayed 2018-10-06 09:54:43 -07:00
Jarek Kowalski
304b43d27c cli: additional output options for 'block list' 2018-09-15 17:32:58 -07:00
Jarek Kowalski
e76f1f8a72 refactored various show command into one 2017-10-24 21:59:40 -07:00
Jarek Kowalski
a48d54f00b added 'block show' similar to 'object show', but operating on virtual blocks 2017-10-24 21:15:22 -07:00