Commit Graph

17 Commits

Author SHA1 Message Date
Jarek Kowalski
4fd0bcf7dc cli: 'snapshot create' switched all stderr output to use logger (#663)
This allows 'kopia snap create --all --log-level=warning --no-progress'
to produce no console output unless an error is encountered.
2020-10-05 17:28:00 -07:00
Jarek Kowalski
9551d2495d upload: scan the directory to be uploaded in parallel to estimate the amount of data to be uploaded (#622)
This allows better progress indicator in the CLI and UI.
The percentage completed is not displayed until estimate is available.

Quick demo: https://asciinema.org/a/O7ktcWSgaGUPfJwhzc65mMWM1
2020-09-17 23:59:18 -07:00
Jarek Kowalski
f0b97b960b Fixed checkpointing to not restart the entire upload process (#594)
* object: added Checkpoint() method to object writer

* upload: refactored code structure to allow better checkpointing

* upload: removed Checkpoint() method from UploadProgress

* Update fs/entry.go

Co-authored-by: Julio López <julio+gh@kasten.io>
2020-09-12 22:36:22 -07:00
Jarek Kowalski
1a8fcb086c Added endurance test which tests kopia over long time scale (#558)
Globally replaced all use of time with internal 'clock' package
which provides indirection to time.Now()

Added support for faking clock in Kopia via KOPIA_FAKE_CLOCK_ENDPOINT

logfile: squelch annoying log message

testenv: added faketimeserver which serves time over HTTP

testing: added endurance test which tests kopia over long time scale

This creates kopia repository and simulates usage of Kopia over multiple
months (using accelerated fake time) to trigger effects that are only
visible after long time passage (maintenance, compactions, expirations).

The test is not used part of any test suite yet but will run in
post-submit mode only, preferably 24/7.

testing: refactored internal/clock to only support injection when
'testing' build tag is present
2020-08-26 23:03:46 -07:00
Jarek Kowalski
9a6dea898b Linter upgrade to v1.30.0 (#526)
* fixed godot linter errors
* reformatted source with gofumpt
* disabled some linters
* fixed nolintlint warnings
* fixed gci warnings
* lint: fixed 'nestif' warnings
* lint: fixed 'exhaustive' warnings
* lint: fixed 'gocritic' warnings
* lint: fixed 'noctx' warnings
* lint: fixed 'wsl' warnings
* lint: fixed 'goerr113' warnings
* lint: fixed 'gosec' warnings
* lint: upgraded linter to 1.30.0
* lint: more 'exhaustive' warnings

Co-authored-by: Nick <nick@kasten.io>
2020-08-12 19:28:53 -07:00
Jarek Kowalski
293cb10471 cosmetic changes to maintenance behavior (#461)
- run maintenance even if the command is about to return an error
  (otherwise if folks have persistent error causing snapshots to fail
  they will never run maintenance)
- disable progress output after snapshotting so that
  'kopia snapshot --all' output is clean
2020-06-01 23:38:18 -07:00
Jarek Kowalski
1f1682b2cc Snapshot checkpointing (#410)
* snapshot: support for periodic checkpointing of snapshots in progress

For each snapshot that takes longer than 45 minutes, we trigger
internal cancellation, save the manifest and restart the snapshot
at which point all files will be cached.

This helps ensure the property that no file or directory objects
in the repository remain unreachable from a snapshot root for more than
one hour, which is important from GC perspective.

* nit: unified spelling 'cancelled' => 'canceled'
2020-04-07 17:54:21 -07:00
Jarek Kowalski
9b68a631e6 Highlight snapshot errors in the UI and CLI (#376)
* upload: exposed numFailed and failedEntries on directory summary

* cli: better present snapshot errors

* htmlui: display snapshot errors
2020-03-22 14:18:47 -07:00
Jarek Kowalski
d930f06a27 cli: added flags to control progress output
--no-progress - disables progress completely
--progress-update-interval=T - controls how frequently progress is updated

Fixes #344
2020-03-17 05:11:42 -07:00
Jarek Kowalski
331ff076bb cli: added number of files that are in the process of being hashed 2020-03-14 15:56:09 -07:00
Jarek Kowalski
cd35e3bab5 cli: 'snapshot migrate' improvements to help with data migration
- cleaned up migration progress output
- fixed migration idempotency
- added migration of policies
- renamed --parallelism to --parallel
- improved e2e test
- do not prompt for password to source repository if persisted
2020-03-07 21:47:32 -08:00
Jarek Kowalski
897483299f Kopia UI & CLI: support for progress indicator (#268)
Percentage based on last-known snapshot size

* server: exposed last completed snapshot size in the API
* cli: added support for progress indicator (percentage based on last-known snapshot size)
* htmlui: added progress indicator in the UI (percentage based on last-known snapshot size)
2020-02-24 17:55:02 -08:00
Jarek Kowalski
4736e9037e revamped progress output and cleaned up logging
See https://asciinema.org/a/ykx6uzEhKY3451fWEnX9nm9uo
2020-02-10 19:08:35 -08:00
Jarek Kowalski
ac70a38101 lint: upgraded to 1.22.2 and make lint issues a build failure
fixed or silenced linter warnings, mostly due to magic numeric constants
2020-01-03 16:39:30 -08:00
Jarek Kowalski
6217df1a87 lint: switched to 1.21 and fixed a ton of whitespace issues discovered
by new wsl linter
2019-11-26 06:49:49 -08:00
Jarek Kowalski
72520029b0 golangci-lint: added more linters
Also fixed pre-existing lint errors.
2019-06-02 22:56:57 -07:00
Jarek Kowalski
08d62d60f1 revamped progress bar by including hashing and uploading part separately 2018-09-14 11:56:48 -07:00