Commit Graph

1089 Commits

Author SHA1 Message Date
Jarek Kowalski
69d1ccbedf upload: Added CountingUploadProgress and new methods to UploadProgress 2020-02-13 17:23:50 -08:00
Nick
3cb7e37fc1 Add comparer interface and fswalker implementation (#226)
Add comparer interface which gathers data on a path and
compares that data to a new path, returning error if the path
differs in any way from the input data. The details of what
constitutes a difference is left to the implementation.

FSWalker implementation uses Walk and Report to do the data
gathering and comparison. Filters are applied to sort out any
differences that might be expected (e.g. ctime, atime, mtime,
rename of root directory after restore).
2020-02-13 17:07:18 -08:00
Prasad Ghangal
28ab5e0e5f Add test for session token support
Signed-off-by: Prasad Ghangal <prasad.ghangal@gmail.com>
2020-02-12 08:19:24 -08:00
Prasad Ghangal
1c3858a906 Add AWS Session Token support to Kopia
Signed-off-by: Prasad Ghangal <prasad.ghangal@gmail.com>
2020-02-12 08:19:24 -08:00
Jarek Kowalski
a8503a007c server: allow starting server on randomly-allocated port by specifing port 0 (#224)
allow starting server on randomly-allocated port by specifing port 0
2020-02-11 17:08:50 -08:00
Jarek Kowalski
862fc69bff cli: marked --ignore-{file,dir}-errors as enums, so that CLI parser enforces correct values 2020-02-11 06:41:20 -08:00
Nick
162db3e861 [minor] Fix error handling policy print (#221)
Fixes print for error handling policy. Previously was printing
pointer address, now dereference pointer to print boolean itself.
Nil check in case the caller passes a policy with nil. In that
case print "false" which is the behavior in uploader for nil
pointer.
2020-02-10 20:22:55 -08:00
Jarek Kowalski
f8720eda62 travis: re-enabled integration tests on Windows after disabling GPG 2020-02-10 19:13:43 -08:00
Jarek Kowalski
4736e9037e revamped progress output and cleaned up logging
See https://asciinema.org/a/ykx6uzEhKY3451fWEnX9nm9uo
2020-02-10 19:08:35 -08:00
Jarek Kowalski
8d9a042a44 travis: don't run GPG on Windows 2020-02-10 18:57:58 -08:00
Jarek Kowalski
24c425ab2f travis: disable integration tests on windows to see if that fixes builds hanging 2020-02-10 18:42:17 -08:00
Jarek Kowalski
0d9f1dc043 temporary: print task list after running tests on windows, some left-over process may be causing travis builds to fail 2020-02-10 17:52:08 -08:00
Nick
1a24c34df8 Ignore read errors based on policy settings (#207)
* Ignore read errors based on policy settings

Added an error handling policy section. Can independently control error handling for directory and file read errors, toggle-able from the `policy set` command to either "true", "false", or "inherit". If any read error is hit, the error handling will check the effective policy on whether to ignore it or not. Currently there is no differentiation between read error types, though in the future we may want to add the `errors.Is(err, os.ErrPermission)` conditional.

Fix was implemented such that the policy ignores read errors ONLY on child entries of the source. So a snapshot will still fail if the source root directory itself can't be read, but you can ignore the error if a file or a subdirectory in the snapshot source root can't be read. I did this to address some otherwise strange behavior where you would successfully snapshot (because you ignored the error), but couldn't restore that snapshot because nothing really happened during the operation.
2020-02-10 17:35:53 -08:00
Jarek Kowalski
90bd51df14 disable failing sftp test on Windows, to be investigated 2020-02-09 22:54:25 -08:00
Jarek Kowalski
75929f65e9 travis: add integration tests and install-noui on Windows 2020-02-09 22:46:19 -08:00
Jarek Kowalski
edca1733b6 repo: moved password persistence to repository layer 2020-02-09 20:55:07 -08:00
Jarek Kowalski
29e5750686 travis: added bare-bones Windows build that does go test
fixed some issues that prevented go test from passing on Windows:

- webdav client used \ instead of /
- need retries around mmap.Open()
- paths are prefixed with C:\ on windows
- time.Now() does not always move forward on Windows
2020-02-09 20:22:14 -08:00
Jarek Kowalski
729c1e0314 lint: upgrade to 1.23.3 2020-02-09 19:46:27 -08:00
Nick Wright
67e8b6d262 Add lint and test to osx travis-releaser target
Add lint and test to the travis-releaser target for osx.
2020-02-09 18:42:38 -08:00
Nick Wright
6b96f6c019 Place fswalker packages behind build flag
Temporary workaround for compile issues on MacOS and Windows due
to upstream fswalker bug. Only build the reporter and walker
packages as GOOS=linux for now.
2020-02-09 18:42:38 -08:00
Julio Lopez
c247258f41 Simply count medium-sized index blobs
The slice of medium-sized blobs is used to obtain the count of such
blobs from the length of the slice, but it is not used otherwise.
2020-02-08 08:51:37 -08:00
Niraj Tolia
2dc8982485 Website tweaks
- Make sure Slack is front-and-center for users too (community page)
- Tweak homepage
2020-02-07 18:38:38 -08:00
Nick
383c042bf5 Adding low-level FSWalker walker/reporter functionality
Adds a wrapper around `Walk` that takes a Policy (protobuf definition) and performs a walk using it as configuration. The resulting Walk struct pointer is returned. The only exported functionality is unfortunately to read the Policy as a protobuf text file, so the implementation creates a temporary policy file whose lifetime is the duration of the call.

Adds a wrapper around the the FSWalker reporter `Compare` functionality. Takes a config file and two Walk pointers and compares the walks, returning the pb-defined Report struct. Again, the only exported functionality for reading config information is to read it as a protobuf text file. Creates a temporary config file, whose lifetime is the duration of the call, to pass in to the fswalker function.
2020-02-07 12:09:39 -08:00
Julio Lopez
4625e5ba9e Remove content.CompactOptions.MinSmallBlobs
Use MaxSmallBlobs instead. MaxSmallBlobs was not being really used.
Replaced uses of MinSmallBlobs with MaxSmallBlobs and removed
MinSmallBlobs
2020-02-06 21:51:51 -08:00
Niraj Tolia
f509c46a3d Website text edits for clarity 2020-02-06 21:43:06 -08:00
Niraj Tolia
ac19fe5ec3 Use Kopia favicons instead of docsy ones (#204)
Use Kopia favicons instead of docsy ones
Generated based on docs from https://www.docsy.dev/docs/adding-content/iconsimages/
2020-02-06 21:42:12 -08:00
Niraj Tolia
171dfd9f4b nit 2020-02-06 16:50:57 -08:00
Niraj Tolia
d9724b5a62 Make subtext bold for visibility 2020-02-06 16:50:57 -08:00
Niraj Tolia
37af12e61c Darken image a bit 2020-02-06 16:50:57 -08:00
Niraj Tolia
e4d65270af Tweak text 2020-02-06 16:50:57 -08:00
Niraj Tolia
a2729cc52e Tweak 2020-02-06 16:50:57 -08:00
Niraj Tolia
395c1b6091 Update website
- New image of armor inspired by kopia's Polish meaning and the
  protection it provides
- Tweak text for greater clarity

Image from https://unsplash.com/photos/JOzv_pAkcMk under a permissive
license that doesn't require attribution -
https://unsplash.com/license
2020-02-06 16:50:57 -08:00
Julio Lopez
11e3d584de Update nolint directive 2020-02-06 16:50:21 -08:00
Jarek Kowalski
f8ab7340c8 travis: parallelize build using make -j4 2020-02-06 14:46:23 -08:00
Niraj Tolia
4f863bf1d5 Update README.md
Co-Authored-By: Julio López <julio+gh@kasten.io>
2020-02-06 06:55:17 -08:00
Niraj Tolia
c61f18c29c Updated README to reflect we support compression 2020-02-06 06:55:17 -08:00
Niraj Tolia
1e9a01a112 Fix contrib link
Until we have a CONTRIBUTING.md, this will do.
2020-02-05 22:24:17 -08:00
Niraj Tolia
7f6ecdb6c2 Badge fix and review feedback 2020-02-05 21:38:16 -08:00
Niraj Tolia
0559cb8a71 Spelling fixes 2020-02-05 21:38:16 -08:00
Niraj Tolia
e51e4e44fe Make Kopia-specific changes (inspired by Vitess) 2020-02-05 21:38:16 -08:00
Niraj Tolia
0ba102cfa7 Add Governance model
Import from http://oss-watch.ac.uk/resources/meritocraticgovernancemodel
2020-02-05 21:38:16 -08:00
Niraj Tolia
b6095c0e61 Add Code of Conduct 2020-02-05 21:38:16 -08:00
Jarek Kowalski
758dbccd01 kopia-ui: UI/API server security (TLS & base auth)
This automatically launches Kopia CLI with new flags:

`--tls-generate-cert` which generates in-memory TLS certs for the server
and prints its fingerprint to stderr.

`--random-password` which sets up random base auth password

KopiaUI will then listen to the stderr messages and parse the cert
fingerprint and password and automatically provide them in the browser
it opens to show the UI.

This ensures that any local process (or script in a browser, etc.)
other than KopiaUI itself won't have access to the server API nor
UI because they won't know the ephemeral random password and won't
be able to intercept it due to TLS protection.
2020-02-05 17:08:47 -08:00
Jarek Kowalski
0a42ceab27 gcs: added --embed-credentials options which persists --credentials-file as part of Kopia configuration 2020-02-04 22:35:26 -08:00
Jarek Kowalski
0e9e5fe7a7 travis: only deploy website from linux job 2020-02-03 22:48:05 -08:00
Jarek Kowalski
d2360327f8 travis: caching 2020-02-03 19:08:38 -08:00
Jarek Kowalski
80f5a403ce travis: only build the UI on macOS since Linux will build everything except it 2020-02-03 19:08:38 -08:00
Jarek Kowalski
6493a56270 travis: fixed --skip-sign logic 2020-02-02 22:23:08 -08:00
Jarek Kowalski
2b07045ef9 travis: refactored release to be easier to understand
There's now one target, `travis-release` that can be run
locally, or on Travis CI except for:

- code signing (Travis on non-PR runs)
- publishing artifacts to GH releases (Travis on tagged releases)
- creating long-term repository for testing (Travis on tagged releases)
2020-02-02 22:23:08 -08:00
Jarek Kowalski
390f7904e3 changelog: disable sorting 2020-02-01 17:11:09 -08:00