Commit Graph
1089 Commits
Author SHA1 Message Date
Jarek Kowalski 69d1ccbedf upload: Added CountingUploadProgress and new methods to UploadProgress 2020-02-13 17:23:50 -08:00
Nick 3cb7e37fc1 Add comparer interface and fswalker implementation (#226)
Add comparer interface which gathers data on a path and
compares that data to a new path, returning error if the path
differs in any way from the input data. The details of what
constitutes a difference is left to the implementation.

FSWalker implementation uses Walk and Report to do the data
gathering and comparison. Filters are applied to sort out any
differences that might be expected (e.g. ctime, atime, mtime,
rename of root directory after restore).
2020-02-13 17:07:18 -08:00
Prasad Ghangal 28ab5e0e5f Add test for session token support
Signed-off-by: Prasad Ghangal <prasad.ghangal@gmail.com>
2020-02-12 08:19:24 -08:00
Prasad Ghangal 1c3858a906 Add AWS Session Token support to Kopia
Signed-off-by: Prasad Ghangal <prasad.ghangal@gmail.com>
2020-02-12 08:19:24 -08:00
Jarek Kowalski a8503a007c server: allow starting server on randomly-allocated port by specifing port 0 (#224)
allow starting server on randomly-allocated port by specifing port 0
2020-02-11 17:08:50 -08:00
Jarek Kowalski 862fc69bff cli: marked --ignore-{file,dir}-errors as enums, so that CLI parser enforces correct values 2020-02-11 06:41:20 -08:00
Nick 162db3e861 [minor] Fix error handling policy print (#221)
Fixes print for error handling policy. Previously was printing
pointer address, now dereference pointer to print boolean itself.
Nil check in case the caller passes a policy with nil. In that
case print "false" which is the behavior in uploader for nil
pointer.
2020-02-10 20:22:55 -08:00
Jarek Kowalski f8720eda62 travis: re-enabled integration tests on Windows after disabling GPG 2020-02-10 19:13:43 -08:00
Jarek Kowalski 4736e9037e revamped progress output and cleaned up logging
See https://asciinema.org/a/ykx6uzEhKY3451fWEnX9nm9uo
2020-02-10 19:08:35 -08:00
Jarek Kowalski 8d9a042a44 travis: don't run GPG on Windows 2020-02-10 18:57:58 -08:00
Jarek Kowalski 24c425ab2f travis: disable integration tests on windows to see if that fixes builds hanging 2020-02-10 18:42:17 -08:00
Jarek Kowalski 0d9f1dc043 temporary: print task list after running tests on windows, some left-over process may be causing travis builds to fail 2020-02-10 17:52:08 -08:00
Nick 1a24c34df8 Ignore read errors based on policy settings (#207)
* Ignore read errors based on policy settings

Added an error handling policy section. Can independently control error handling for directory and file read errors, toggle-able from the `policy set` command to either "true", "false", or "inherit". If any read error is hit, the error handling will check the effective policy on whether to ignore it or not. Currently there is no differentiation between read error types, though in the future we may want to add the `errors.Is(err, os.ErrPermission)` conditional.

Fix was implemented such that the policy ignores read errors ONLY on child entries of the source. So a snapshot will still fail if the source root directory itself can't be read, but you can ignore the error if a file or a subdirectory in the snapshot source root can't be read. I did this to address some otherwise strange behavior where you would successfully snapshot (because you ignored the error), but couldn't restore that snapshot because nothing really happened during the operation.
2020-02-10 17:35:53 -08:00
Jarek Kowalski 90bd51df14 disable failing sftp test on Windows, to be investigated 2020-02-09 22:54:25 -08:00
Jarek Kowalski 75929f65e9 travis: add integration tests and install-noui on Windows 2020-02-09 22:46:19 -08:00
Jarek Kowalski edca1733b6 repo: moved password persistence to repository layer 2020-02-09 20:55:07 -08:00
Jarek Kowalski 29e5750686 travis: added bare-bones Windows build that does go test
fixed some issues that prevented go test from passing on Windows:

- webdav client used \ instead of /
- need retries around mmap.Open()
- paths are prefixed with C:\ on windows
- time.Now() does not always move forward on Windows
2020-02-09 20:22:14 -08:00
Jarek Kowalski 729c1e0314 lint: upgrade to 1.23.3 2020-02-09 19:46:27 -08:00
Nick Wright 67e8b6d262 Add lint and test to osx travis-releaser target
Add lint and test to the travis-releaser target for osx.
2020-02-09 18:42:38 -08:00
Nick Wright 6b96f6c019 Place fswalker packages behind build flag
Temporary workaround for compile issues on MacOS and Windows due
to upstream fswalker bug. Only build the reporter and walker
packages as GOOS=linux for now.
2020-02-09 18:42:38 -08:00
Julio Lopez c247258f41 Simply count medium-sized index blobs
The slice of medium-sized blobs is used to obtain the count of such
blobs from the length of the slice, but it is not used otherwise.
2020-02-08 08:51:37 -08:00
Niraj Tolia 2dc8982485 Website tweaks
- Make sure Slack is front-and-center for users too (community page)
- Tweak homepage
2020-02-07 18:38:38 -08:00
Nick 383c042bf5 Adding low-level FSWalker walker/reporter functionality
Adds a wrapper around `Walk` that takes a Policy (protobuf definition) and performs a walk using it as configuration. The resulting Walk struct pointer is returned. The only exported functionality is unfortunately to read the Policy as a protobuf text file, so the implementation creates a temporary policy file whose lifetime is the duration of the call.

Adds a wrapper around the the FSWalker reporter `Compare` functionality. Takes a config file and two Walk pointers and compares the walks, returning the pb-defined Report struct. Again, the only exported functionality for reading config information is to read it as a protobuf text file. Creates a temporary config file, whose lifetime is the duration of the call, to pass in to the fswalker function.
2020-02-07 12:09:39 -08:00
Julio Lopez 4625e5ba9e Remove content.CompactOptions.MinSmallBlobs
Use MaxSmallBlobs instead. MaxSmallBlobs was not being really used.
Replaced uses of MinSmallBlobs with MaxSmallBlobs and removed
MinSmallBlobs
2020-02-06 21:51:51 -08:00
Niraj Tolia f509c46a3d Website text edits for clarity 2020-02-06 21:43:06 -08:00
Niraj Tolia ac19fe5ec3 Use Kopia favicons instead of docsy ones (#204)
Use Kopia favicons instead of docsy ones
Generated based on docs from https://www.docsy.dev/docs/adding-content/iconsimages/
2020-02-06 21:42:12 -08:00
Niraj Tolia 171dfd9f4b nit 2020-02-06 16:50:57 -08:00
Niraj Tolia d9724b5a62 Make subtext bold for visibility 2020-02-06 16:50:57 -08:00
Niraj Tolia 37af12e61c Darken image a bit 2020-02-06 16:50:57 -08:00
Niraj Tolia e4d65270af Tweak text 2020-02-06 16:50:57 -08:00
Niraj Tolia a2729cc52e Tweak 2020-02-06 16:50:57 -08:00
Niraj Tolia 395c1b6091 Update website
- New image of armor inspired by kopia's Polish meaning and the
  protection it provides
- Tweak text for greater clarity

Image from https://unsplash.com/photos/JOzv_pAkcMk under a permissive
license that doesn't require attribution -
https://unsplash.com/license
2020-02-06 16:50:57 -08:00
Julio Lopez 11e3d584de Update nolint directive 2020-02-06 16:50:21 -08:00
Jarek Kowalski f8ab7340c8 travis: parallelize build using make -j4 2020-02-06 14:46:23 -08:00
Niraj ToliaandJulio López 4f863bf1d5 Update README.md
Co-Authored-By: Julio López <julio+gh@kasten.io>
2020-02-06 06:55:17 -08:00
Niraj Tolia c61f18c29c Updated README to reflect we support compression 2020-02-06 06:55:17 -08:00
Niraj Tolia 1e9a01a112 Fix contrib link
Until we have a CONTRIBUTING.md, this will do.
2020-02-05 22:24:17 -08:00
Niraj Tolia 7f6ecdb6c2 Badge fix and review feedback 2020-02-05 21:38:16 -08:00
Niraj Tolia 0559cb8a71 Spelling fixes 2020-02-05 21:38:16 -08:00
Niraj Tolia e51e4e44fe Make Kopia-specific changes (inspired by Vitess) 2020-02-05 21:38:16 -08:00
Niraj Tolia 0ba102cfa7 Add Governance model
Import from http://oss-watch.ac.uk/resources/meritocraticgovernancemodel
2020-02-05 21:38:16 -08:00
Niraj Tolia b6095c0e61 Add Code of Conduct 2020-02-05 21:38:16 -08:00
Jarek Kowalski 758dbccd01 kopia-ui: UI/API server security (TLS & base auth)
This automatically launches Kopia CLI with new flags:

`--tls-generate-cert` which generates in-memory TLS certs for the server
and prints its fingerprint to stderr.

`--random-password` which sets up random base auth password

KopiaUI will then listen to the stderr messages and parse the cert
fingerprint and password and automatically provide them in the browser
it opens to show the UI.

This ensures that any local process (or script in a browser, etc.)
other than KopiaUI itself won't have access to the server API nor
UI because they won't know the ephemeral random password and won't
be able to intercept it due to TLS protection.
2020-02-05 17:08:47 -08:00
Jarek Kowalski 0a42ceab27 gcs: added --embed-credentials options which persists --credentials-file as part of Kopia configuration 2020-02-04 22:35:26 -08:00
Jarek Kowalski 0e9e5fe7a7 travis: only deploy website from linux job 2020-02-03 22:48:05 -08:00
Jarek Kowalski d2360327f8 travis: caching 2020-02-03 19:08:38 -08:00
Jarek Kowalski 80f5a403ce travis: only build the UI on macOS since Linux will build everything except it 2020-02-03 19:08:38 -08:00
Jarek Kowalski 6493a56270 travis: fixed --skip-sign logic 2020-02-02 22:23:08 -08:00
Jarek Kowalski 2b07045ef9 travis: refactored release to be easier to understand
There's now one target, `travis-release` that can be run
locally, or on Travis CI except for:

- code signing (Travis on non-PR runs)
- publishing artifacts to GH releases (Travis on tagged releases)
- creating long-term repository for testing (Travis on tagged releases)
2020-02-02 22:23:08 -08:00
Jarek Kowalski 390f7904e3 changelog: disable sorting 2020-02-01 17:11:09 -08:00