This allows KopiaUI server to start when the repository directory
is not mounted or otherwise unavailable. Connection attempts will
be retried indefinitely and user will see new `Initializing` page.
This also exposes `Open` and `Connect` as tasks allowing the user to see
logs directly in the UI and cancel the operation.
Also simplified validation test suite, which will simply test whether
the provider supports DoNotRecreate or properly rejects it without
external configuration.
* Add a new PutBlob option and blob error type
When `DoNotRecreate` is set as true, the blob put operation should
only succeed if no blob with the given blob ID already exists.
Othwerwise, `ErrBlobAlreadyExists` is returned.
* Validate default storage providers' support
By default, storage providers should not support idempotent creates.
This commit adds error handling to exit early if `DoNotRecreate` is
set to true. The commit also verifies this behavior in the provider
validation test.
* Implement support for new option in GCS storage
* Push PutBlob option handling down to Impl
When PutBlob options were introduced, error handling logic for them
was implemented for the Sharded storage interface. However, the
behavior of different providers that implement Sharded can be
different, so it's better to push the options down to be processed in
the provider implementations.
* Introduce new error type for unsupported put opts
To unify error handling code and make it more maintainable, introduce
a new error type `blob.ErrUnsupportedPutBlobOption`, which is to be
returned whenever a storage provider implementation is given put
options it does not support.
When a sharded directory is missing do not attempt to create all
its parents, but only children of the repository root.
This way when a top-level directory is unmounted, we won't recreate
it unnecessarily.
This is implemented for filesystem and SFTP providers.
* feat: persisting retention options in repository blob
- plumb retention parameters through wrapped storage
- generalize aes encryption mechanism
- rewrite the retention blob on password change
- do not write retention blob when empty
* handle retention-blob not-found failures
* cli params to set retention modes on repository create
* enable versioned map mock storage with retention settings
* adding unit tests
* write format and retention blob with retention settings if available
* rename certain functions and constants specific to format blob
* delete retention cache on password-change
* fix: replace SetTime() api call with TouchBlob()
* Update repo/repository_test.go
Co-authored-by: Nick <nick@kasten.io>
* pr feedback and codecov improvements
* fix: rename retention-blob structures to generic blob-cfg
* fix: remove minio dependency on retention constants
Co-authored-by: Shikhar Mall <shikhar@kasten.io>
Co-authored-by: Nick <nick@kasten.io>
* fix(security): prevent cross-site request forgery in the UI website
This fixes a [cross-site request forgery (CSRF)](https://en.wikipedia.org/wiki/Cross-site_request_forgery)
vulnerability in self-hosted UI for Kopia server.
The vulnerability allows potential attacker to make unauthorized API
calls against a running Kopia server. It requires an attacker to trick
the user into visiting a malicious website while also logged into a
Kopia website.
The vulnerability only affected self-hosted Kopia servers with UI. The
following configurations were not vulnerable:
* Kopia Repository Server without UI
* KopiaUI (desktop app)
* command-line usage of `kopia`
All users are strongly recommended to upgrade at the earliest
convenience.
* pr feedback
* testing: added logging to figure out root cause of the flake in TestSnapshotCounters
* fixed test flake caused by not waiting for upload task to be created
This adds new set of APIs `/api/v1/control/*` which can be used to administratively control a running server.
Once the server is started, the administrative user can control it
using CLI commands:
export KOPIA_SERVER_ADDRESS=...
export KOPIA_SERVER_CERT_FINGERPRINT=...
export KOPIA_SERVER_PASSWORD=...
* `kopia server status` - displays status of sources managed by the server
* `kopia server snapshot` - triggers server-side upload of snapshots for managed sources
* `kopia server cancel` - cancels upload of snapshots for managed sources
* `kopia server pause` - pauses scheduled snapshots for managed sources
* `kopia server resume` - resumes scheduled snapshots for managed sources
* `kopia server refresh` - causes server to resynchronize with externally-made changes, such as policies or new sources
* `kopia server flush` - causes server to flush all pending writes
* `kopia server shutdown` - graceful shutdown of the server
Authentication uses new user `server-control` and is disabled
by default. To enable it when starting the server, provide the password
using one of the following methods:
* `--server-control-password`
* `--random-server-control-password`
* `.htpasswd` file
* `KOPIA_SERVER_CONTROL_PASSWORD` environment variable
This change allows us to tighten the API security and remove some
methods that UI user was able to call, but which were not needed.
* sharded: plumbed through blob.PutOptions
* blob: removed blob.Storage.SetTime() method
This was only used for `kopia repo sync-to` and got replaced with
an equivalent blob.PutOptions.SetTime, which wehn set to non-zero time
will attempt to set the modification time on a file.
Since some providers don't support changing modification time, we
are able to emulate it using per-blob metadata (on B2, Azure and GCS),
sadly S3 is still unsupported, because it does not support returning
metadata in list results.
Also added PutOptions.GetTime, which when set to not nil, will
populate the provided variable with actual time that got assigned
to the blob.
Added tests that verify that each provider supports GetTime
and SetTime according to this spec.
* blob: additional test coverage for filesystem storage
* blob: added PutBlobAndGetMetadata() helper and used where appropriate
* fixed test failures
* pr feedback
* Update repo/blob/azure/azure_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* blobtesting: fixed object_locking_map.go
* blobtesting: removed SetTime from ObjectLockingMap
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
The source code for htmlui is now in github.com/kopia/htmlui
GitHub Actions will drop compiled builds in github.com/kopia/htmluibuild
where they are now used as a go module dependency.
This greatly simplifies the build and improves the security,
because Kopia will be consuming pre-built htmlui.
This also means kopia can now installed with embedded UI using:
`go install github.com/kopia/kopia@latest`
Previously epoch manager would discard (on read) any index blobs
associated with epoch N written after epoch N+2 has already started.
There was no signal to the writer that the write is actually bad. This
change reports slow writes as a failure and disables read-time filtering.
This is important for repositories copied using sync-to because it
may not necessarily preserve the modification time leading to problems
with restore.
* blob: speed up filesystem storage test
* blob: introduced osInterface to allow unit testing of filesystem calls
* testing: added a bunch of unit tests for corner cases in filesystem storage
Fixed a few cases where we were missing the retry loop.
* cli: refactored snapshot list
* cli: show range tags in snapshot list
For example if N snapshots are coalesced together because they
have identical roots we may emit now:
```
2021-03-31 23:09:27 PDT ked3400debc7dd61baffab070bafd59cd (monthly-10)
2021-04-30 06:12:53 PDT kd0576d212e55a831b7ff1636f90a7233 (monthly-4..9)
+ 5 identical snapshots until 2021-09-30 23:00:19 PDT
2021-10-31 23:22:25 PDT k846bf22aa2863d27f05e820f840b14f8 (monthly-3)
2021-11-08 21:29:31 PST k5793ddcd61ef27b93c75ab74a5828176 (latest-1..3,hourly-1..13,daily-1..7,weekly-1..4,monthly-1..2,annual-1)
+ 18 identical snapshots until 2021-12-04 10:09:54 PST
```
* server: server-side coalescing of snapshot
* ui: added coalescing of retention tags
* ui: changed how PolicyEditor is instantiated via a route
* server: added paths/resolve API
* server: refresh affected source manager after policy change
Also switched 15-second refresh cycle which is way too aggressive
to 30-minute cycle (manual refresh button can be used if needed).
* policy: allow overriding top-level policy for estimation
* server: changed source create API to always require policy
* ui: streamlined new snapshot and estimate flow
* linter fix
* logging: added log rotation and improved predictability of log sweep
With this change logs will be rotated every 50 MB, which prevents
accumulation of giant files while server is running.
This change will also guarantee that log sweep completes at least once
before each invocation of Kopia finishes. Previously it was a goroutine
that was not monitored for completion.
Flags can be used to override default behaviors:
* `--max-log-file-segment-size`
* `--no-wait-for-log-sweep` - disables waiting for full log sweep
Fixes#1561
* logging: added --log-dir-max-total-size-mb flag
This limits the total size of all logs in a directory to 1 GB.
This is done by protecting newly added cache items from being swept for
X amount of time where X defaults to:
* `metadata` - 24 hours (new)
* `data` - 10 min (new)
* `indexes` - 1 hours (same as today)
Fixes#1540
Added policy.Definition which allows us to precisely report where
each piece of policy came from.
Fixed a one-off bug with "noParent", which prevented merging of parent
policies one level too soon.
Added a whole bunch of merging helpers and generic reflection-based
test that ensures every single merge is tested.
* throtting: implemented a Throttler based on token bucket and configurable window.
* cli: rewired throttle options to use common Limits structure and helpers
The JSON is backwards compatible.
* blob: remove explicit throttling from gcs,s3,b2 & azure
* cleanup: removed internal/throttle
* repo: add throttling wrapper around storage at the repository level
* throttling: expose APIs to get limits and add validation
* server: expose API to get/set throttle in a running server
* pr feedback
* blob: changed default shards from {3,3} to {1,3}
Turns out for very large repository around 100TB (5M blobs),
we end up creating max ~16M directories which is way too much
and slows down listing. Currently each leaf directory only has a handful
of files.
Simple sharding of {3} should work much better and will end up creating
directories with meaningful shard sizes - 12 K files per directory
should not be too slow and will reduce the overhead of listing by
4096 times.
The change is done in a backwards-compatible way and will respect
custom sharding (.shards) file written by previous 0.9 builds
as well as older repositories that don't have the .shards file (which
we assume to be {3,3}).
* fixed compat tests
* fixed new gocritic violations
* fixed new 'contextcheck' violations
* fixed 'gosec' warnings
* suppressed ireturn and varnamelen linters
* fixed tenv violations, enabled building robustness tests on arm64
* fixed remaining linux failures
* makefile: fixed 'lint-all' target when running on arm64
* linter: increase deadline
* disable nilnil linter - to be enabled in separate PR
The issue in #1439 was caused by goroutine context being associated
with the HTTP request so it became canceled soon after the request was
over, thus the goroutine to run maintenance never ran.
Fixed by adding ctxutil.Detach()
Also fixed logging by passing top-level contexts to requests
and added --log-server-requests flag to `server start` which enables
request logging.
Instead of attempting maintenance every 10 minutes we will do a longer
sleep until the predicted next maintenance time (or 4 hours, whichever
is shorter).
Related #1439
This was introduced by a refactoring in #1361 - unlike
ioutil.ReadDir() which internally handles list/delete race and always
returns os.FileInfo, Info() on os.DirEntry can fail if a file
is deleted right after listing it.
Fixes#1486
