* feat(cli): send error notifications and snapshot reports
Notifications will be sent to all configured notification profiles
according to their severity levels.
The following events will trigger notifications:
- Snapshot is created (CLI only, severity >= report)
- Server Maintenance error occurs (CLI, server and UI, severity >= error)
- Any other CLI error occurs (CLI only, severity >= error).
A flag `--no-error-notifications` can be used to disable error notifications.
* added template tests
* improved time formatting in templates
* plumb through notifytemplate.Options
* more testing for formatting options
* fixed default date format to RFC1123
* Configure compressor for k and x prefixed content
Adds metadata compression setting to policy
Add support to configure compressor for k and x prefixed content
Set zstd-fastest as the default compressor for metadata in the policy
Adds support to set and show metadata compression to kopia policy commands
Adds metadata compression config to dir writer
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Pass concatenate options with ConcatenateOptions struct
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move content compression handling to caller
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Move handling manifests to manifest pkg
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Correct const in server_test
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Remove unnecessary whitespace
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
* Disable metadata compression for < V2 format
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
---------
Signed-off-by: Prasad Ghangal <prasad.ganghal@veeam.com>
This may be a breaking change for users who rely on particular kopia metrics (unlikely):
- introduced blob-level metrics:
* `kopia_blob_download_full_blob_bytes_total`
* `kopia_blob_download_partial_blob_bytes_total`
* `kopia_blob_upload_bytes_total`
* `kopia_blob_storage_latency_ms` - per-method latency distribution
* `kopia_blob_errors_total` - per-method error counter
- updated cache metrics to indicate particular cache
* `kopia_cache_hit_bytes_total{cache="CACHE_TYPE"}`
* `kopia_cache_hit_total{cache="CACHE_TYPE"}`
* `kopia_cache_malformed_total{cache="CACHE_TYPE"}`
* `kopia_cache_miss_total{cache="CACHE_TYPE"}`
* `kopia_cache_miss_errors_total{cache="CACHE_TYPE"}`
* `kopia_cache_miss_bytes_total{cache="CACHE_TYPE"}`
* `kopia_cache_store_errors_total{cache="CACHE_TYPE"}`
where `CACHE_TYPE` is one of `contents`, `metadata` or `index-blobs`
- reorganized and unified content-level metrics:
* `kopia_content_write_bytes_total`
* `kopia_content_write_duration_nanos_total`
* `kopia_content_compression_attempted_bytes_total`
* `kopia_content_compression_attempted_duration_nanos_total`
* `kopia_content_compression_savings_bytes_total`
* `kopia_content_compressible_bytes_total`
* `kopia_content_non_compressible_bytes_total`
* `kopia_content_after_compression_bytes_total`
* `kopia_content_decompressed_bytes_total`
* `kopia_content_decompressed_duration_nanos_total`
* `kopia_content_encrypted_bytes_total`
* `kopia_content_encrypted_duration_nanos_total`
* `kopia_content_hashed_bytes_total`
* `kopia_content_hashed_duration_nanos_total`
* `kopia_content_deduplicated_bytes_total`
* `kopia_content_read_bytes_total`
* `kopia_content_read_duration_nanos_total`
* `kopia_content_decrypted_bytes_total`
* `kopia_content_decrypted_duration_nanos_total`
* `kopia_content_uploaded_bytes_total`
Also introduced `internal/metrics` framework which constructs Prometheus metrics in a uniform way and will allow us to include some of these metrics in telemetry report in future PRs.
Benchmarked from macOS client to a Linux server over Wifi connection:
(2-5ms latency)
linux 5.14.8 (1.1 GB) to a clean repository:
Before: 240s After: 27s (90% faster)
Fixes#2372
* refactor(repository): ensure we always parse content.ID and object.ID
This changes the types to be incompatible with string to prevent direct
conversion to and from string.
This has the additional benefit of reducing number of memory allocations
and bytes for all IDs.
content.ID went from 2 allocations to 1:
typical case 32 characters + 16 bytes per-string overhead
worst-case 65 characters + 16 bytes per-string overhead
now: 34 bytes
object.ID went from 2 allocations to 1:
typical case 32 characters + 16 bytes per-string overhead
worst-case 65 characters + 16 bytes per-string overhead
now: 36 bytes
* move index.{ID,IDRange} methods to separate files
* replaced index.IDFromHash with content.IDFromHash externally
* minor tweaks and additional tests
* Update repo/content/index/id_test.go
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* Update repo/content/index/id_test.go
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* pr feedback
* post-merge fixes
* pr feedback
* pr feedback
* fixed subtle regression in sortedContents()
This was actually not producing invalid results because of how base36
works, just not sorting as efficiently as it could.
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* fix(security): prevent cross-site request forgery in the UI website
This fixes a [cross-site request forgery (CSRF)](https://en.wikipedia.org/wiki/Cross-site_request_forgery)
vulnerability in self-hosted UI for Kopia server.
The vulnerability allows potential attacker to make unauthorized API
calls against a running Kopia server. It requires an attacker to trick
the user into visiting a malicious website while also logged into a
Kopia website.
The vulnerability only affected self-hosted Kopia servers with UI. The
following configurations were not vulnerable:
* Kopia Repository Server without UI
* KopiaUI (desktop app)
* command-line usage of `kopia`
All users are strongly recommended to upgrade at the earliest
convenience.
* pr feedback
* refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* chore: remove //nolint:gosec for os.ReadFile
At the time of this commit, the G304 rule of gosec does not include the
`os.ReadFile` function. We remove `//nolint:gosec` temporarily until
https://github.com/securego/gosec/pull/706 is merged.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* repo: big performance improvement for WriteContent with repo server
When re-uploading previously snapshotted directory we fetch directory
content `k<hash>` and very frequently end up writing the exact same
content. By caching last N content IDs we can avoid costly round-trip
to the server since we know that content ID was present in the session.
Also added small number of asynchronous writes, which also helps with
upload performance. Background writes are awaited before Flush().
Performance when snapshotting lots of small files (source code):
31.9 GB files:471205 dirs:75817, warm cache
Before: 260s
After: 55s (4-5x faster)
* fixed tests
* introduced passwordpersist package which has password persistence
strategies (keyring, file, none, multiple) with possibility of adding
more in the future.
* moved all password persistence logic out of 'repo'
* removed global variable repo.EnableKeyRing
* nit: replaced harcoded string constants with named constants
* acl: added management of ACL entries
* auth: implemented DefaultAuthorizer which uses ACLs if any entries are found in the system and falls back to LegacyAuthorizer if not
* cli: switch to DefaultAuthorizer when starting server
* cli: added ACL management
* server: refactored authenticator + added refresh
Authenticator is now an interface which also supports Refresh.
* authz: refactored authorizer to be an interface + added Refresh()
* server: refresh authentication and authorizer
* e2e tests for ACLs
* server: handling of SIGHUP to refresh authn/authz caches
* server: reorganized flags to specify auth options:
- removed '--allow-repository-users' - it's always on
- one of --without-password, --server-password or --random-password
can be specified to specify password for the UI user
- htpasswd-file - can be specified to provide password for UI or remote
users
* cli: moved 'kopia user' to 'kopia server user'
* server: allow all UI actions if no authenticator is set
* acl: removed priority until we have a better understood use case for it
* acl: added validation of allowed labels when adding ACL entries
* site: added docs for ACLs
This formalizes the concept of a 'UI user' which is a local
user that can call APIs the same way that UI does it.
The server will now allow access to:
- UI user (identified using `--server-username` with password specified
using `--server-password' or `--random-password`)
- remote users with usersnames/passwords specified in `--htpasswd-file`
- remote users defined in the repository using `kopia users add`
when `--allow-repository-users` is passed.
The UI user only has access to methods specifically designated as such
(normally APIs used by the UI + few special ones such as 'shutdown').
Remote users (identified via `user@host`) don't get access to UI APIs.
There are some APIs that can be accessed by any authenticated
caller (UI or remote):
- /api/v1/flush
- /api/v1/repo/status
- /api/v1/repo/sync
- /api/v1/repo/parameters
To make this easier to understand in code, refactored server handlers
to require specifing what kind of authorization is required
at registration time.
* cache: refactored reusable portion of cache into separate package
* repo: plumbed through caching for remote repository clients
* repo: plumb through cache in the unit tests
* cache: ensure we only allow absolute cache paths, fixed cache path resolution for remote repositories
* manifest: removed explicit refresh
Instead, content manager is exposing a revision counter that changes
on each mutation or index change. Manifest manager will be invalidated
whenever this is encountered.
* server: refactored initialization API
* server: added unit tests for repository server APIs (HTTP and REST)
* server: ensure we don't upload contents that already exist
This saves bandwidth, since the client can compute hash locally
and ask the server whether the object exists before starting the upload.
