* feat(cli): send error notifications and snapshot reports
Notifications will be sent to all configured notification profiles
according to their severity levels.
The following events will trigger notifications:
- Snapshot is created (CLI only, severity >= report)
- Server Maintenance error occurs (CLI, server and UI, severity >= error)
- Any other CLI error occurs (CLI only, severity >= error).
A flag `--no-error-notifications` can be used to disable error notifications.
* added template tests
* improved time formatting in templates
* plumb through notifytemplate.Options
* more testing for formatting options
* fixed default date format to RFC1123
* feat(repository): apply retention policies server-side
This allows append-only snapshots where the client can never delete
arbitrary manifests and policies are maintained on the server.
The client only needs permissions to create snapshots in a given, which
automatically gives them permission to invoke the server-side method
for their own snapshots only.
* Update cli/command_acl_add.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/api_manifest.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/api_manifest.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
* Update internal/server/grpc_session.go
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
---------
Co-authored-by: Guillaume <Gui13@users.noreply.github.com>
This change adds a new streaming response to the FindManifests API. The
server will deliver the response in chunks of N manifests where N is
requested by the client. This allows the client to process the response
in chunks and improves pipelining of responses.
For now client will hold the entire response in memory since this
is what FindManifests() API currently does. This will be fixed in a
follow up change.
Replaces #2713Fixes#2660
Instead of passing static content.FormattingOptions (and caching it)
we now introduce an interface to provide its values.
This will allow the values to dynamically change at runtime in the
future to support cases like live migration.
* refactor(repository): ensure we always parse content.ID and object.ID
This changes the types to be incompatible with string to prevent direct
conversion to and from string.
This has the additional benefit of reducing number of memory allocations
and bytes for all IDs.
content.ID went from 2 allocations to 1:
typical case 32 characters + 16 bytes per-string overhead
worst-case 65 characters + 16 bytes per-string overhead
now: 34 bytes
object.ID went from 2 allocations to 1:
typical case 32 characters + 16 bytes per-string overhead
worst-case 65 characters + 16 bytes per-string overhead
now: 36 bytes
* move index.{ID,IDRange} methods to separate files
* replaced index.IDFromHash with content.IDFromHash externally
* minor tweaks and additional tests
* Update repo/content/index/id_test.go
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* Update repo/content/index/id_test.go
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* pr feedback
* post-merge fixes
* pr feedback
* pr feedback
* fixed subtle regression in sortedContents()
This was actually not producing invalid results because of how base36
works, just not sorting as efficiently as it could.
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
This removes big shared lock held for for the duration of each request
and replaces it with trivially short lock to capture the current
state of the server/repository before passing it to handlers.
Handlers are now limited to only accessing a small subset of Server
functionality to be able to better reason about them.
* cli: added a flag to create repository with v2 index features
* content: plumb through compression.ID parameter to content.Manager.WriteContent()
* content: expose content.Manager.SupportsContentCompression
This allows object manager to decide whether to create compressed object
or let the content manager do it.
* object: if compression is requested and the repo supports it, pass compression ID to the content manager
* cli: show compression status in 'repository status'
* cli: output compression information in 'content list' and 'content stats'
* content: compression and decompression support
* content: unit tests for compression
* object: compression tests
* testing: added integration tests against v2 index
* testing: run all e2e tests with and without content-level compression
* htmlui: added UI for specifying index format on creation
* cli: additional tests for 'content ls' and 'content stats'
* applied pr suggestions
Removed Warning, Notify and Fatal:
* `Warning` => `Error` or `Info`
* `Notify` => `Info`
* `Fatal` was never used.
Note that --log-level=warning is still supported for backwards
compatibility, but it is the same as --log-level=error.
Co-authored-by: Julio López <julio+gh@kasten.io>
* nit: replaced harcoded string constants with named constants
* acl: added management of ACL entries
* auth: implemented DefaultAuthorizer which uses ACLs if any entries are found in the system and falls back to LegacyAuthorizer if not
* cli: switch to DefaultAuthorizer when starting server
* cli: added ACL management
* server: refactored authenticator + added refresh
Authenticator is now an interface which also supports Refresh.
* authz: refactored authorizer to be an interface + added Refresh()
* server: refresh authentication and authorizer
* e2e tests for ACLs
* server: handling of SIGHUP to refresh authn/authz caches
* server: reorganized flags to specify auth options:
- removed '--allow-repository-users' - it's always on
- one of --without-password, --server-password or --random-password
can be specified to specify password for the UI user
- htpasswd-file - can be specified to provide password for UI or remote
users
* cli: moved 'kopia user' to 'kopia server user'
* server: allow all UI actions if no authenticator is set
* acl: removed priority until we have a better understood use case for it
* acl: added validation of allowed labels when adding ACL entries
* site: added docs for ACLs
* repo: refactored connect code set up cache for server repositories
- improved logic to close the cache on last connection
- preemptively add all contents with a prefix to the cache
- refactored how config is loaded and saved
Now cache dir will be stored as relative and resolved to absolute as
part of loading and saving the file, in all other places cache dir
is expected to be absolute.
* server: removed cache directory from the API and UI
This won't be easily available and does not seem useful to expose
anyway.
* cli: enabled cache commands for server repositories
* cli: added KOPIA_CACHE_DIRECTORY environment variable
This is used on two occassions - when setting up connection (it gets
persisted in the config) and later when opening (to override the
cache location from config). It makes setting up docker container with
mounted cache somewhat easier with one environment variable.
* cli: show cache size for the server cache
* tls: present more helpful error message that includes SHA256 fingerprint of the TLS server on mismatch
* server: return the name of user who attempted to login when authentication fails
* manifest: removed explicit refresh
Instead, content manager is exposing a revision counter that changes
on each mutation or index change. Manifest manager will be invalidated
whenever this is encountered.
* server: refactored initialization API
* server: added unit tests for repository server APIs (HTTP and REST)
* server: ensure we don't upload contents that already exist
This saves bandwidth, since the client can compute hash locally
and ask the server whether the object exists before starting the upload.
* grpcapi: added GPRC API for the repository server
* repo: added transparent retries to GRPC repository client
Normally GRPC reconnects automatically, which can survive server
restarts (minus transient errors).
In our case we're establishing a stream which will be broken and
needs to be restarted after io.EOF is detected.
It safe to do transparent retries for read-only (repo.Repository),
but not safe for write sessions (repo.RepositoryWriter), because the
session may re-connect to different server that won't have the buffered
content write available in memory.
