* manifest: removed explicit refresh
Instead, content manager is exposing a revision counter that changes
on each mutation or index change. Manifest manager will be invalidated
whenever this is encountered.
* server: refactored initialization API
* server: added unit tests for repository server APIs (HTTP and REST)
* server: ensure we don't upload contents that already exist
This saves bandwidth, since the client can compute hash locally
and ask the server whether the object exists before starting the upload.
* blob: refactored upload reporting
Instead of plumbing this through blob storage context, we are passing
and explicit callback that reports uploads as they happen.
* htmlui: improved counter presentation
* nit: added missing UI route which fixes Reload behavior on the Tasks page
* fuse: switched to github.com/hanwen/go-fuse/v2 implementation
This supports latest MacFUSE (tested on 4.0.5 on Intel Mac).
* fuse: disabled direct IO which fixed macOS behavior
See https://github.com/hanwen/go-fuse/issues/390
* Add manual field to SchedulingPolicy
* CLI: Set and show for policy with manual field
* CLI: Edit policy support for manual field
* Check manual when creating snapshot for all source
* End to end test for snapshot create all
* Add UI option for setting Manual field
* uitask: added package for managing and introspection into tasks running inside the process
* server: added API for getting details of tasks running inside the server
* htmlui: added new tab called 'Tasks'
This allows access to progres, logs and cancelation for long-running
tasks (Snapshots, Maintenance, and in the future Restore, Estimate,
Verify)
* snapshot: improve counters returned from the upload
* user: added user profile (username&password for authentication) and CRUD methods
* manifest: helpers for disambiguating manifest entries
* authn: added repository-based user authenticator
* cli: added commands to manipulate user accounts and passwords
* cli: added --allow-repository-users option to 'server start'
* Update cli/command_user_info.go
Co-authored-by: Julio López <julio+gh@kasten.io>
* Always return false when the user is not found.
* cli: added '--insecure' flag to 'kopia server start'
This is a breaking change for development scenarios to prevent people
from unknowingly launching insecure servers.
Attempt to start a server without either TLS or password protection
results in an error now (unless --insecure is also passed).
KopiaUI already launches server with TLS and random password, so it
does not require it.
* grpcapi: added GPRC API for the repository server
* repo: added transparent retries to GRPC repository client
Normally GRPC reconnects automatically, which can survive server
restarts (minus transient errors).
In our case we're establishing a stream which will be broken and
needs to be restarted after io.EOF is detected.
It safe to do transparent retries for read-only (repo.Repository),
but not safe for write sessions (repo.RepositoryWriter), because the
session may re-connect to different server that won't have the buffered
content write available in memory.
* lint: upgraded to 1.36 which fixes flakes
* ci: removed separate lint action - normal lint is fast enough and less flaky
* ci: also deleted linter_version_test.go
* blob: rewrote blob storage providers retry logic
Previously we were trying to guess which errors are retriable
(often poorly, since things like DNS or HTTP errors were not retried)
This change makes it so the provider does not need built-in retry logic
and instead retrying wrapper is added which retries on all errors other
than
blob.ErrNotFound, blob.ErrInvalidRange and blob.ErrSetTimeUnsupported.
This required thorough testing of negative conditions that would trigger
blob.ErrInvalidLength (we already had those). Verified that tests pass
against Azure, B2, S3, GCS, Rclone, WebDAV, SFTP.
* lint: fixed spurious failure
* sftp: return proper error on invalid seek
* pr feedback
* Option to print out the commands for using crypto, splitter and compression
Co-authored-by: Janne Johansson <janne.johansson@safespring.com>
Co-authored-by: Jarek Kowalski <jaak@jkowalski.net>
Fixes#689
Add symlink overwrite behavior to fix "file exists" error when restoring a symlink that already exists
Before creating the restored symlink, check `os.Lstat`:
- If it returns an error indicating the file does not exist, proceed to symlink creation
- If it returns any other error, propagate the error up to the caller
- If the fileInfo indicates the entry is a symlink AND `--no-overwrite-symlinks` was set in the restore command, propagate an error to the caller
- If `--no-overwrite-symlinks` was NOT set, remove the existing symlink before proceeding to symlink creation
- Else the file exists but it is not of type symlink. Halt the operation and propagate an error indicating we tried to restore a symlink over a file system entry that already existed but was not a symlink.
Added case to `TestSnapshotRestore` that fails before this fix and succeeds after. The case is simply to restore the same snapshot into the same directory twice in a row, where the second restore will be on top of the first one.
Added test case to ensure `--no-overwrite-symlinks` throws an error as expected if restoring into a directory where a symlink already exists at the path symlink creation is attempted.
Added test case to ensure that the restore operation fails if a symlink is needed to be restored to the same path as an existing non-symlink filesystem entry with the same name.
* Skip overwrite test on Windows
If test is run as non-admin it is likely to fail on Windows
with insufficient permissions to overwrite the previously
restored data.
* Add brief summary of overwrite behavior to help
Add a brief summary to the restore command help text
of expected behavior when restoring into a target location
that has existing data present.
* server: reworked authn/authz
Previously authentication was done as an wrapper handler and
authorization was inlined. This change moves authn/authz handlers
inside the server and implements separate authorization module that's
individually tested.
Also fixed an issue where server users were not able to see global
or host-level policies.
* PR feedback
- `repo.Repository` is now read-only and only has methods that can be supported over kopia server
- `repo.RepositoryWriter` has read-write methods that can be supported over kopia server
- `repo.DirectRepository` is read-only and contains all methods of `repo.Repository` plus some low-level methods for data inspection
- `repo.DirectRepositoryWriter` contains write methods for `repo.DirectRepository`
- `repo.Reader` removed and merged with `repo.Repository`
- `repo.Writer` became `repo.RepositoryWriter`
- `*repo.DirectRepository` struct became `repo.DirectRepository`
interface
Getting `{Direct}RepositoryWriter` requires using `NewWriter()` or `NewDirectWriter()` on a read-only repository and multiple simultaneous writers are supported at the same time, each writing to their own indexes and pack blobs.
`repo.Open` returns `repo.Repository` (which is also `repo.RepositoryWriter`).
* content: removed implicit flush on content manager close
* repo: added tests for WriteSession() and implicit flush behavior
* invalidate manifest manager after write session
* cli: disable maintenance in 'kopia server start'
Server will close the repository before completing.
* repo: unconditionally close RepositoryWriter in {Direct,}WriteSession
* repo: added panic in case somebody tries to create RepositoryWriter after closing repository
- used atomic to manage SharedManager.closed
* removed stale example
* linter: fixed spurious failures
Co-authored-by: Julio López <julio+gh@kasten.io>
- renamed content.Manager to content.WriteManager
- merged lockFreeManager and CommittedReadManager into SharedManager
- also reassigned some methods to SharedManager (no code move)
* content: fixed time-based auto-flush behavior to behave like Flush()
Previously it would sometimes be possible for a content whose write
started before time-based flush to finish writing afterwards (and it
would be included in the new index).
Refactored the code so that time-based flush happens before WriteContent
write and behaves exactly the same was as real Flush() so all writes
started before it will be awaited during the flush.
Also previous regression test was incorrect since it was mocking the
wrong blob method.
* content: refactored index blob manager crypto to separate file
This will be reused for encrypting session info.
* content: added support for session markers
Session marker (`s` blob) is written BEFORE the first data blob
(`p` or `q`) that belongs to new index segment (`n` is written).
Session marker is removed AFTER the index blob (`n`) has been written.
All pack and index blobs belonging to a session will have the session
ID as its suffix, so that if a reader can see `s<sessionID>` blob, they
will ignore any `p` and `q` blobs with the same suffix.
* maintenance: ignore blobs belonging to active sessions when running blob garbage collection
* cli: added 'sessions list' for listing active sessions
* content: added retrying writing previously failed blobs before writing new one
* object: refactored Open() and VerifyObject() to be stateless
(no code movement yet to facilitate review)
* mechanical: moved function more appropriate files
* object: remove object manager tracing which was unused
* manifest: refactored management of committed manifests into separate component
This is very similar to content.CommittedReadManager and allows
multiple writes to coexist and share the underlying committed manifest
manager.
Changed compaction to only include previously-committed manifests and
not any pending ones.
Added locking assertions for extra safety.
