* cli: added --safety=full|none flag to maintenance commands
This allows selection between safe, high-latency maintenance parameters
which allow concurrent access (`full`) or low-latency which may be
unsafe in certain situations when concurrent Kopia processes are
running.
This is a breaking change for advanced CLI commands, where it removes
timing parameters and replaces them with single `--safety` option.
* 'blob gc'
* 'content rewrite'
* 'snapshot gc'
* pr renames
* maintenance: fixed computation of safe time for --safety=none
* maintenance: improved logging for blob gc
* maintenance: do not rewrite truly short, densely packed packs
* mechanical: pass eventual consistency settle time via CompactOptions
* maintenance: add option to disable eventual consistency time buffers with --safety=none
* maintenance: trigger flush at the end of snapshot gc
* maintenance: reload indexes after compaction that drops deleted entries, this allows single-pass maintenance with --safety=none to delete all unused blobs
* testing: allow debugging of integration tests inside VSCode
* testing: added end-to-end maintenance test that verifies that full maintenance with --safety=none removes all data
* content: fixed data race in IterateUnreferencedBlobs
* upload: fixed data race between uploader and estimator
* testing: fixed data race in repo/blob/logging test
* makefile: run tests on CI/linux/amd64 with -race
* robustness: fixed test race
* content: fixed data race getContentDataUnlocked that triggers TestParallelWrites - looks scary but in practice very hard to trigger in real life and does not cause data corruption
* testing: reduce test complexity under race detector
* server: fixed minor race in refreshStatus()
* testing: reduced depth of sharedTestDataDir2
* ci: run race detector in separate job
* ci: run unit test race detector in parallel to integration tests
* nit: replaced harcoded string constants with named constants
* acl: added management of ACL entries
* auth: implemented DefaultAuthorizer which uses ACLs if any entries are found in the system and falls back to LegacyAuthorizer if not
* cli: switch to DefaultAuthorizer when starting server
* cli: added ACL management
* server: refactored authenticator + added refresh
Authenticator is now an interface which also supports Refresh.
* authz: refactored authorizer to be an interface + added Refresh()
* server: refresh authentication and authorizer
* e2e tests for ACLs
* server: handling of SIGHUP to refresh authn/authz caches
* server: reorganized flags to specify auth options:
- removed '--allow-repository-users' - it's always on
- one of --without-password, --server-password or --random-password
can be specified to specify password for the UI user
- htpasswd-file - can be specified to provide password for UI or remote
users
* cli: moved 'kopia user' to 'kopia server user'
* server: allow all UI actions if no authenticator is set
* acl: removed priority until we have a better understood use case for it
* acl: added validation of allowed labels when adding ACL entries
* site: added docs for ACLs
Protect filesystem subtrees from concurrent manipulation during critical sections
if engine actions are called asynchronously. This change provides coordination
between the `Snapshotter` and the `FileWriter`. For example, the `FileWriter`
should be blocked from perturbing the same directory tree if a
Gather-Snapshot is taking place along that tree simultaneously.
This will ensure the fingerprint data accumulated during the `Gather` phase
will correspond unambiguously to the data included in the snapshot.
Extend build flags to kopia snapshotter
This package now imports fswalker which can only be built for
darwin,amd64 or linux,amd64
This formalizes the concept of a 'UI user' which is a local
user that can call APIs the same way that UI does it.
The server will now allow access to:
- UI user (identified using `--server-username` with password specified
using `--server-password' or `--random-password`)
- remote users with usersnames/passwords specified in `--htpasswd-file`
- remote users defined in the repository using `kopia users add`
when `--allow-repository-users` is passed.
The UI user only has access to methods specifically designated as such
(normally APIs used by the UI + few special ones such as 'shutdown').
Remote users (identified via `user@host`) don't get access to UI APIs.
There are some APIs that can be accessed by any authenticated
caller (UI or remote):
- /api/v1/flush
- /api/v1/repo/status
- /api/v1/repo/sync
- /api/v1/repo/parameters
To make this easier to understand in code, refactored server handlers
to require specifing what kind of authorization is required
at registration time.
* Add StreamingFile interface
* unit test for virtualfs
* CLI: Snapshot create support for stdin sources
* Uploader support for fs.StreamingFile
* End to end test for stdin source snapshot
* upload test to improve coverage
Add metadata R/W locking for asynchronous accesses to robustness engine metadata.
Remove the index from the Store interface and maintain it only in Checker, where it's used.
* Path lock utility for coordination between robustness engine actions
Add a utility to ensure path-based synchronization between goroutines. If a path is locked, a subsequent Lock will block if the requested path is the same, or a child/parent (recursive), of the locked path.
This assists with coordination between asynchronous robustness engine actions that may rely on the underlying data directory remaining unchanged. For example:
- between gathering a filesystem fingerprint and taking a snapshot.
- when one WriteFilesAtDepth command has traversed into a directory that another goroutine has picked for deletion.
* Fix linter
* cache: refactored reusable portion of cache into separate package
* repo: plumbed through caching for remote repository clients
* repo: plumb through cache in the unit tests
* cache: ensure we only allow absolute cache paths, fixed cache path resolution for remote repositories
* ci: refactored CI/CD logic & Makefile
- removed all travis CI emulation environment variables and replaced with:
CI_TAG=<empty>|tag
IS_PULL_REQUEST=false|true
- refactored all OS and architecture-specific decisions to use around standard GOOS/GOARCH values instead of uname/OS
- re-added self-hosted runner for ARMHF (3 replicas)
- added brand new self-hosted runner for ARM64 (3 replicas)
- disabled attempts to publish and sign on forks
- improved integration test log output to better see timings and sub-tests
- print longest tests (unit tests and integration) after each run
- verified that all configurations build successfully on a clone (jkowalski/kopia)
- run make setup in parallel
* testing: fixed tests on ARM and ARM64
- fixed ARM-specific alignment issue
- cleaned up test logging
- fixed huge params warning threshold because it was tripping on ARM.
- reduced test complexity to make them fit in 15 minutes
Fixes#690
This is a breaking change for folks who are expecting snapshots to fail
quickly without writing a snapshot manifest in case of an error.
Before this change, any source read failure would cause the entire
snapshot to fail (and not write a snapshot manifest as a result),
unless `ignoreFileErrors` or `ignoreDirectoryErrors` was set.
The new behavior is to continue snapshotting remaining files and
directories (this can be disabled by passing `--fail-fast` flag or
setting `KOPIA_SNAPSHOT_FAIL_FAST=1` environment variable) and defer
returning an error until the very end.
After snapshotting we will always attempt to write the snapshot manifest
(except when the root of the snapshot itself cannot be opened). In case
of a fail-fast error, the manifest will be marked as 'partial' and
the directory tree will contain only partial set of files.
In case of any errors, the manifest (and each directory object) will
list the number if failures and no more than 10 examples of failed
files/directories along with their respective errors.
Once the snapshot is complete we will return non-zero exit code to the
operating system if there were any fatal errors during snapshotting.
With this change we are repurposing `ignoreFileErrors` and
`ignoreDirectoryErrors` to designate some errors as non-fatal.
Non-fatal errors are reported as warnings in the logs and will not
cause a non-zero exit code to be returned.
* blob: refactored upload reporting
Instead of plumbing this through blob storage context, we are passing
and explicit callback that reports uploads as they happen.
* htmlui: improved counter presentation
* nit: added missing UI route which fixes Reload behavior on the Tasks page
* Add manual field to SchedulingPolicy
* CLI: Set and show for policy with manual field
* CLI: Edit policy support for manual field
* Check manual when creating snapshot for all source
* End to end test for snapshot create all
* Add UI option for setting Manual field
* user: added user profile (username&password for authentication) and CRUD methods
* manifest: helpers for disambiguating manifest entries
* authn: added repository-based user authenticator
* cli: added commands to manipulate user accounts and passwords
* cli: added --allow-repository-users option to 'server start'
* Update cli/command_user_info.go
Co-authored-by: Julio López <julio+gh@kasten.io>
* Always return false when the user is not found.
* cli: added '--insecure' flag to 'kopia server start'
This is a breaking change for development scenarios to prevent people
from unknowingly launching insecure servers.
Attempt to start a server without either TLS or password protection
results in an error now (unless --insecure is also passed).
KopiaUI already launches server with TLS and random password, so it
does not require it.
* grpcapi: added GPRC API for the repository server
* repo: added transparent retries to GRPC repository client
Normally GRPC reconnects automatically, which can survive server
restarts (minus transient errors).
In our case we're establishing a stream which will be broken and
needs to be restarted after io.EOF is detected.
It safe to do transparent retries for read-only (repo.Repository),
but not safe for write sessions (repo.RepositoryWriter), because the
session may re-connect to different server that won't have the buffered
content write available in memory.
Fixes#689
Add symlink overwrite behavior to fix "file exists" error when restoring a symlink that already exists
Before creating the restored symlink, check `os.Lstat`:
- If it returns an error indicating the file does not exist, proceed to symlink creation
- If it returns any other error, propagate the error up to the caller
- If the fileInfo indicates the entry is a symlink AND `--no-overwrite-symlinks` was set in the restore command, propagate an error to the caller
- If `--no-overwrite-symlinks` was NOT set, remove the existing symlink before proceeding to symlink creation
- Else the file exists but it is not of type symlink. Halt the operation and propagate an error indicating we tried to restore a symlink over a file system entry that already existed but was not a symlink.
Added case to `TestSnapshotRestore` that fails before this fix and succeeds after. The case is simply to restore the same snapshot into the same directory twice in a row, where the second restore will be on top of the first one.
Added test case to ensure `--no-overwrite-symlinks` throws an error as expected if restoring into a directory where a symlink already exists at the path symlink creation is attempted.
Added test case to ensure that the restore operation fails if a symlink is needed to be restored to the same path as an existing non-symlink filesystem entry with the same name.
* Skip overwrite test on Windows
If test is run as non-admin it is likely to fail on Windows
with insufficient permissions to overwrite the previously
restored data.
* Add brief summary of overwrite behavior to help
Add a brief summary to the restore command help text
of expected behavior when restoring into a target location
that has existing data present.
- `repo.Repository` is now read-only and only has methods that can be supported over kopia server
- `repo.RepositoryWriter` has read-write methods that can be supported over kopia server
- `repo.DirectRepository` is read-only and contains all methods of `repo.Repository` plus some low-level methods for data inspection
- `repo.DirectRepositoryWriter` contains write methods for `repo.DirectRepository`
- `repo.Reader` removed and merged with `repo.Repository`
- `repo.Writer` became `repo.RepositoryWriter`
- `*repo.DirectRepository` struct became `repo.DirectRepository`
interface
Getting `{Direct}RepositoryWriter` requires using `NewWriter()` or `NewDirectWriter()` on a read-only repository and multiple simultaneous writers are supported at the same time, each writing to their own indexes and pack blobs.
`repo.Open` returns `repo.Repository` (which is also `repo.RepositoryWriter`).
* content: removed implicit flush on content manager close
* repo: added tests for WriteSession() and implicit flush behavior
* invalidate manifest manager after write session
* cli: disable maintenance in 'kopia server start'
Server will close the repository before completing.
* repo: unconditionally close RepositoryWriter in {Direct,}WriteSession
* repo: added panic in case somebody tries to create RepositoryWriter after closing repository
- used atomic to manage SharedManager.closed
* removed stale example
* linter: fixed spurious failures
Co-authored-by: Julio López <julio+gh@kasten.io>
- renamed content.Manager to content.WriteManager
- merged lockFreeManager and CommittedReadManager into SharedManager
- also reassigned some methods to SharedManager (no code move)
* content: fixed time-based auto-flush behavior to behave like Flush()
Previously it would sometimes be possible for a content whose write
started before time-based flush to finish writing afterwards (and it
would be included in the new index).
Refactored the code so that time-based flush happens before WriteContent
write and behaves exactly the same was as real Flush() so all writes
started before it will be awaited during the flush.
Also previous regression test was incorrect since it was mocking the
wrong blob method.
* content: refactored index blob manager crypto to separate file
This will be reused for encrypting session info.
* content: added support for session markers
Session marker (`s` blob) is written BEFORE the first data blob
(`p` or `q`) that belongs to new index segment (`n` is written).
Session marker is removed AFTER the index blob (`n`) has been written.
All pack and index blobs belonging to a session will have the session
ID as its suffix, so that if a reader can see `s<sessionID>` blob, they
will ignore any `p` and `q` blobs with the same suffix.
* maintenance: ignore blobs belonging to active sessions when running blob garbage collection
* cli: added 'sessions list' for listing active sessions
* content: added retrying writing previously failed blobs before writing new one
* Improved .kopiaignore pattern matching
.kopiaignore pattern matching now (hopefully) conforms to the .gitignore specification (https://git-scm.com/docs/gitignore)
Replaced old package "ignore" with a newly written "wcmatch" that manages the globbing. This should support all the patterns that .gitignore supports.
Some changes in ignorefs that dealt with how the patterns were matched.
This fixes#571
* Fixed invalid matching of non-rooted patterns that contained a slash.
If a pattern contains a slash in the middle of the pattern this should only match relative to the .gitignore file, i.e. the same as if it started with a '/' according to the .gitignore spec.
Example:
foo/bar should match "/foo/bar", but not "/other/foo/bar".
whereas
"bar" matches both "/bar" and "/foo/bar"
* Uncommented previously failing tests.
* Fixed problem with matching "nested" .kopiaignore files.
Ignore-patterns must be applied from the root .kopiaignore down the hierarchy, so that an ignore file in a subdirectory can negate a pattern from a parent directory.
* Uncommented tests that should now work.
* Created end-to-end tests verifying .kopiaignore behavior.
This is related to #571 and #773, but provided as a separate PR to include tests that did not work before PR #773.
* Commented failing tests.
These tests will be re-enabled when #773 is done.
* Added additional commented tests of .kopiaignore
These will be uncommented in #773.
* content: introduced ContentReadManager
This only introduces new type and reassigns methods
* content: moved all CommittedReadManager methods to one file
* content: more code movement
* content: refactored read manager setup
* content: refactored test hook that allowed cleaner passing of custom own writes cache
* testing: prevented spurious test flakes caused by kopia subprocesses messing with stderr
This was not causing actual failures, but misreporting error messages.
* testing: ensure random names are always unique by adding a counter
This also fixed a test bug where the test was incorrectly passing
password via environment variable and it was (incorrectly) expected
to be ignored.
Password is determined in the following order:
- flag/environment variable (highest priority)
- persistent storage
- asking user (lowest priority)
* linter: upgraded to 1.33, disabled some linters
* lint: fixed 'errorlint' errors
This ensures that all error comparisons use errors.Is() or errors.As().
We will be wrapping more errors going forward so it's important that
error checks are not strict everywhere.
Verified that there are no exceptions for errorlint linter which
guarantees that.
* lint: fixed or suppressed wrapcheck errors
* lint: nolintlint and misc cleanups
Co-authored-by: Julio López <julio+gh@kasten.io>
* When running against direct repository, it will verify that all
backing blobs exist based on results of listing.
* Deprecated annoying --all-sources flag which is now default if no
sources are provided.
This can be specified at `repo create` or `repo connect` to enable
actions. By default actions are disabled to avoid security risks
associated with executing code.
Alternatively during `snapshot create` one can specify
`--force-enable-actions` or `--force-disable-actions`
* policy: add actions
* fs: added LocalFilesystemPath() which can optionally return local filesystem
path (if entry is local)
* cli: added support for setting policy actions
* upload: support for executing actions before/after folder (non-inheritable)
and before/after snapshots (inheritable)
* testing: end-to-end test for actions
* additional tests for actions with embedded scripts
Remove unnecessary intermediate variables.
Send SIGTERM instead of SIGKILL to terminate child kopia server process.
Set Pdeathsig on Linux for child kopia server process.
Trivial: reduce scope of hostFioDataPathStr variable.
Trivial: rename local variable.
Trivial: Use log.Fatalln instead of log + exit(1).
Improve error message in robustness test to tell apart failure cause.
