instead moved to run as part of maintenance ('kopia maintenance run')
added 'kopia maintenance run --force' flag which runs maintenance even
if not owned
Connect the snapshotter, the storer, and the comparer. Invoke
the snapshotter to take/restore/delete snapshots on the repo,
the comparer to gather metadata before the snapshot and
after the restore, and the storer to save metadata for later
lookup when verifying restores.
* content: added support for cache of own writes
Thi keeps track of which blobs (n and m) have been written by the
local repository client, so that even if the storage listing
is eventually consistent (as in S3), we get somewhat sane behavior.
Note that this is still assumming read-after-create semantics, which
S3 also guarantees, otherwise it's very hard to do anything useful.
* compaction: support for compaction logs
Instead of compaction immediately deleting source index blobs, we now
write log entries (with `m` prefix) which are merged on reads
and applied only if the blob list includes all inputs and outputs, in
which case the inputs are discarded since they are known to have been
superseded by the outputs.
This addresses eventual consistency issues in stores such as S3,
which don't guarantee list-after-put or list-after-delete. With such
stores the repository is ultimately eventually consistent and there's
not much that can be done about it, unless we use second strongly
consistent storage (such as GCS) for the index only.
* content: updated list cache to cache both `n` and `m`
* repo: fixed cache clear on windows
Clearing cache requires closing repository first, as Windows is holding
the files locked.
This requires ability to close the repository twice.
* content: refactored index blob management into indexBlobManager
* testing: fixed blobtesting.Map storage to allow overwrites
* blob: added debug output String() to blob.Metadata
* testing: added indexBlobManager stress test
This works by using N parallel "actors", each repeatedly performing
operations on indexBlobManagers all sharing single eventually consistent
storage.
Each actor runs in a loop and randomly selects between:
- *reading* all contents in indexes and verifying that it includes
all contents written by the actor so far and that contents are
correctly marked as deleted
- *creating* new contents
- *deleting* one of previously-created contents (by the same actor)
- *compacting* all index files into one
The test runs on accelerated time (every read of time moves it by 0.1
seconds) and simulates several hours of running.
In case of a failure, the log should provide enough debugging
information to trace the exact sequence of events leading up to the
failure - each log line is prefixed with actorID and all storage
access is logged.
* makefile: increase test timeout
* content: fixed index blob manager race
The race is where if we delete compaction log too early, it may lead to
previously deleted contents becoming temporarily live again to an
outside observer.
Added test case that reproduces the issue, verified that it fails
without the fix and passed with one.
* testing: improvements to TestIndexBlobManagerStress test
- better logging to be able to trace the root cause in case of a failure
- prevented concurrent compaction which is unsafe:
The sequence:
1. A creates contentA1 in INDEX-1
2. B creates contentB1 in INDEX-2
3. A deletes contentA1 in INDEX-3
4. B does compaction, but is not seeing INDEX-3 (due to EC or simply
because B started read before #3 completed), so it writes
INDEX-4==merge(INDEX-1,INDEX-2)
* INDEX-4 has contentA1 as active
5. A does compaction but it's not seeing INDEX-4 yet (due to EC
or because read started before #4), so it drops contentA1, writes
INDEX-5=merge(INDEX-1,INDEX-2,INDEX-3)
* INDEX-5 does not have contentA1
7. C sees INDEX-5 and INDEX-5 and merge(INDEX-4,INDEX-5)
contains contentA1 which is wrong, because A has been deleted
(and there's no record of it anywhere in the system)
* content: when building pack index ensure index bytes are different each time by adding 32 random bytes
Change log:
https://github.com/aws/aws-sdk-go/releases
Notably:
SDK Bugs
service/s3/s3crypto: Add missing return in encryption client (#3258)
Fixes a missing return in the encryption client that was causing a nil
dereference panic.
New usage:
```
kopia snapshot delete manifestID... [--delete]
kopia snapshot delete rootObjectID... [--delete]
```
Fixes#435
cli: added --unsafe-ignore-source as alias for `--delete`
This is a hidden flag for backwards compatibility. It will be removed.
This is done by introducing N unsynchronized caches, which simulate
what frontend of a cloud storage system might do, that causes eventual
consistency behavior.
Support for remote content repository where all contents and
manifests are fetched over HTTP(S) instead of locally
manipulating blob storage
* server: implement content and manifest access APIs
* apiclient: moved Kopia API client to separate package
* content: exposed content.ValidatePrefix()
* manifest: added JSON serialization attributes to EntryMetadata
* repo: changed repo.Open() to return Repository instead of *DirectRepository
* repo: added apiServerRepository
* cli: added 'kopia repository connect server'
This sets up repository connection via the API server instead of
directly-manipulated storage.
* server: add support for specifying a list of usernames/password via --htpasswd-file
* tests: added API server repository E2E test
* server: only return manifests (policies and snapshots) belonging to authenticated user
* maintenance: encrypt maintenance schedule block
* maintenance: created snapshotmaintenance package that wraps maintenance and performs snapshot GC + regular maintenance in one shot, used in CLI and server
* PR feedback.
* mechanical rename of package snapshot/gc => snapshot/snapshotgc
* maintenance: record maintenance run times and statuses
Also stopped dropping deleted contents during quick maintenance, since
doing this safely requires coordinating with snapshot GC which is
part of full maintenance.
* cli: 'maintenance info' outputs maintenance run history
* maintenance: only drop index entries when it's safe to do so
This is based on the timestamp of previous successful GC that's old
enough to resolve all race conditions between snapshot creation and GC.
* maintenance: added internal flush to RewriteContents() to better measure its time
Unlike regular cache, which caches segments of blobs on a per-content
basis, metadata cache will fetch and store the entire metadata blob (q)
when any of the contents in it is accessed.
Given that there are relatively few metadata blobs compared to data (p)
blobs, this will reduce the traffic to the underlying store and improve
performance of Snapshot GC which only relies on metadata contents.
Maintenance: support for automatic GC
Moved maintenance algorithms from 'cli' to 'repo/maintenance' package
Added support for CLI commands:
kopia gc - performs quick maintenance
kopia gc --full- perform full maintenance
Full maintenance performs snapshot gc, but it's not safe to do this automatically possibly in parallel to snapshots being taken. This will be addressed ~0.7 timeframe.
When prefix is not specified on ObjectWriter, we force
'x' content prefix on intermediate contents, so object IDs
will look like:
Ix{hash}
This ensures the index contents will be stored in `q` blobs,
making `snapshot gc` easier.
With large cache sizes it takes several seconds to perform the initial
sweep. This change only checks that the cache is functional without
full sweep.
Also cleaned up contentCache by introducing interface and having
two implementation - one which does caching, and one that does not.
* snapshot: support for periodic checkpointing of snapshots in progress
For each snapshot that takes longer than 45 minutes, we trigger
internal cancellation, save the manifest and restart the snapshot
at which point all files will be cached.
This helps ensure the property that no file or directory objects
in the repository remain unreachable from a snapshot root for more than
one hour, which is important from GC perspective.
* nit: unified spelling 'cancelled' => 'canceled'
They now uniformly support 3 flags:
--prefix=P selects contents with the specified prefix
--prefixed selects contents with ANY prefix
--non-prefixed selects non-prefixed contents
Also changed content manager iteration API to support ranges.
cli: add --prefix to 'blob gc' and 'blob stats'
* server: when serving HTML UI, prefix the title with string from KOPIA_UI_TITLE_PREFIX envar
* kopia-ui: support for multiple repositories + portability
This is a major rewrite of the app/ codebase which changes
how configuration for repositories is maintained and how it flows
through the component hierarchy.
Portable mode is enabled by creating 'repositories' subdirectory before
launching the app.
on macOS:
<parent>/KopiaUI.app
<parent>/repositories/
On Windows, option #1 - nested directory
<parent>\KopiaUI.exe
<parent>\repositories\
On Windows, option #2 - parallel directory
<parent>\some-dir\KopiaUI.exe
<parent>\repositories\
In portable mode, repositories will have 'cache' and 'logs' nested
in it.
* Makefile: support Windows Appveyor environment
* appveyor: added .appveyor.yml
* travis: disable windows environment
* Makefile: do not run vet-time-inject on Windows
* Makefile: removed the use of backtick invocation of subshell which does not work on windows
* Makefile: make integration-tests work on Windows outside of Travis
Instead of taking timestamp + TRAVIS_BUILD_NUMBER at build time,
we take the time of the commit, to be deterministic. This requires
a little bit of makefile magic to strip leading zeroes to make the
version number that's semver compliant.
This will allow coordinating the build between different CI tools,
so that Windows can be built on AppVeyor and macOS/Linux on Travis CI.
