* testing: added test that simulates running auto maintenance over several weeks of time.
This ensures all maintenance tasks run with reasonable frequency.
* testing: modify time interval to 30 minutes which uncovered a bug where we're starving full blob deletion
* maintenance: fixed full rewrite logic to allow full rewrite after a quick rewrite
* maintenance: when performing quick maintenance after full maintenance we sometimes need to do full blob deletion to ensure liveness
* maintenance: refactored to improve readability
Removed Warning, Notify and Fatal:
* `Warning` => `Error` or `Info`
* `Notify` => `Info`
* `Fatal` was never used.
Note that --log-level=warning is still supported for backwards
compatibility, but it is the same as --log-level=error.
Co-authored-by: Julio López <julio+gh@kasten.io>
* maintenance: refactored code by introducing TaskType constants
* maintenance: added additional safety rules for DeleteUnreferencedBlobs and RewriteContents
* DeleteUnreferencedBlobs will run if now() > time(last-content-rewrite) + 1h
(this ensures other kopia clients will have time refresh their indexes
to the rewritten blobs before we delete them)
* RewriteContents runs only if time(last-blob-deletion) > time(last-content-rewrite)
(this makes sure we don’t starve blob deletions if somebody runs
maintenance too frequently)
With those two rules in place each maintenance cycle will effectively
do either RewriteContents or DeleteUnreferencedBlobs but not both
(except when --safety=none).
* Handle forbidden responses for missing directories
Some WebDAV implementations respond with 403 Forbidden when a missing
directory is encountered.
To handle this situation, any time an error is encountered while writing
a blob, we first try to create requisite directories and if that fails,
we return the error.
* cli: added --safety=full|none flag to maintenance commands
This allows selection between safe, high-latency maintenance parameters
which allow concurrent access (`full`) or low-latency which may be
unsafe in certain situations when concurrent Kopia processes are
running.
This is a breaking change for advanced CLI commands, where it removes
timing parameters and replaces them with single `--safety` option.
* 'blob gc'
* 'content rewrite'
* 'snapshot gc'
* pr renames
* maintenance: fixed computation of safe time for --safety=none
* maintenance: improved logging for blob gc
* maintenance: do not rewrite truly short, densely packed packs
* mechanical: pass eventual consistency settle time via CompactOptions
* maintenance: add option to disable eventual consistency time buffers with --safety=none
* maintenance: trigger flush at the end of snapshot gc
* maintenance: reload indexes after compaction that drops deleted entries, this allows single-pass maintenance with --safety=none to delete all unused blobs
* testing: allow debugging of integration tests inside VSCode
* testing: added end-to-end maintenance test that verifies that full maintenance with --safety=none removes all data
* content: fixed data race in IterateUnreferencedBlobs
* upload: fixed data race between uploader and estimator
* testing: fixed data race in repo/blob/logging test
* makefile: run tests on CI/linux/amd64 with -race
* robustness: fixed test race
* content: fixed data race getContentDataUnlocked that triggers TestParallelWrites - looks scary but in practice very hard to trigger in real life and does not cause data corruption
* testing: reduce test complexity under race detector
* server: fixed minor race in refreshStatus()
* testing: reduced depth of sharedTestDataDir2
* ci: run race detector in separate job
* ci: run unit test race detector in parallel to integration tests
* test: allow creating S3 buckets with object locking enabled
Split createBucket into createClient and makeBucket.
Both createClient and makeBucket will be used for testing
buckets with object locking enabled
* blob: needMD5 helper and tests
* blob: send content MD5 when object locking retention is enabled
From S3 documentation:
>
> If you configure a default retention period on a bucket, requests to
> upload objects in such a bucket must include the Content-MD5 header.
>
> The Content-MD5 header is required for any request to upload an
> object with a retention period configured using Amazon S3 Object Lock.
>
Ref:
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html#object-lock-bucket-config
- https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
* blob: ignore errors when getting S3 bucket object-lock config
If there is an error, assume the content MD5 is not neede on PUT.
* test: split getProviderOptions and getProviderOptionsAndCleanup
* test: TestNoNeedMD5Providers
* Send MD5 when store requires Content-MD5 header
Required for writing blobs to an AWS S3 bucket with object-locking
enabled and a retention period for the blob.
* Check whether the error message contains "content-md5"
Instead of doing an exact message comparison to reduce
the impact of potential message changes.
* added framework for unit testing against remote real rclone remotes,
added google drive backend
* added parallelism to blobtesting which revealed some races during
PutBlob with WebDAV.
