mirror/kopia
1
0
Fork 0
mirror of https://github.com/kopia/kopia.git synced 2026-01-29 08:43:09 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
deb1099c52184fa5028275bd4e9bc357409e4e17
kopia/Makefile
Jarek Kowalski d68273a576 Improvements for dealing with eventually-consistent stores (S3) (#437) 
* content: added support for cache of own writes

Thi keeps track of which blobs (n and m) have been written by the
local repository client, so that even if the storage listing
is eventually consistent (as in S3), we get somewhat sane behavior.

Note that this is still assumming read-after-create semantics, which
S3 also guarantees, otherwise it's very hard to do anything useful.

* compaction: support for compaction logs

Instead of compaction immediately deleting source index blobs, we now
write log entries (with `m` prefix) which are merged on reads
and applied only if the blob list includes all inputs and outputs, in
which case the inputs are discarded since they are known to have been
superseded by the outputs.

This addresses eventual consistency issues in stores such as S3,
which don't guarantee list-after-put or list-after-delete. With such
stores the repository is ultimately eventually consistent and there's
not much that can be done about it, unless we use second strongly
consistent storage (such as GCS) for the index only.

* content: updated list cache to cache both `n` and `m`

* repo: fixed cache clear on windows

Clearing cache requires closing repository first, as Windows is holding
the files locked.

This requires ability to close the repository twice.

* content: refactored index blob management into indexBlobManager

* testing: fixed blobtesting.Map storage to allow overwrites

* blob: added debug output String() to blob.Metadata

* testing: added indexBlobManager stress test

This works by using N parallel "actors", each repeatedly performing
operations on indexBlobManagers all sharing single eventually consistent
storage.

Each actor runs in a loop and randomly selects between:

- *reading* all contents in indexes and verifying that it includes
  all contents written by the actor so far and that contents are
  correctly marked as deleted
- *creating* new contents
- *deleting* one of previously-created contents (by the same actor)
- *compacting* all index files into one

The test runs on accelerated time (every read of time moves it by 0.1
seconds) and simulates several hours of running.

In case of a failure, the log should provide enough debugging
information to trace the exact sequence of events leading up to the
failure - each log line is prefixed with actorID and all storage
access is logged.

* makefile: increase test timeout

* content: fixed index blob manager race

The race is where if we delete compaction log too early, it may lead to
previously deleted contents becoming temporarily live again to an
outside observer.

Added test case that reproduces the issue, verified that it fails
without the fix and passed with one.

* testing: improvements to TestIndexBlobManagerStress test

- better logging to be able to trace the root cause in case of a failure
- prevented concurrent compaction which is unsafe:

The sequence:

1. A creates contentA1 in INDEX-1
2. B creates contentB1 in INDEX-2
3. A deletes contentA1 in INDEX-3
4. B does compaction, but is not seeing INDEX-3 (due to EC or simply
   because B started read before #3 completed), so it writes
   INDEX-4==merge(INDEX-1,INDEX-2)
   * INDEX-4 has contentA1 as active
5. A does compaction but it's not seeing INDEX-4 yet (due to EC
   or because read started before #4), so it drops contentA1, writes
   INDEX-5=merge(INDEX-1,INDEX-2,INDEX-3)
   * INDEX-5 does not have contentA1
7. C sees INDEX-5 and INDEX-5 and merge(INDEX-4,INDEX-5)
   contains contentA1 which is wrong, because A has been deleted
   (and there's no record of it anywhere in the system)

* content: when building pack index ensure index bytes are different each time by adding 32 random bytes
2020-05-31 17:11:20 -07:00

264 lines
7.4 KiB
Makefile
Raw Blame History

COVERAGE_PACKAGES=github.com/kopia/kopia/repo/...,github.com/kopia/kopia/fs/...,github.com/kopia/kopia/snapshot/...
GO_TEST=go test
PARALLEL=8
TEST_FLAGS?=
KOPIA_INTEGRATION_EXE=$(CURDIR)/dist/integration/kopia.exe
FIO_DOCKER_TAG=ljishen/fio
all: test lint vet integration-tests
retry=
ifneq ($(TRAVIS_OS_NAME),)
retry=$(CURDIR)/tools/retry.sh
endif
include tools/tools.mk
-include ./Makefile.local.mk
install: html-ui-bindata
go install -tags embedhtml
quick-install:
# same as install but assumes HTMLUI has been built
go install -tags embedhtml
install-noui:
go install
escape-analysis:
go build -gcflags '-m -l' github.com/kopia/kopia/...
clean:
make clean-tools
make -C htmlui clean
rm -rf dist/ internal/server/htmlui_bindata.go
play:
go run cmd/playground/main.go
lint: $(linter)
$(linter) --deadline 180s run $(linter_flags)
lint-and-log: $(linter)
$(linter) --deadline 180s run $(linter_flags) | tee .linterr.txt
vet-time-inject:
ifneq ($(TRAVIS_OS_NAME),windows)
! find repo snapshot -name '*.go' -not -path 'repo/blob/logging/*' -not -name '*_test.go' \
-exec grep -n -e time.Now -e time.Since -e time.Until {} + \
| grep -v -e allow:no-inject-time
endif
vet: vet-time-inject
go vet -all .
travis-setup: travis-install-gpg-key travis-install-test-credentials all-tools
go mod download
make -C htmlui node_modules
make -C app node_modules
ifneq ($(TRAVIS_OS_NAME),)
-git checkout go.mod go.sum
endif
website:
$(MAKE) -C site build
html-ui:
$(MAKE) -C htmlui build-html CI=true
html-ui-tests:
$(MAKE) -C htmlui test CI=true
html-ui-bindata: html-ui $(go_bindata)
(cd htmlui/build && $(go_bindata) -fs -tags embedhtml -o "$(CURDIR)/internal/server/htmlui_bindata.go" -pkg server -ignore '.map' . static/css static/js static/media)
html-ui-bindata-fallback: $(go_bindata)
(cd internal/server && $(go_bindata) -fs -tags !embedhtml -o "$(CURDIR)/internal/server/htmlui_fallback.go" -pkg server index.html)
kopia-ui:
$(MAKE) -C app build-electron
travis-release:
$(retry) $(MAKE) goreleaser
$(retry) $(MAKE) kopia-ui
$(MAKE) lint vet test-with-coverage html-ui-tests
$(retry) $(MAKE) layering-test
$(retry) $(MAKE) integration-tests
ifeq ($(TRAVIS_OS_NAME),linux)
$(MAKE) robustness-tool-tests
$(MAKE) website
$(MAKE) stress-test
$(MAKE) travis-create-long-term-repository
$(MAKE) upload-coverage
endif
# goreleaser - builds binaries for all platforms
GORELEASER_OPTIONS=--rm-dist --parallelism=6
sign_gpg=1
ifneq ($(TRAVIS_PULL_REQUEST),false)
# not running on travis, or travis in PR mode, skip signing
sign_gpg=0
endif
ifeq ($(TRAVIS_OS_NAME),windows)
# signing does not work on Windows on Travis
sign_gpg=0
endif
ifeq ($(sign_gpg),0)
GORELEASER_OPTIONS+=--skip-sign
endif
publish_binaries=1
ifeq ($(TRAVIS_TAG),)
# not a tagged release
GORELEASER_OPTIONS+=--snapshot
publish_binaries=0
endif
ifneq ($(TRAVIS_OS_NAME),linux)
publish_binaries=0
endif
ifeq ($(publish_binaries),0)
GORELEASER_OPTIONS+=--skip-publish
endif
print_build_info:
@echo TRAVIS_TAG: $(TRAVIS_TAG)
@echo TRAVIS_PULL_REQUEST: $(TRAVIS_PULL_REQUEST)
@echo TRAVIS_OS_NAME: $(TRAVIS_OS_NAME)
goreleaser: $(goreleaser) print_build_info
-git diff | cat
$(goreleaser) release $(GORELEASER_OPTIONS)
ifeq ($(TRAVIS_PULL_REQUEST),false)
upload-coverage: $(GOVERALLS_TOOL)
$(GOVERALLS_TOOL) -service=travis-ci -coverprofile=tmp.cov
else
upload-coverage:
@echo Not uploading coverage during PR build.
endif
dev-deps:
GO111MODULE=off go get -u golang.org/x/tools/cmd/gorename
GO111MODULE=off go get -u golang.org/x/tools/cmd/guru
GO111MODULE=off go get -u github.com/nsf/gocode
GO111MODULE=off go get -u github.com/rogpeppe/godef
GO111MODULE=off go get -u github.com/lukehoban/go-outline
GO111MODULE=off go get -u github.com/newhook/go-symbols
GO111MODULE=off go get -u github.com/sqs/goreturns
test-with-coverage:
$(GO_TEST) -count=1 -coverprofile=tmp.cov --coverpkg $(COVERAGE_PACKAGES) -timeout 90s $(shell go list ./...)
test-with-coverage-pkgonly:
$(GO_TEST) -count=1 -coverprofile=tmp.cov -timeout 90s github.com/kopia/kopia/...
test:
$(GO_TEST) -count=1 -timeout 180s ./...
vtest:
$(GO_TEST) -count=1 -short -v -timeout 180s ./...
dist-binary:
go build -o $(KOPIA_INTEGRATION_EXE) github.com/kopia/kopia
integration-tests: export KOPIA_EXE ?= $(KOPIA_INTEGRATION_EXE)
integration-tests: dist-binary
$(GO_TEST) $(TEST_FLAGS) -count=1 -parallel $(PARALLEL) -timeout 600s github.com/kopia/kopia/tests/end_to_end_test
robustness-tool-tests:
FIO_DOCKER_IMAGE=$(FIO_DOCKER_TAG) \
$(GO_TEST) $(TEST_FLAGS) -count=1 -timeout 90s github.com/kopia/kopia/tests/tools/...
stress-test:
KOPIA_LONG_STRESS_TEST=1 $(GO_TEST) -count=1 -timeout 200s github.com/kopia/kopia/tests/stress_test
$(GO_TEST) -count=1 -timeout 200s github.com/kopia/kopia/tests/repository_stress_test
layering-test:
ifneq ($(uname),Windows)
# verify that code under repo/ can only import code also under repo/ + some
# whitelisted internal packages.
find repo/ -name '*.go' | xargs grep "^\t\"github.com/kopia/kopia" \
| grep -v -e github.com/kopia/kopia/repo \
-e github.com/kopia/kopia/internal \
-e github.com/kopia/kopia/issues && exit 1 || echo repo/ layering ok
endif
godoc:
godoc -http=:33333
coverage: test-with-coverage coverage-html
coverage-html:
go tool cover -html=tmp.cov
official-release:
git tag $(RELEASE_VERSION) -m $(RELEASE_VERSION)
git push -u upstream $(RELEASE_VERSION)
goreturns:
find . -name '*.go' | xargs goreturns -w --local github.com/kopia/kopia
# this indicates we're running on Travis CI and NOT processing pull request.
ifeq ($(TRAVIS_PULL_REQUEST),false)
# https://travis-ci.community/t/windows-build-timeout-after-success-ps-shows-gpg-agent/4967/4
travis-install-gpg-key:
ifeq ($(TRAVIS_OS_NAME),windows)
@echo Not installing GPG key on Windows...
else
@echo Installing GPG key...
openssl aes-256-cbc -K "$(encrypted_fa1db4b894bb_key)" -iv "$(encrypted_fa1db4b894bb_iv)" -in kopia.gpg.enc -out /tmp/kopia.gpg -d
gpg --import /tmp/kopia.gpg
endif
travis-install-test-credentials:
@echo Installing test credentials...
ifneq ($(TRAVIS_OS_NAME),windows)
openssl aes-256-cbc -K "$(encrypted_fa1db4b894bb_key)" -iv "$(encrypted_fa1db4b894bb_iv)" -in tests/credentials/gcs/test_service_account.json.enc -out repo/blob/gcs/test_service_account.json -d
openssl aes-256-cbc -K "$(encrypted_fa1db4b894bb_key)" -iv "$(encrypted_fa1db4b894bb_iv)" -in tests/credentials/sftp/id_kopia.enc -out repo/blob/sftp/id_kopia -d
openssl aes-256-cbc -K "$(encrypted_fa1db4b894bb_key)" -iv "$(encrypted_fa1db4b894bb_iv)" -in tests/credentials/sftp/known_hosts.enc -out repo/blob/sftp/known_hosts -d
endif
travis-install-cloud-sdk: travis-install-test-credentials
if [ ! -d $(HOME)/google-cloud-sdk ]; then curl https://sdk.cloud.google.com | CLOUDSDK_CORE_DISABLE_PROMPTS=1 bash; fi
$(HOME)/google-cloud-sdk/bin/gcloud auth activate-service-account --key-file repo/blob/gcs/test_service_account.json
else
travis-install-gpg-key:
@echo Not installing GPG key.
travis-install-test-credentials:
@echo Not installing test credentials.
travis-install-cloud-sdk:
@echo Not installing Cloud SDK.
endif
ifneq ($(TRAVIS_TAG),)
travis-create-long-term-repository: dist-binary travis-install-cloud-sdk
echo Creating long-term repository $(TRAVIS_TAG)...
KOPIA_EXE=$(KOPIA_INTEGRATION_EXE) ./tests/compat_test/gen-compat-repo.sh
else
travis-create-long-term-repository:
echo Not creating long-term repository.
endif
Reference in New Issue View Git Blame Copy Permalink