Commit Graph

12 Commits

Author SHA1 Message Date
Jarek Kowalski
792cc874dc repo: allow reusing of object writer buffers (#1315)
This reduces memory consumption and speeds up backups.

1. Backing up kopia repository (3.5 GB files:133102 dirs:20074):

before: 25s, 490 MB
after: 21s, 445 MB

2. Large files (14.8 GB, 76 files)

before: 30s, 597 MB
after: 28s, 495 MB

All tests repeated 5 times for clean local filesystem repo.
2021-09-25 14:54:31 -07:00
Jarek Kowalski
7e68d8e4c1 Consolidated format version flags (#1284) 2021-09-08 18:44:03 -07:00
Jarek Kowalski
730ba7b94a Repository password change support (#1197)
* repo: added 'enable password change' flag (defaults to true for new repositories), which prevents embedding replicas of kopia.repository in pack blobs

* cli: added 'repo change-password' which can change the password of a connected repository

* repo: nit - renamed variables and functions dealing with key derivation

* repo: fixed cache validation HMAC secret to use stored HMAC secret instead of password-derived one

* cli: added test for repo change-password

* repo: negative cases for attempting to change password in an old repository

* Update cli/command_repository_change_password.go

Co-authored-by: Julio Lopez <julio+gh@kasten.io>

Co-authored-by: Julio Lopez <julio+gh@kasten.io>
2021-07-17 07:58:02 -07:00
Jarek Kowalski
4b251bdaac mechanical: added ctx parameter to repo.{Direct}WriteSession callback (#1114) 2021-06-02 23:12:30 -07:00
Jarek Kowalski
40510c043d Support for content-level compression (#1076)
* cli: added a flag to create repository with v2 index features

* content: plumb through compression.ID parameter to content.Manager.WriteContent()

* content: expose content.Manager.SupportsContentCompression

This allows object manager to decide whether to create compressed object
or let the content manager do it.

* object: if compression is requested and the repo supports it, pass compression ID to the content manager

* cli: show compression status in 'repository status'

* cli: output compression information in 'content list' and 'content stats'

* content: compression and decompression support

* content: unit tests for compression

* object: compression tests

* testing: added integration tests against v2 index

* testing: run all e2e tests with and without content-level compression

* htmlui: added UI for specifying index format on creation

* cli: additional tests for 'content ls' and 'content stats'

* applied pr suggestions
2021-05-22 05:35:27 -07:00
Jarek Kowalski
2c2c9d52e0 nit: refactored repetitive reportesting setup code (#916) 2021-03-29 14:52:14 -07:00
Jarek Kowalski
de840547e6 Improved upload reporting (#832)
* blob: refactored upload reporting

Instead of plumbing this through blob storage context, we are passing
and explicit callback that reports uploads as they happen.

* htmlui: improved counter presentation

* nit: added missing UI route which fixes Reload behavior on the Tasks page
2021-02-13 10:51:11 -08:00
Jarek Kowalski
fa7976599c repo: refactored repository interfaces (#780)
- `repo.Repository` is now read-only and only has methods that can be supported over kopia server
- `repo.RepositoryWriter` has read-write methods that can be supported over kopia server
- `repo.DirectRepository` is read-only and contains all methods of `repo.Repository` plus some low-level methods for data inspection
- `repo.DirectRepositoryWriter` contains write methods for `repo.DirectRepository`

- `repo.Reader` removed and merged with `repo.Repository`
- `repo.Writer` became `repo.RepositoryWriter`
- `*repo.DirectRepository` struct became `repo.DirectRepository`
  interface

Getting `{Direct}RepositoryWriter` requires using `NewWriter()` or `NewDirectWriter()` on a read-only repository and multiple simultaneous writers are supported at the same time, each writing to their own indexes and pack blobs.

`repo.Open` returns `repo.Repository` (which is also `repo.RepositoryWriter`).

* content: removed implicit flush on content manager close
* repo: added tests for WriteSession() and implicit flush behavior
* invalidate manifest manager after write session

* cli: disable maintenance in 'kopia server start'
  Server will close the repository before completing.

* repo: unconditionally close RepositoryWriter in {Direct,}WriteSession
* repo: added panic in case somebody tries to create RepositoryWriter after closing repository
  - used atomic to manage SharedManager.closed

* removed stale example
* linter: fixed spurious failures

Co-authored-by: Julio López <julio+gh@kasten.io>
2021-01-20 11:41:47 -08:00
Julio López
70df5f738c testing: Refactor faketime (#597)
* Allow auto-advance in faketime.TimeAdvance
* Leverage TimeAdvance in faketime.AutoAdvance
* Concurrent test for faketime.AdvanceTime
2020-09-10 00:52:14 -07:00
Jarek Kowalski
9a6dea898b Linter upgrade to v1.30.0 (#526)
* fixed godot linter errors
* reformatted source with gofumpt
* disabled some linters
* fixed nolintlint warnings
* fixed gci warnings
* lint: fixed 'nestif' warnings
* lint: fixed 'exhaustive' warnings
* lint: fixed 'gocritic' warnings
* lint: fixed 'noctx' warnings
* lint: fixed 'wsl' warnings
* lint: fixed 'goerr113' warnings
* lint: fixed 'gosec' warnings
* lint: upgraded linter to 1.30.0
* lint: more 'exhaustive' warnings

Co-authored-by: Nick <nick@kasten.io>
2020-08-12 19:28:53 -07:00
Jarek Kowalski
be4b897579 Support for remote repository (#427)
Support for remote content repository where all contents and
manifests are fetched over HTTP(S) instead of locally
manipulating blob storage

* server: implement content and manifest access APIs
* apiclient: moved Kopia API client to separate package
* content: exposed content.ValidatePrefix()
* manifest: added JSON serialization attributes to EntryMetadata
* repo: changed repo.Open() to return Repository instead of *DirectRepository
* repo: added apiServerRepository
* cli: added 'kopia repository connect server'
  This sets up repository connection via the API server instead of
  directly-manipulated storage.
* server: add support for specifying a list of usernames/password via --htpasswd-file
* tests: added API server repository E2E test
* server: only return manifests (policies and snapshots) belonging to authenticated user
2020-05-02 21:41:49 -07:00
Julio López
d9ce3d0ad6 Inject time in Kopia components (#314)
Motivation: Allow time injection for (unit) tests, to more easily test and
verify time-dependent invariants.

Add time injection support for:

* repo.Manager
* manifest.Manager
* snapshot.Uploader

Then, wire up to these components. The content.Manager already had support for
time injection, but was not wired up from the time function passed to repo creation.

Add an internal/faketime package for testing. Mainly code movement from testing
code in the repo/content package. Motivation: make it available to other packages
outside content Also, add simple tests for faketime functions.
2020-03-10 00:42:10 -07:00