* testing: added logging to figure out root cause of the flake in TestSnapshotCounters
* fixed test flake caused by not waiting for upload task to be created
This adds new set of APIs `/api/v1/control/*` which can be used to administratively control a running server.
Once the server is started, the administrative user can control it
using CLI commands:
export KOPIA_SERVER_ADDRESS=...
export KOPIA_SERVER_CERT_FINGERPRINT=...
export KOPIA_SERVER_PASSWORD=...
* `kopia server status` - displays status of sources managed by the server
* `kopia server snapshot` - triggers server-side upload of snapshots for managed sources
* `kopia server cancel` - cancels upload of snapshots for managed sources
* `kopia server pause` - pauses scheduled snapshots for managed sources
* `kopia server resume` - resumes scheduled snapshots for managed sources
* `kopia server refresh` - causes server to resynchronize with externally-made changes, such as policies or new sources
* `kopia server flush` - causes server to flush all pending writes
* `kopia server shutdown` - graceful shutdown of the server
Authentication uses new user `server-control` and is disabled
by default. To enable it when starting the server, provide the password
using one of the following methods:
* `--server-control-password`
* `--random-server-control-password`
* `.htpasswd` file
* `KOPIA_SERVER_CONTROL_PASSWORD` environment variable
This change allows us to tighten the API security and remove some
methods that UI user was able to call, but which were not needed.
Coalesce retention period tests. This has the following effects:
- serializes the tests
- reuses the bucket and reduces bucket creation calls
Also:
- check whether bucket exists before attempting to create it
- nit: rename test for clarity and rename non-AWS specific tests
- nit: cleanup test logging and contexts
* sharded: plumbed through blob.PutOptions
* blob: removed blob.Storage.SetTime() method
This was only used for `kopia repo sync-to` and got replaced with
an equivalent blob.PutOptions.SetTime, which wehn set to non-zero time
will attempt to set the modification time on a file.
Since some providers don't support changing modification time, we
are able to emulate it using per-blob metadata (on B2, Azure and GCS),
sadly S3 is still unsupported, because it does not support returning
metadata in list results.
Also added PutOptions.GetTime, which when set to not nil, will
populate the provided variable with actual time that got assigned
to the blob.
Added tests that verify that each provider supports GetTime
and SetTime according to this spec.
* blob: additional test coverage for filesystem storage
* blob: added PutBlobAndGetMetadata() helper and used where appropriate
* fixed test failures
* pr feedback
* Update repo/blob/azure/azure_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* blobtesting: fixed object_locking_map.go
* blobtesting: removed SetTime from ObjectLockingMap
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
The source code for htmlui is now in github.com/kopia/htmlui
GitHub Actions will drop compiled builds in github.com/kopia/htmluibuild
where they are now used as a go module dependency.
This greatly simplifies the build and improves the security,
because Kopia will be consuming pre-built htmlui.
This also means kopia can now installed with embedded UI using:
`go install github.com/kopia/kopia@latest`
Turns out B2 library will treat seekable readers in a special way
and will pass those directly to http.NewRequest() for upload. Other
readers are copied to a temporary buffer first.
The #1589 made the reader io.ReadSeeker so it was passed to http.NewRequest(),
which for payloads of length zero uses heuristics to determine if the
Content-Length should be passed as zero or not passed at all.
https://cs.opensource.google/go/go/+/refs/tags/go1.17.5:src/net/http/request.go;l=890;drc=e6dda19888180c5159460486d30c0412e4980748
Since reader was not one of the magic types, Content-Length was not
passed at all, causing B2 server to choke on zero-length files.
Previously epoch manager would discard (on read) any index blobs
associated with epoch N written after epoch N+2 has already started.
There was no signal to the writer that the write is actually bad. This
change reports slow writes as a failure and disables read-time filtering.
This is important for repositories copied using sync-to because it
may not necessarily preserve the modification time leading to problems
with restore.
* blob: speed up filesystem storage test
* blob: introduced osInterface to allow unit testing of filesystem calls
* testing: added a bunch of unit tests for corner cases in filesystem storage
Fixed a few cases where we were missing the retry loop.
