* sharded: plumbed through blob.PutOptions
* blob: removed blob.Storage.SetTime() method
This was only used for `kopia repo sync-to` and got replaced with
an equivalent blob.PutOptions.SetTime, which wehn set to non-zero time
will attempt to set the modification time on a file.
Since some providers don't support changing modification time, we
are able to emulate it using per-blob metadata (on B2, Azure and GCS),
sadly S3 is still unsupported, because it does not support returning
metadata in list results.
Also added PutOptions.GetTime, which when set to not nil, will
populate the provided variable with actual time that got assigned
to the blob.
Added tests that verify that each provider supports GetTime
and SetTime according to this spec.
* blob: additional test coverage for filesystem storage
* blob: added PutBlobAndGetMetadata() helper and used where appropriate
* fixed test failures
* pr feedback
* Update repo/blob/azure/azure_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* Update repo/blob/filesystem/filesystem_storage.go
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
* blobtesting: fixed object_locking_map.go
* blobtesting: removed SetTime from ObjectLockingMap
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
The source code for htmlui is now in github.com/kopia/htmlui
GitHub Actions will drop compiled builds in github.com/kopia/htmluibuild
where they are now used as a go module dependency.
This greatly simplifies the build and improves the security,
because Kopia will be consuming pre-built htmlui.
This also means kopia can now installed with embedded UI using:
`go install github.com/kopia/kopia@latest`
Previously epoch manager would discard (on read) any index blobs
associated with epoch N written after epoch N+2 has already started.
There was no signal to the writer that the write is actually bad. This
change reports slow writes as a failure and disables read-time filtering.
This is important for repositories copied using sync-to because it
may not necessarily preserve the modification time leading to problems
with restore.
* blob: speed up filesystem storage test
* blob: introduced osInterface to allow unit testing of filesystem calls
* testing: added a bunch of unit tests for corner cases in filesystem storage
Fixed a few cases where we were missing the retry loop.
* cli: refactored snapshot list
* cli: show range tags in snapshot list
For example if N snapshots are coalesced together because they
have identical roots we may emit now:
```
2021-03-31 23:09:27 PDT ked3400debc7dd61baffab070bafd59cd (monthly-10)
2021-04-30 06:12:53 PDT kd0576d212e55a831b7ff1636f90a7233 (monthly-4..9)
+ 5 identical snapshots until 2021-09-30 23:00:19 PDT
2021-10-31 23:22:25 PDT k846bf22aa2863d27f05e820f840b14f8 (monthly-3)
2021-11-08 21:29:31 PST k5793ddcd61ef27b93c75ab74a5828176 (latest-1..3,hourly-1..13,daily-1..7,weekly-1..4,monthly-1..2,annual-1)
+ 18 identical snapshots until 2021-12-04 10:09:54 PST
```
* server: server-side coalescing of snapshot
* ui: added coalescing of retention tags
* ui: changed how PolicyEditor is instantiated via a route
* server: added paths/resolve API
* server: refresh affected source manager after policy change
Also switched 15-second refresh cycle which is way too aggressive
to 30-minute cycle (manual refresh button can be used if needed).
* policy: allow overriding top-level policy for estimation
* server: changed source create API to always require policy
* ui: streamlined new snapshot and estimate flow
* linter fix
* logging: added log rotation and improved predictability of log sweep
With this change logs will be rotated every 50 MB, which prevents
accumulation of giant files while server is running.
This change will also guarantee that log sweep completes at least once
before each invocation of Kopia finishes. Previously it was a goroutine
that was not monitored for completion.
Flags can be used to override default behaviors:
* `--max-log-file-segment-size`
* `--no-wait-for-log-sweep` - disables waiting for full log sweep
Fixes#1561
* logging: added --log-dir-max-total-size-mb flag
This limits the total size of all logs in a directory to 1 GB.
This is done by protecting newly added cache items from being swept for
X amount of time where X defaults to:
* `metadata` - 24 hours (new)
* `data` - 10 min (new)
* `indexes` - 1 hours (same as today)
Fixes#1540
Added policy.Definition which allows us to precisely report where
each piece of policy came from.
Fixed a one-off bug with "noParent", which prevented merging of parent
policies one level too soon.
Added a whole bunch of merging helpers and generic reflection-based
test that ensures every single merge is tested.
* throtting: implemented a Throttler based on token bucket and configurable window.
* cli: rewired throttle options to use common Limits structure and helpers
The JSON is backwards compatible.
* blob: remove explicit throttling from gcs,s3,b2 & azure
* cleanup: removed internal/throttle
* repo: add throttling wrapper around storage at the repository level
* throttling: expose APIs to get limits and add validation
* server: expose API to get/set throttle in a running server
* pr feedback
* blob: changed default shards from {3,3} to {1,3}
Turns out for very large repository around 100TB (5M blobs),
we end up creating max ~16M directories which is way too much
and slows down listing. Currently each leaf directory only has a handful
of files.
Simple sharding of {3} should work much better and will end up creating
directories with meaningful shard sizes - 12 K files per directory
should not be too slow and will reduce the overhead of listing by
4096 times.
The change is done in a backwards-compatible way and will respect
custom sharding (.shards) file written by previous 0.9 builds
as well as older repositories that don't have the .shards file (which
we assume to be {3,3}).
* fixed compat tests
* fixed new gocritic violations
* fixed new 'contextcheck' violations
* fixed 'gosec' warnings
* suppressed ireturn and varnamelen linters
* fixed tenv violations, enabled building robustness tests on arm64
* fixed remaining linux failures
* makefile: fixed 'lint-all' target when running on arm64
* linter: increase deadline
* disable nilnil linter - to be enabled in separate PR
The issue in #1439 was caused by goroutine context being associated
with the HTTP request so it became canceled soon after the request was
over, thus the goroutine to run maintenance never ran.
Fixed by adding ctxutil.Detach()
Also fixed logging by passing top-level contexts to requests
and added --log-server-requests flag to `server start` which enables
request logging.
Instead of attempting maintenance every 10 minutes we will do a longer
sleep until the predicted next maintenance time (or 4 hours, whichever
is shorter).
Related #1439
This was introduced by a refactoring in #1361 - unlike
ioutil.ReadDir() which internally handles list/delete race and always
returns os.FileInfo, Info() on os.DirEntry can fail if a file
is deleted right after listing it.
Fixes#1486
* logging: added Logger.Debugw(message, key1, value1, ..., keyN, valueN)
This is based on ZAP and allows structural logs to be emitted.
* cli: added --json-log-console and --json-log-file flags
* logging: updated storage logging wrapper to use structural logging
* pr feedback
The dual time measurement is described in
https://go.googlesource.com/proposal/+/master/design/12914-monotonic.md
The fix is to discard hidden monotonic time component of time.Time
by converting to unix time and back.
Reviewed usage of clock.Now() and replaced with timetrack.StartTimer()
when measuring time.
The problem in #1402 was that passage of time was measured using
the monotonic time and not wall clock time. When the computer goes
to sleep, monotonic time is still monotonic while wall clock time makes
a leap when the computer wakes up. This is the behavior that
epoch manager (and most other compontents in Kopia) rely upon.
Fixes#1402
Co-authored-by: Julio Lopez <julio+gh@kasten.io>
This was caused by additional resolution of path names only done in UI,
which caused \\hostname\share to be treated as relative and resolved
against the home directory.
Fixes#1385Fixes#1362
