Fixed regression caused by #1980
Switching to new mmap library uncovered a bug where we were not properly
closing indices that are no longer in use, since previous mmap had
a finalizer defined to do so.
This fixes regression introduced in #1960.
Tested by backing up Linux 5.14.8 source on M1 Mac (average of 15 runs):
Before: duration=6.7s avg_heap_objects=7411657 avg_heap_bytes=871794888
After: duration=5.6s (17% faster) avg_heap_objects=5947800 (20% less) avg_heap_bytes=795762120 (9% less)
* fix(snapshots): fixed random deadlock when Uploader results in a failure
The deadlock was caused by not properly waiting for all asynchronous
work to complete before closing the worker pool.
Introduced `workshare.AsyncGroup.Close()` and some assertions.
* fixed select race
* linter fix
* pr feedback
* Remove remaining internal uses of Readdir
* Remove old helpers and interface functions.
* Update tests for updated fs.Directory interface
* Fix index out of range error in snapshot walker
Record one error if an error occurred and it's not limiting errors
* Use helper functions more; exit loops early
Follow up on reviewer comments and reduce code duplication, use more
targetted functions like Directory.Child, and exit directory iteration
early if possible.
* Remove fs.Entries type and unused functions
Leave some functions dealing with sorting and finding entries in fs
package. This retains tests for those functions while still allowing
mockfs to access them.
* Simplify function return
* test MinContentAgeSubjectToGC
* lint: move check for whether content is deleted to the caller to reduce gocycle complexity
* nit: add new lines before return
* log GC stats
* fix(maintenance): use a fixed time for protecting newly created content
Previously, the reference time used to determine whether a content had been
recently created would change througout a snapshot GC execution. For
long-running GC tasks, this non-deterministically shrinked the safety window
specified in `MinContentAgeSubjectToGC`.
Now, the snapshot GC starting time is used as a fix refererence for the
safety check.
* remove test entry point to avoid double execution of the test
New flag `--enable-jaeger-collector` and the corresponding
`KOPIA_ENABLE_JAEGER_COLLECTOR` environment variable enables Jaeger
exporter, which by default sends OTEL traces to Jaeger collector on
http://localhost:14268/api/traces
To change this, use environment variables:
* `OTEL_EXPORTER_JAEGER_ENDPOINT`
* `OTEL_EXPORTER_JAEGER_USER`
* `OTEL_EXPORTER_JAEGER_PASSWORD`
When tracing is disabled, the impact on performance is negligible.
To see this in action:
1. Download latest Jaeger all-in-one from https://www.jaegertracing.io/download/
2. Run `jaeger-all-in-one` binary without any parameters.
3. Run `kopia --enable-jaeger-collector snapshot create ...`
4. Go to http://localhost:16686/search and search for traces
When enabled, metrics are pushed to the provided Prometheus Push
Gateway at the start and end of each command and periodically every
few seconds.
```
--metrics-push-addr=http://address:port
--metrics-push-interval=5s
--metrics-push-job=kopia
--metrics-push-grouping=a:b --metrics-push-grouping=c:d
--metrics-push-username=user
--metrics-push-password=pass
```
* feat(repository): switched to using go-mmap for indexes
Previous mmap implementation only allowed io.ReaderAt API
this one allows direct access to the underlying bytes which helps
remove a bunch of buffers, copying and allocation in index parsing.
* let the compiler do bounds check
* removed one more unnecessary allocation
* pr feedback
When combined with #1963, it significantly reduces memory usage.
When backing up Kopia enlistment with various binaries 2.8GB
(files:74180 dirs:12322):
Before: max memory 440MB, time 5.8s
After: max memory 360MB, time 5.4s
Avoids adding an entry to the maintenance schedule for a maintenance
run that may not execute at all.
Also, avoids the potential race of overwriting the schedule of an
already running maintenance task.
nit: rename function to `checkClockSkewBounds`
* refactor(repository): ensure we always parse content.ID and object.ID
This changes the types to be incompatible with string to prevent direct
conversion to and from string.
This has the additional benefit of reducing number of memory allocations
and bytes for all IDs.
content.ID went from 2 allocations to 1:
typical case 32 characters + 16 bytes per-string overhead
worst-case 65 characters + 16 bytes per-string overhead
now: 34 bytes
object.ID went from 2 allocations to 1:
typical case 32 characters + 16 bytes per-string overhead
worst-case 65 characters + 16 bytes per-string overhead
now: 36 bytes
* move index.{ID,IDRange} methods to separate files
* replaced index.IDFromHash with content.IDFromHash externally
* minor tweaks and additional tests
* Update repo/content/index/id_test.go
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* Update repo/content/index/id_test.go
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* pr feedback
* post-merge fixes
* pr feedback
* pr feedback
* fixed subtle regression in sortedContents()
This was actually not producing invalid results because of how base36
works, just not sorting as efficiently as it could.
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
* feat(cli): implementation for 'kopia snapshot fix'
This allows modifications and fixes to the snapshots after they have
been taken.
Supported are:
* `kopia snapshot fix remove-invalid-files [--verify-files-percent=X]`
Removes all directory entries where the underlying files cannot be
read based on index analysis (this does not read the files, only index
structures so is reasonably quick).
`--verify-files-percent=100` can be used to trigger full read for
all files.
* `kopia snapshot fix remove-files --object-id=<object-id>`
Removes the object with a given ID from the entire snapshot tree.
Useful when you accidentally snapshot a sensitive file.
* `kopia snapshot fix remove-files --filename=<wildcard>`
Removes the files with a given name from the entire snapshot tree.
Useful when you accidentally snapshot a sensitive file.
By default all snapshots are analyzed and rewritten. To limit the scope
use:
--source=user@host:/path
--manifest-id=manifestID
By default the rewrite operation writes new directory entries but
does not replace the manifests. To do that pass `--commit`.
Related #1906Fixes#799
reorganized CLI per PR suggestion
* additional logging for diff command
* added Clone() method to snapshot manifst and directory entry
* added a comprehensive test, moved DirRewriter to separate file
* pr feedback
* more pr feedback
* improved logging output
* disable test in -race configuration since it's way to slow
* pr feedback
- nit: re-group struct fields
- nit: use consts
- nit: remove unnecessary fmt.Errorf(...) usage
- nit: avoid unnecessarily calling defaultActionControls when there is already a value
- robustness: increase readability in actions map declaration
- Prefer named functions over closures for the actions.
- nit fix typo
- nit: simplify robustness Log.StringThisRun
Iterates only over the tail of the slice, which avoids iterating over the entire slice
- expand command flag description for clarification
- include blob id in blob get error in the cache
- nit: remove unused BOTO_PATH
- nit: fix comment
- cleanup: remove unnecessary function declaration in interface
- leverage 'testify' to simplify test
When environment variable `KOPIA_IGNORE_MAINTENANCE_REWRITE_ERROR` is set, this will ignore rewrite problems for deleted contents because of mising blobs. In case of data corruption this should let maintenance complete and eventually completely drop unused content from the index.
This is not something that regular folks should ever use, except when recovering repository after a data loss after being instructed to do so by Kopia crew.
For #1946
* New interface method to iterate over dir entries
* Fix build and test failures from interface
* Fix entry iteration for StaticDirectory
* Make utility function for directory iteration
* Fix lint errors
* No wrapcheck on fs.ReaddirToIterate
* Be consistent for IterateEntry implementations
