* Initial implementation of ecc using Encryptor interface
* Created benchmark ecc command
* Fixing the order inside the wrapper
* Removed rs_bw because it is always worse
* Fixing naming and adding more comments
* Different approaches depending of file size/space overhead
* Fixes requested in PR
* Fixed lint errors
* Fixes requested in the PR
* Fixed import order
* Fixed more lint errors
* build(ci): added rclone 1.57.0 to Docker container images
/bin/rclone is configured to use config in /app/rclone/rclone.conf
so the user can pass -v ~/.config/rclone:/app/rclone
* pr feedback
* Replace htmlui_fallback.go with go:embed
* Replace go-bindata generated UI with go:embed
* Update site Go version to 1.16
* Update BUILD.md to reflect workflow with go:embed
The new files policy oneFileSystem ignores files that are mounted to
other filesystems similarly to tar's --one-file-system switch. For
example, if this is enabled, backing up / should now automatically
ignore /dev, /proc, etc, so the directory entries themselves don't
appear in the backup. The value of the policy is 'false' by default.
This is implemented by adding a non-windows-field Device (of type
DeviceInfo, reflecting the implementation of Owner) to the Entry
interface. DeviceInfo holds the dev and rdev acquired with stat (same
way as with Owner), but in addition to that it also holds the same
values for the parent directory. It would seem that doing this in some
other way, ie. in ReadDir, would require modifying the ReadDir
interface which seems a too large modification for a feature this
small.
This change introduces a duplication of 'stat' call to the files, as
the Owner feature already does a separate call. I doubt the
performance implications are noticeable, though with some refactoring
both Owner and Device fields could be filled in in one go.
Filling in the field has been placed in fs/localfs/localfs.go where
entryFromChildFileInfo has acquired a third parameter giving the the
parent entry. From that information the Device of the parent is
retrieved, to be passed off to platformSpecificDeviceInfo which does
the rest of the paperwork. Other fs implementations just put in the
default values.
The Dev and Rdev fields returned by the 'stat' call have different
sizes on different platforms, but for convenience they are internally
handled the same. The conversion is done with local_fs_32bit.go and
local_fs_64bit.go which are conditionally compiled on different
platforms.
Finally the actual check of the condition is in ignorefs.go function
shouldIncludeByDevice which is analoguous to the other similarly named
functions.
Co-authored-by: Erkki Seppälä <flux@inside.org>
The benchmarks creates 20 GB of files in different configurations
* 10 x 2 GB files
* 100 x 200 MB files
* 1000 x 20 MB files
and backs them up to a local filesystem repository measuring time,
CPU and RAM usage.
The benchmarking script uses GCP instance (n1-standard-8) with fast NVME
flash to eliminate local filesystem latency.
Current performance numbers show major improvement in latency in
0.7.0-rc1 due to splitter throughput optimization (#606).
This is enabled by `kopia server --ui` and can be viewed in a browser
at http://localhost:51515/
Right now it can only list snapshots and policies (barely).
This change removes lots of pointless string and custom format parsing code and instead relies on protobuf to do the thing. JSON is still an option thanks to proto3-generated output.
- Refactored ObjectID to use protobuf
- Indirect blocks to use protobuf (in the form of seek table)
- Added generic proto stream reader and writer
- ObjectIDFormat became proto enum
Also:
- fixed major issue where indirect object ID entries were not encrypted, since they were storing block IDs and not object IDs.
- dropped support for majority of formats, only supporting HMAC-{SHA256,SHA512,SHA512_384} hashes with AES256.
This produces object IDs of 32 or 64 characters long.
- changed how pretty-printing works for 'kopia show'
