Jarek Kowalski
846a46b879
changed how caching is controlled in block manager by using context instead of options
2018-06-10 10:20:52 -07:00
Jarek Kowalski
6838d820c8
revamped object verification to be parallelized, added option to randomly verify a percentage of object contents
2018-06-09 22:15:36 -07:00
Jarek Kowalski
e5e5577cfd
fixed linter warnings
2018-06-09 14:06:43 -07:00
Jarek Kowalski
fc618b0c66
changed block.PhysicalBlockID to a string
2018-06-07 18:42:22 -07:00
Jarek Kowalski
6ee5c81e21
renamed 'pack block ID' to 'pack file' everywhere
2018-06-07 07:52:17 -07:00
Jarek Kowalski
f869996a1c
aligned 'ls' output
2018-06-06 21:38:32 -07:00
Jarek Kowalski
9ea74b2862
when reporting stats to GA do it asynchronously at start and do not block for more than 1s on exit
2018-06-06 21:38:08 -07:00
Jarek Kowalski
7738fa536f
revamped 'object verify' to support verifying one or more sources easily
2018-06-06 21:18:44 -07:00
Jarek Kowalski
ebd8b0a943
CLI improvements
2018-06-04 22:55:57 -07:00
Jarek Kowalski
416475a092
added RewriteBlock() support which can rewrite either deleted or non-deleted block to a new pack
...
Added 'block gc' and 'block rewrite' command-line subcommands.
2018-06-04 22:09:10 -07:00
Jarek Kowalski
02c5e90671
added new cli flags to 'storage list'
2018-06-02 10:46:59 -07:00
Jarek Kowalski
5ba913380e
added support for soft-delete that preserves original pack ID/offset/length.
...
removed legacy soft-delete support without this information
2018-05-30 21:22:07 -07:00
Jarek Kowalski
53db414ff7
Added simple analytics mechanism based on Google Analytics for tracking features usage and latency.
...
Controlled on command line via --analytics-consent (defaults to asking user)
2018-05-30 21:22:07 -07:00
Jarek Kowalski
80a3bdfd4a
fixed race condition when multiple kopia instances are downloading indexes in parallel
2018-05-20 10:45:13 -07:00
Jarek Kowalski
e36d02c59b
emit per-dir incomplete reason
...
store entire rootEntry on snapshot nmanifest
format 'snapshot list' consistently for both root and nested items
2018-05-19 22:00:14 -07:00
Jarek Kowalski
e6c8144b93
output pack block id in 'blockindex show active'
2018-05-17 23:26:43 -07:00
Jarek Kowalski
96d27c2293
improved block list and index format
2018-05-17 22:41:54 -07:00
Jarek Kowalski
ec779b14c7
refactored and cleaned up policy management, added retention tags to snapshot list
2018-05-17 20:31:37 -07:00
Jarek Kowalski
c907580aaf
when running 'snapshot create --all' ignore individual source error failures
2018-05-15 20:54:45 -07:00
Jarek Kowalski
d6477377f9
changed how index compaction works:
...
instead of having time-based naming, block manager will perform occasional compaction at startup time and delete unwanted blocks
The protocol is safe from concurrency standpoint and provides eventual consistency.
We only delete blocks after creating compacted blocks.
We retry loading index until (any) consistent index is fetched (possibly from cache) and all underlying blocks are also available, not necessarily the latest ones.
TODO - we need to periodically snapshot block index contents, so that if we have a bug somewhere in compaction code, we have a way of restoring working indexes.
2018-05-15 20:10:50 -07:00
Jarek Kowalski
7e6154f6f3
fixed progress bar
2018-05-13 21:55:00 -07:00
Jarek Kowalski
12c28e537d
simplified ObjectID code and changed it to a string
2018-05-09 22:22:37 -07:00
Jarek Kowalski
62ce401cd6
exposed block.Manager.ListBlocks() which only returns block IDs instead of full info
2018-05-05 16:16:48 -07:00
Jarek Kowalski
3a02ef24e0
retire block.ContentID and replace with string
2018-05-05 16:16:48 -07:00
Jarek Kowalski
1342fd910a
switched to using custom block index designed for efficient lookup without having to parse it
...
the block index stores tightly packed sorted block IDs
Typically a single entry 128-bit content ID uses 37 bytes of index storage.
2018-05-05 16:14:31 -07:00
Jarek Kowalski
250ceff6fd
removed support for pack index V1
2018-04-29 14:10:20 -07:00
Jarek Kowalski
5ede755524
added pack index v2
2018-04-29 13:44:12 -07:00
Jarek Kowalski
4428743a5b
added --raw option to 'blockindex show'
2018-04-29 13:44:12 -07:00
Jarek Kowalski
726d33e0d5
added levelDB-based persistent block index
2018-04-29 13:44:12 -07:00
Jarek Kowalski
c4c74586c6
refactored block manager so that loaded block index is separate from pending block index
...
expanded block.Info to include additional details, removed packBlockInfo
removed repackaging support for now, since it's greatly complicating the implementation
2018-04-29 13:44:00 -07:00
Jarek Kowalski
f94fcea14b
moved block cache to a subdirectory
2018-04-23 19:56:01 -07:00
Jarek Kowalski
924e0a4594
renamed index proto types to IndexV1
2018-04-08 18:45:22 -07:00
Jarek Kowalski
301f1d049a
added strongly-typed block.ContentID and renamed StorageBlock to ContentBlockID
2018-04-08 16:34:38 -07:00
Jarek Kowalski
4566213f5d
switched formatter to pass IV explicitly
...
extracted packIndex interface and implemented protoPackIndex to wrap all index data manipulation
2018-04-08 12:20:32 -07:00
Jarek Kowalski
d8201229d8
plumbed through ctx in storage.Storage APIs and all uses
2018-04-03 17:39:54 -07:00
Jarek Kowalski
a608c2f443
tweaked snapshot list
2018-03-24 12:22:21 -07:00
Jarek Kowalski
1c4e0e1e47
removed legacy oid format
2018-03-19 19:31:03 -07:00
Jarek Kowalski
2e61cc5688
code simplification, enabled gocyclo linter
2018-03-19 14:08:19 -07:00
Jarek Kowalski
fa5eae501a
removed cleanup for now
2018-03-19 12:26:28 -07:00
Jarek Kowalski
453bab3560
fixed some lint errors, mostly dead code and missing error checks
2018-03-19 12:26:28 -07:00
Jarek Kowalski
4657f16cc8
CLI improvement for automatic logging and log rotation
2018-03-16 22:21:57 -07:00
Jarek Kowalski
47b1e09ba9
added default CLI commands
2018-03-15 09:39:39 -07:00
Jarek Kowalski
54aeea8ee6
fixed block list for inline blocks
2018-03-14 19:27:52 -07:00
Jarek Kowalski
70ed5647e4
added parallel uploading support
2018-03-14 18:22:50 -07:00
Jarek Kowalski
a50546de0e
fixed previous manifest selection for incremental snapshots
2018-03-12 16:50:01 -07:00
Jarek Kowalski
772ef9a286
added 'ls' alias for 'snapshot list'
2018-03-04 18:17:18 -08:00
Jarek Kowalski
92330ac247
refactored policy manager to use SourceInfo
2018-02-19 13:20:32 -08:00
Jarek Kowalski
16e44cb37f
renamed policy.Manager as snapshot.PolicyManager and policy.Policy as snapshot.Policy
2018-02-19 10:59:15 -08:00
Jarek Kowalski
d97f49285f
policy CLI and API cleanup
2018-02-19 10:53:29 -08:00
Jarek Kowalski
e710dcdff3
additional integration tests
2018-02-18 11:59:32 -08:00
Jarek Kowalski