Jarek Kowalski
7f6c08835e
changed naming of retention tags to be more explanatory
2018-05-18 20:03:50 -07:00
Jarek Kowalski
e6c8144b93
output pack block id in 'blockindex show active'
2018-05-17 23:26:43 -07:00
Jarek Kowalski
490fa6130b
changed pack naming scheme to be p{random-128-bit}
2018-05-17 23:00:24 -07:00
Jarek Kowalski
96d27c2293
improved block list and index format
2018-05-17 22:41:54 -07:00
Jarek Kowalski
ec779b14c7
refactored and cleaned up policy management, added retention tags to snapshot list
2018-05-17 20:31:37 -07:00
Jarek Kowalski
c907580aaf
when running 'snapshot create --all' ignore individual source error failures
2018-05-15 20:54:45 -07:00
Jarek Kowalski
d6477377f9
changed how index compaction works:
...
instead of having time-based naming, block manager will perform occasional compaction at startup time and delete unwanted blocks
The protocol is safe from concurrency standpoint and provides eventual consistency.
We only delete blocks after creating compacted blocks.
We retry loading index until (any) consistent index is fetched (possibly from cache) and all underlying blocks are also available, not necessarily the latest ones.
TODO - we need to periodically snapshot block index contents, so that if we have a bug somewhere in compaction code, we have a way of restoring working indexes.
2018-05-15 20:10:50 -07:00
Jarek Kowalski
7e6154f6f3
fixed progress bar
2018-05-13 21:55:00 -07:00
Jarek Kowalski
ad6330fe58
disable test caching
2018-05-09 22:26:25 -07:00
Jarek Kowalski
6f7b3b8a80
fixed stress test failures due to expiration of a context
2018-05-09 22:22:37 -07:00
Jarek Kowalski
12c28e537d
simplified ObjectID code and changed it to a string
2018-05-09 22:22:37 -07:00
Jarek Kowalski
04acb44669
cleanup
2018-05-09 22:22:32 -07:00
Jarek Kowalski
0ba37c4ccb
run longer stress test on travis
2018-05-09 07:42:12 -07:00
Jarek Kowalski
a30af40f7a
added 1s sleep to fix flakiness in manifest_manager_test.go due to rounding timestamps down to 1s
2018-05-09 06:28:06 -07:00
Jarek Kowalski
db0fd9ed28
added packindex.IsSubset()
2018-05-06 16:37:35 -07:00
Jarek Kowalski
9c57f22c0f
simplified committed index block interface, speed up block block manager test by a lot
2018-05-05 16:16:48 -07:00
Jarek Kowalski
62ce401cd6
exposed block.Manager.ListBlocks() which only returns block IDs instead of full info
2018-05-05 16:16:48 -07:00
Jarek Kowalski
3a02ef24e0
retire block.ContentID and replace with string
2018-05-05 16:16:48 -07:00
Jarek Kowalski
be3c4e725e
added stress test
2018-05-05 16:14:31 -07:00
Jarek Kowalski
39aa586b80
fixed map storage data aliasing
2018-05-05 16:14:31 -07:00
Jarek Kowalski
1342fd910a
switched to using custom block index designed for efficient lookup without having to parse it
...
the block index stores tightly packed sorted block IDs
Typically a single entry 128-bit content ID uses 37 bytes of index storage.
2018-05-05 16:14:31 -07:00
Jarek Kowalski
250ceff6fd
removed support for pack index V1
2018-04-29 14:10:20 -07:00
Jarek Kowalski
3464059031
cleanup
2018-04-29 13:53:35 -07:00
Jarek Kowalski
5ede755524
added pack index v2
2018-04-29 13:44:12 -07:00
Jarek Kowalski
5912c628bb
added copyPackIndex()
2018-04-29 13:44:12 -07:00
Jarek Kowalski
4428743a5b
added --raw option to 'blockindex show'
2018-04-29 13:44:12 -07:00
Jarek Kowalski
726d33e0d5
added levelDB-based persistent block index
2018-04-29 13:44:12 -07:00
Jarek Kowalski
c4c74586c6
refactored block manager so that loaded block index is separate from pending block index
...
expanded block.Info to include additional details, removed packBlockInfo
removed repackaging support for now, since it's greatly complicating the implementation
2018-04-29 13:44:00 -07:00
Jarek Kowalski
f94fcea14b
moved block cache to a subdirectory
2018-04-23 19:56:01 -07:00
Jarek Kowalski
df7a9e217a
added support for 'v1' pack format that stores each packed block individually encrypted as opposed to encrypting the entire pack
2018-04-23 17:51:00 -07:00
Jarek Kowalski
0f63f04c14
changed how pack block are staged to first add them as inline, then convert to a block as necessary and not the opposite
2018-04-21 12:56:18 -07:00
Jarek Kowalski
ab4d2c7f8e
store format version with the index, not used yet
2018-04-21 10:36:53 -07:00
Jarek Kowalski
52bcb6223f
when loading active index, load 10 minutes extra to correct for clock skew and in-flight uploads
2018-04-21 10:36:53 -07:00
Jarek Kowalski
49dfc43ad3
example: stopped persisting credentials
2018-04-21 06:28:53 -07:00
Jarek Kowalski
8763c31db9
added examples of using repository API
2018-04-15 20:12:02 -07:00
Jarek Kowalski
7d08d9921b
fail to initialize repository if the format block already exists
2018-04-15 20:09:54 -07:00
Jarek Kowalski
7159c2ed07
removed packInfo.isEmpty
2018-04-14 19:26:58 -07:00
Jarek Kowalski
a2e8954d85
changed how deleted blocks are managed to allow indexes to be reloaded
2018-04-14 19:26:58 -07:00
Jarek Kowalski
14e38b59bf
cleaned up how active block index is loaded
2018-04-14 19:26:58 -07:00
Jarek Kowalski
825f051515
added constraint that the block.ContentID prefix must be a single non-hex character or an empty string.
...
added unit tests for packIndex
2018-04-14 14:47:05 -07:00
Jarek Kowalski
924e0a4594
renamed index proto types to IndexV1
2018-04-08 18:45:22 -07:00
Jarek Kowalski
301f1d049a
added strongly-typed block.ContentID and renamed StorageBlock to ContentBlockID
2018-04-08 16:34:38 -07:00
Jarek Kowalski
4566213f5d
switched formatter to pass IV explicitly
...
extracted packIndex interface and implemented protoPackIndex to wrap all index data manipulation
2018-04-08 12:20:32 -07:00
Jarek Kowalski
d8201229d8
plumbed through ctx in storage.Storage APIs and all uses
2018-04-03 17:39:54 -07:00
Jarek Kowalski
c702e15489
improved logging of hash cache misses
2018-03-25 16:54:12 -07:00
Jarek Kowalski
2b49a5df7b
added per-directory summary
2018-03-24 12:22:21 -07:00
Jarek Kowalski
e997e07b0c
added support for summaries to JSON stream
2018-03-24 12:22:21 -07:00
Jarek Kowalski
a608c2f443
tweaked snapshot list
2018-03-24 12:22:21 -07:00
Jarek Kowalski
1c4e0e1e47
removed legacy oid format
2018-03-19 19:31:03 -07:00
Jarek Kowalski
2e61cc5688
code simplification, enabled gocyclo linter
2018-03-19 14:08:19 -07:00
