Commit Graph

10 Commits

Author SHA1 Message Date
Jarek Kowalski
fc618b0c66 changed block.PhysicalBlockID to a string 2018-06-07 18:42:22 -07:00
Jarek Kowalski
d6477377f9 changed how index compaction works:
instead of having time-based naming, block manager will perform occasional compaction at startup time and delete unwanted blocks

The protocol is safe from concurrency standpoint and provides eventual consistency.
We only delete blocks after creating compacted blocks.
We retry loading index until (any) consistent index is fetched (possibly from cache) and all underlying blocks are also available, not necessarily the latest ones.

TODO - we need to periodically snapshot block index contents, so that if we have a bug somewhere in compaction code, we have a way of restoring working indexes.
2018-05-15 20:10:50 -07:00
Jarek Kowalski
4566213f5d switched formatter to pass IV explicitly
extracted packIndex interface and implemented protoPackIndex to wrap all index data manipulation
2018-04-08 12:20:32 -07:00
Jarek Kowalski
d8201229d8 plumbed through ctx in storage.Storage APIs and all uses 2018-04-03 17:39:54 -07:00
Jarek Kowalski
453bab3560 fixed some lint errors, mostly dead code and missing error checks 2018-03-19 12:26:28 -07:00
Jarek Kowalski
47b1e09ba9 added default CLI commands 2018-03-15 09:39:39 -07:00
Jarek Kowalski
e710dcdff3 additional integration tests 2018-02-18 11:59:32 -08:00
Jarek Kowalski
7af93182a4 changed format of block index to be based on protobuf (using gogo), the deserialization is 10-12x faster than JSON, which matters for a large number of packs 2018-01-02 19:51:03 -08:00
Jarek Kowalski
d3b854ef8e optimized format for pack index, added compacted block with suffix -zTIMESTAMP where TIMESTAMP is base16-encoded unix nanoseconds of all blocks that this block supersedes, this allows much more efficient loading of blocks without having to delete anything 2017-11-23 15:55:56 -08:00
Jarek Kowalski
e76f1f8a72 refactored various show command into one 2017-10-24 21:59:40 -07:00