Commit Graph

18 Commits

Author SHA1 Message Date
Jarek Kowalski
c57531b751 switched Go logging library to github.com/op/go-logging which lets us override log levels per module 2018-07-09 21:28:12 -07:00
Jarek Kowalski
ba6419339a refactored storage.ListBlocks() to be callback based which simplifies usage significantly 2018-06-23 13:47:43 -07:00
Jarek Kowalski
83036f1a0d upload progress 2018-06-23 13:10:54 -07:00
Jarek Kowalski
e9c11ea106 refactored block cache and list cache 2018-06-10 16:09:47 -07:00
Jarek Kowalski
9a17179615 reorganized local cache to have 256 shards instead of tens of thousands, this improves scan speed and startup time with lots of cached items 2018-06-10 08:15:07 -07:00
Jarek Kowalski
fc618b0c66 changed block.PhysicalBlockID to a string 2018-06-07 18:42:22 -07:00
Jarek Kowalski
d6477377f9 changed how index compaction works:
instead of having time-based naming, block manager will perform occasional compaction at startup time and delete unwanted blocks

The protocol is safe from concurrency standpoint and provides eventual consistency.
We only delete blocks after creating compacted blocks.
We retry loading index until (any) consistent index is fetched (possibly from cache) and all underlying blocks are also available, not necessarily the latest ones.

TODO - we need to periodically snapshot block index contents, so that if we have a bug somewhere in compaction code, we have a way of restoring working indexes.
2018-05-15 20:10:50 -07:00
Jarek Kowalski
f94fcea14b moved block cache to a subdirectory 2018-04-23 19:56:01 -07:00
Jarek Kowalski
14e38b59bf cleaned up how active block index is loaded 2018-04-14 19:26:58 -07:00
Jarek Kowalski
301f1d049a added strongly-typed block.ContentID and renamed StorageBlock to ContentBlockID 2018-04-08 16:34:38 -07:00
Jarek Kowalski
4566213f5d switched formatter to pass IV explicitly
extracted packIndex interface and implemented protoPackIndex to wrap all index data manipulation
2018-04-08 12:20:32 -07:00
Jarek Kowalski
d8201229d8 plumbed through ctx in storage.Storage APIs and all uses 2018-04-03 17:39:54 -07:00
Jarek Kowalski
453bab3560 fixed some lint errors, mostly dead code and missing error checks 2018-03-19 12:26:28 -07:00
Jarek Kowalski
6707d01a22 switched local caching to use storage.Storage 2018-03-04 16:58:20 -08:00
Jarek Kowalski
a9f8741e8e changed local caching to be based on virtual blocks instead of physical blocks 2018-03-04 12:13:57 -08:00
Jarek Kowalski
98070064c8 index block list caching optimization - only load last 2 full compactions when scanning for active index blocks, this prevents the scan time from growing indefinitely 2018-01-01 17:11:41 -08:00
Jarek Kowalski
596c2b122a disk cache: remove list after a write
fix close
2017-12-31 16:15:58 -08:00
Jarek Kowalski
30c11dc926 refactored block manager to support on-disk caching 2017-11-27 18:07:16 -08:00