Jarek Kowalski
c57531b751
switched Go logging library to github.com/op/go-logging which lets us override log levels per module
2018-07-09 21:28:12 -07:00
Jarek Kowalski
ba6419339a
refactored storage.ListBlocks() to be callback based which simplifies usage significantly
2018-06-23 13:47:43 -07:00
Jarek Kowalski
83036f1a0d
upload progress
2018-06-23 13:10:54 -07:00
Jarek Kowalski
e9c11ea106
refactored block cache and list cache
2018-06-10 16:09:47 -07:00
Jarek Kowalski
9a17179615
reorganized local cache to have 256 shards instead of tens of thousands, this improves scan speed and startup time with lots of cached items
2018-06-10 08:15:07 -07:00
Jarek Kowalski
fc618b0c66
changed block.PhysicalBlockID to a string
2018-06-07 18:42:22 -07:00
Jarek Kowalski
d6477377f9
changed how index compaction works:
...
instead of having time-based naming, block manager will perform occasional compaction at startup time and delete unwanted blocks
The protocol is safe from concurrency standpoint and provides eventual consistency.
We only delete blocks after creating compacted blocks.
We retry loading index until (any) consistent index is fetched (possibly from cache) and all underlying blocks are also available, not necessarily the latest ones.
TODO - we need to periodically snapshot block index contents, so that if we have a bug somewhere in compaction code, we have a way of restoring working indexes.
2018-05-15 20:10:50 -07:00
Jarek Kowalski
f94fcea14b
moved block cache to a subdirectory
2018-04-23 19:56:01 -07:00
Jarek Kowalski
14e38b59bf
cleaned up how active block index is loaded
2018-04-14 19:26:58 -07:00
Jarek Kowalski
301f1d049a
added strongly-typed block.ContentID and renamed StorageBlock to ContentBlockID
2018-04-08 16:34:38 -07:00
Jarek Kowalski
4566213f5d
switched formatter to pass IV explicitly
...
extracted packIndex interface and implemented protoPackIndex to wrap all index data manipulation
2018-04-08 12:20:32 -07:00
Jarek Kowalski
d8201229d8
plumbed through ctx in storage.Storage APIs and all uses
2018-04-03 17:39:54 -07:00
Jarek Kowalski
453bab3560
fixed some lint errors, mostly dead code and missing error checks
2018-03-19 12:26:28 -07:00
Jarek Kowalski
6707d01a22
switched local caching to use storage.Storage
2018-03-04 16:58:20 -08:00
Jarek Kowalski
a9f8741e8e
changed local caching to be based on virtual blocks instead of physical blocks
2018-03-04 12:13:57 -08:00
Jarek Kowalski
98070064c8
index block list caching optimization - only load last 2 full compactions when scanning for active index blocks, this prevents the scan time from growing indefinitely
2018-01-01 17:11:41 -08:00
Jarek Kowalski
596c2b122a
disk cache: remove list after a write
...
fix close
2017-12-31 16:15:58 -08:00
Jarek Kowalski
30c11dc926
refactored block manager to support on-disk caching
2017-11-27 18:07:16 -08:00