Jarek Kowalski
ec66349eeb
Fixed nasty bug where if the storage fails to write a block we would clobber pending block data in memory by encrypting it in place. This was happening a lot when backing up to cloud and closing the laptop whihle the upload was going on. Added regression test and a rudimentary mechanism for fault injection in storage.
2018-07-09 21:28:12 -07:00
Jarek Kowalski
801b6eb5eb
cleaned up block manager to make enforcement of invariants easier
2018-07-09 21:28:12 -07:00
Jarek Kowalski
f44234ef3a
improved logging
2018-07-09 21:28:12 -07:00
Jarek Kowalski
2cd426eba0
additional logging for block manager
2018-07-09 21:28:12 -07:00
Jarek Kowalski
c57531b751
switched Go logging library to github.com/op/go-logging which lets us override log levels per module
2018-07-09 21:28:12 -07:00
Jarek Kowalski
32b83be0b4
additional logging for block manager
2018-07-09 21:28:12 -07:00
Jarek Kowalski
bd7fa874b0
write list cache file atomically
2018-07-02 20:38:48 -07:00
Jarek Kowalski
fe52c87a58
better error reporting on data corruption
2018-07-01 13:17:23 -07:00
Jarek Kowalski
75da8f3f31
removed dead code
2018-06-24 15:04:58 -07:00
Jarek Kowalski
ba6419339a
refactored storage.ListBlocks() to be callback based which simplifies usage significantly
2018-06-23 13:47:43 -07:00
Jarek Kowalski
83036f1a0d
upload progress
2018-06-23 13:10:54 -07:00
Jarek Kowalski
91c51f4807
refactored 'block gc' subcommand
2018-06-23 13:05:40 -07:00
Jarek Kowalski
c7d898a674
fixed block manager lock which triggered asserts during parallel read/write
2018-06-23 12:30:19 -07:00
Jarek Kowalski
173c5613c1
fixed aliasing of read buffers in block manager
2018-06-18 19:16:13 -07:00
Jarek Kowalski
85b4bb7cd9
better fix for aliasing issue
2018-06-18 18:23:37 -07:00
Jarek Kowalski
23318a461d
fixed rare data corruption bug due to buffer aliasing
2018-06-18 18:02:36 -07:00
Jarek Kowalski
a6f2e8c3f3
added option to disable block manager flush to ensure that a sequence of writes/deletes is atomically observable
2018-06-14 17:11:00 -07:00
Jarek Kowalski
6b79ab79d0
fixed --no-list-caching
2018-06-13 22:37:45 -07:00
Jarek Kowalski
e21a5ce650
refactored committed block index to separate cache from the lookup
2018-06-12 20:22:37 -07:00
Jarek Kowalski
df3129682c
Added repo.Repository.Refresh{,Periodically}
2018-06-12 18:03:25 -07:00
Jarek Kowalski
4313d5f6ab
added block.Manager.Refresh method
2018-06-10 20:59:49 -07:00
Jarek Kowalski
e9c11ea106
refactored block cache and list cache
2018-06-10 16:09:47 -07:00
Jarek Kowalski
846a46b879
changed how caching is controlled in block manager by using context instead of options
2018-06-10 10:20:52 -07:00
Jarek Kowalski
9a17179615
reorganized local cache to have 256 shards instead of tens of thousands, this improves scan speed and startup time with lots of cached items
2018-06-10 08:15:07 -07:00
Jarek Kowalski
19dcda1f47
removed lock while loading a block that made block manager unnecessarily single-threaded
2018-06-09 18:36:59 -07:00
Jarek Kowalski
e5e5577cfd
fixed linter warnings
2018-06-09 14:06:43 -07:00
Jarek Kowalski
fc618b0c66
changed block.PhysicalBlockID to a string
2018-06-07 18:42:22 -07:00
Jarek Kowalski
6ee5c81e21
renamed 'pack block ID' to 'pack file' everywhere
2018-06-07 07:52:17 -07:00
Jarek Kowalski
985980925f
removed support for v0 block format
2018-06-05 20:17:21 -07:00
Jarek Kowalski
416475a092
added RewriteBlock() support which can rewrite either deleted or non-deleted block to a new pack
...
Added 'block gc' and 'block rewrite' command-line subcommands.
2018-06-04 22:09:10 -07:00
Jarek Kowalski
f6dcd3c155
remove old unused cached index blocks
2018-06-02 10:51:32 -07:00
Jarek Kowalski
5ba913380e
added support for soft-delete that preserves original pack ID/offset/length.
...
removed legacy soft-delete support without this information
2018-05-30 21:22:07 -07:00
Jarek Kowalski
80a3bdfd4a
fixed race condition when multiple kopia instances are downloading indexes in parallel
2018-05-20 10:45:13 -07:00
Jarek Kowalski
490fa6130b
changed pack naming scheme to be p{random-128-bit}
2018-05-17 23:00:24 -07:00
Jarek Kowalski
d6477377f9
changed how index compaction works:
...
instead of having time-based naming, block manager will perform occasional compaction at startup time and delete unwanted blocks
The protocol is safe from concurrency standpoint and provides eventual consistency.
We only delete blocks after creating compacted blocks.
We retry loading index until (any) consistent index is fetched (possibly from cache) and all underlying blocks are also available, not necessarily the latest ones.
TODO - we need to periodically snapshot block index contents, so that if we have a bug somewhere in compaction code, we have a way of restoring working indexes.
2018-05-15 20:10:50 -07:00
Jarek Kowalski
9c57f22c0f
simplified committed index block interface, speed up block block manager test by a lot
2018-05-05 16:16:48 -07:00
Jarek Kowalski
62ce401cd6
exposed block.Manager.ListBlocks() which only returns block IDs instead of full info
2018-05-05 16:16:48 -07:00
Jarek Kowalski
3a02ef24e0
retire block.ContentID and replace with string
2018-05-05 16:16:48 -07:00
Jarek Kowalski
1342fd910a
switched to using custom block index designed for efficient lookup without having to parse it
...
the block index stores tightly packed sorted block IDs
Typically a single entry 128-bit content ID uses 37 bytes of index storage.
2018-05-05 16:14:31 -07:00
Jarek Kowalski
250ceff6fd
removed support for pack index V1
2018-04-29 14:10:20 -07:00
Jarek Kowalski
3464059031
cleanup
2018-04-29 13:53:35 -07:00
Jarek Kowalski
5ede755524
added pack index v2
2018-04-29 13:44:12 -07:00
Jarek Kowalski
5912c628bb
added copyPackIndex()
2018-04-29 13:44:12 -07:00
Jarek Kowalski
726d33e0d5
added levelDB-based persistent block index
2018-04-29 13:44:12 -07:00
Jarek Kowalski
c4c74586c6
refactored block manager so that loaded block index is separate from pending block index
...
expanded block.Info to include additional details, removed packBlockInfo
removed repackaging support for now, since it's greatly complicating the implementation
2018-04-29 13:44:00 -07:00
Jarek Kowalski
f94fcea14b
moved block cache to a subdirectory
2018-04-23 19:56:01 -07:00
Jarek Kowalski
df7a9e217a
added support for 'v1' pack format that stores each packed block individually encrypted as opposed to encrypting the entire pack
2018-04-23 17:51:00 -07:00
Jarek Kowalski
0f63f04c14
changed how pack block are staged to first add them as inline, then convert to a block as necessary and not the opposite
2018-04-21 12:56:18 -07:00
Jarek Kowalski
ab4d2c7f8e
store format version with the index, not used yet
2018-04-21 10:36:53 -07:00
Jarek Kowalski
52bcb6223f
when loading active index, load 10 minutes extra to correct for clock skew and in-flight uploads
2018-04-21 10:36:53 -07:00
Jarek Kowalski