Jarek Kowalski
ec66349eeb
Fixed nasty bug where if the storage fails to write a block we would clobber pending block data in memory by encrypting it in place. This was happening a lot when backing up to cloud and closing the laptop whihle the upload was going on. Added regression test and a rudimentary mechanism for fault injection in storage.
2018-07-09 21:28:12 -07:00
Jarek Kowalski
c57531b751
switched Go logging library to github.com/op/go-logging which lets us override log levels per module
2018-07-09 21:28:12 -07:00
Jarek Kowalski
ba6419339a
refactored storage.ListBlocks() to be callback based which simplifies usage significantly
2018-06-23 13:47:43 -07:00
Jarek Kowalski
83036f1a0d
upload progress
2018-06-23 13:10:54 -07:00
Jarek Kowalski
2f1cf8129b
added skeleton of CLI actions for manipulating server sources
2018-06-18 18:02:36 -07:00
Jarek Kowalski
4faf3cd9d0
finalized CLI to manipulate policies
2018-06-16 12:17:58 -07:00
Jarek Kowalski
70ecebb769
added HTTP API to list policies
2018-06-16 08:46:56 -07:00
Jarek Kowalski
df3129682c
Added repo.Repository.Refresh{,Periodically}
2018-06-12 18:03:25 -07:00
Jarek Kowalski
22b0766570
API server, work in progress
2018-06-10 20:59:49 -07:00
Jarek Kowalski
b105cca96c
added 'kopia server' which runs simple daemon with simple HTTP API server, which will be used to build a web UI
2018-06-10 17:24:42 -07:00
Jarek Kowalski
6838d820c8
revamped object verification to be parallelized, added option to randomly verify a percentage of object contents
2018-06-09 22:15:36 -07:00
Jarek Kowalski
fc618b0c66
changed block.PhysicalBlockID to a string
2018-06-07 18:42:22 -07:00
Jarek Kowalski
6ee5c81e21
renamed 'pack block ID' to 'pack file' everywhere
2018-06-07 07:52:17 -07:00
Jarek Kowalski
438fa4a587
whenever two blocks in index have identical times, prefer the non-deleted one
2018-06-04 22:08:58 -07:00
Jarek Kowalski
5ba913380e
added support for soft-delete that preserves original pack ID/offset/length.
...
removed legacy soft-delete support without this information
2018-05-30 21:22:07 -07:00
Jarek Kowalski
e36d02c59b
emit per-dir incomplete reason
...
store entire rootEntry on snapshot nmanifest
format 'snapshot list' consistently for both root and nested items
2018-05-19 22:00:14 -07:00
Jarek Kowalski
37566b0a05
output per-directory summary when uploading
2018-05-19 07:31:06 -07:00
Jarek Kowalski
d6477377f9
changed how index compaction works:
...
instead of having time-based naming, block manager will perform occasional compaction at startup time and delete unwanted blocks
The protocol is safe from concurrency standpoint and provides eventual consistency.
We only delete blocks after creating compacted blocks.
We retry loading index until (any) consistent index is fetched (possibly from cache) and all underlying blocks are also available, not necessarily the latest ones.
TODO - we need to periodically snapshot block index contents, so that if we have a bug somewhere in compaction code, we have a way of restoring working indexes.
2018-05-15 20:10:50 -07:00
Jarek Kowalski
db0fd9ed28
added packindex.IsSubset()
2018-05-06 16:37:35 -07:00
Jarek Kowalski
3a02ef24e0
retire block.ContentID and replace with string
2018-05-05 16:16:48 -07:00
Jarek Kowalski
39aa586b80
fixed map storage data aliasing
2018-05-05 16:14:31 -07:00
Jarek Kowalski
1342fd910a
switched to using custom block index designed for efficient lookup without having to parse it
...
the block index stores tightly packed sorted block IDs
Typically a single entry 128-bit content ID uses 37 bytes of index storage.
2018-05-05 16:14:31 -07:00
Jarek Kowalski
250ceff6fd
removed support for pack index V1
2018-04-29 14:10:20 -07:00
Jarek Kowalski
5ede755524
added pack index v2
2018-04-29 13:44:12 -07:00
Jarek Kowalski
c4c74586c6
refactored block manager so that loaded block index is separate from pending block index
...
expanded block.Info to include additional details, removed packBlockInfo
removed repackaging support for now, since it's greatly complicating the implementation
2018-04-29 13:44:00 -07:00
Jarek Kowalski
ab4d2c7f8e
store format version with the index, not used yet
2018-04-21 10:36:53 -07:00
Jarek Kowalski
14e38b59bf
cleaned up how active block index is loaded
2018-04-14 19:26:58 -07:00
Jarek Kowalski
924e0a4594
renamed index proto types to IndexV1
2018-04-08 18:45:22 -07:00
Jarek Kowalski
d8201229d8
plumbed through ctx in storage.Storage APIs and all uses
2018-04-03 17:39:54 -07:00
Jarek Kowalski
2b49a5df7b
added per-directory summary
2018-03-24 12:22:21 -07:00
Jarek Kowalski
e997e07b0c
added support for summaries to JSON stream
2018-03-24 12:22:21 -07:00
Jarek Kowalski
1c4e0e1e47
removed legacy oid format
2018-03-19 19:31:03 -07:00
Jarek Kowalski
453bab3560
fixed some lint errors, mostly dead code and missing error checks
2018-03-19 12:26:28 -07:00
Jarek Kowalski
4657f16cc8
CLI improvement for automatic logging and log rotation
2018-03-16 22:21:57 -07:00
Jarek Kowalski
f4c9749246
additional logging for hash cache
2018-03-12 16:49:45 -07:00
Jarek Kowalski
0fa823976f
changed upload order to post-order - we first preprocess all subdirectories of a given directory, followed by files
2018-03-12 09:17:27 -07:00
Jarek Kowalski
b6ae16bfb6
changed how small files are stored to avoid creating very small data files by embedding them directly in the index packs
2018-02-16 18:18:47 -08:00
Jarek Kowalski
4564bc704f
breaking format change: removed support for groups in block manager, this produces bigger, tightly packed blocks
...
align pack blocks to 4096 bytes and insert random preamble
2018-02-10 13:07:10 -08:00
Jarek Kowalski
d7e1276a48
cleaned up storage.Storage API and documentation
2018-01-10 19:13:09 -08:00
Jarek Kowalski
b65a5c5689
added scrubber to remove passwords from 'repo status' output
2018-01-10 19:13:08 -08:00
Jarek Kowalski
7a97f22df4
switched to OS-specific locations for config and cache
2018-01-06 11:58:46 -08:00
Jarek Kowalski
7af93182a4
changed format of block index to be based on protobuf (using gogo), the deserialization is 10-12x faster than JSON, which matters for a large number of packs
2018-01-02 19:51:03 -08:00
Jarek Kowalski
10ca234633
switched to using zerolog for logging, by default log level=info, can be set to debug on commandline
2018-01-01 19:12:35 -08:00
Jarek Kowalski
30c11dc926
refactored block manager to support on-disk caching
2017-11-27 18:07:16 -08:00
Jarek Kowalski
de02941442
plumbed through key time in map storage
2017-11-23 16:43:34 -08:00
Jarek Kowalski
7c41f45e9f
extracted block formatting options into a struct under 'block'
2017-10-29 13:00:24 -07:00
Jarek Kowalski
49a806154a
code reorg
2017-10-29 12:03:49 -07:00
Jarek Kowalski
b6575a97ba
moved metadata manager to 'metadata' package and refactored initialization sequence
2017-10-29 10:29:22 -07:00
Jarek Kowalski
b5d0f2d120
more renames
2017-10-14 16:38:27 -07:00
Jarek Kowalski
c3f31d744f
removed stuttering
2017-10-14 16:38:22 -07:00
Jarek Kowalski