Jarek Kowalski
1342fd910a
switched to using custom block index designed for efficient lookup without having to parse it
...
the block index stores tightly packed sorted block IDs
Typically a single entry 128-bit content ID uses 37 bytes of index storage.
2018-05-05 16:14:31 -07:00
Jarek Kowalski
250ceff6fd
removed support for pack index V1
2018-04-29 14:10:20 -07:00
Jarek Kowalski
5ede755524
added pack index v2
2018-04-29 13:44:12 -07:00
Jarek Kowalski
4428743a5b
added --raw option to 'blockindex show'
2018-04-29 13:44:12 -07:00
Jarek Kowalski
726d33e0d5
added levelDB-based persistent block index
2018-04-29 13:44:12 -07:00
Jarek Kowalski
c4c74586c6
refactored block manager so that loaded block index is separate from pending block index
...
expanded block.Info to include additional details, removed packBlockInfo
removed repackaging support for now, since it's greatly complicating the implementation
2018-04-29 13:44:00 -07:00
Jarek Kowalski
f94fcea14b
moved block cache to a subdirectory
2018-04-23 19:56:01 -07:00
Jarek Kowalski
924e0a4594
renamed index proto types to IndexV1
2018-04-08 18:45:22 -07:00
Jarek Kowalski
301f1d049a
added strongly-typed block.ContentID and renamed StorageBlock to ContentBlockID
2018-04-08 16:34:38 -07:00
Jarek Kowalski
4566213f5d
switched formatter to pass IV explicitly
...
extracted packIndex interface and implemented protoPackIndex to wrap all index data manipulation
2018-04-08 12:20:32 -07:00
Jarek Kowalski
d8201229d8
plumbed through ctx in storage.Storage APIs and all uses
2018-04-03 17:39:54 -07:00
Jarek Kowalski
a608c2f443
tweaked snapshot list
2018-03-24 12:22:21 -07:00
Jarek Kowalski
1c4e0e1e47
removed legacy oid format
2018-03-19 19:31:03 -07:00
Jarek Kowalski
2e61cc5688
code simplification, enabled gocyclo linter
2018-03-19 14:08:19 -07:00
Jarek Kowalski
fa5eae501a
removed cleanup for now
2018-03-19 12:26:28 -07:00
Jarek Kowalski
453bab3560
fixed some lint errors, mostly dead code and missing error checks
2018-03-19 12:26:28 -07:00
Jarek Kowalski
4657f16cc8
CLI improvement for automatic logging and log rotation
2018-03-16 22:21:57 -07:00
Jarek Kowalski
47b1e09ba9
added default CLI commands
2018-03-15 09:39:39 -07:00
Jarek Kowalski
54aeea8ee6
fixed block list for inline blocks
2018-03-14 19:27:52 -07:00
Jarek Kowalski
70ed5647e4
added parallel uploading support
2018-03-14 18:22:50 -07:00
Jarek Kowalski
a50546de0e
fixed previous manifest selection for incremental snapshots
2018-03-12 16:50:01 -07:00
Jarek Kowalski
772ef9a286
added 'ls' alias for 'snapshot list'
2018-03-04 18:17:18 -08:00
Jarek Kowalski
92330ac247
refactored policy manager to use SourceInfo
2018-02-19 13:20:32 -08:00
Jarek Kowalski
16e44cb37f
renamed policy.Manager as snapshot.PolicyManager and policy.Policy as snapshot.Policy
2018-02-19 10:59:15 -08:00
Jarek Kowalski
d97f49285f
policy CLI and API cleanup
2018-02-19 10:53:29 -08:00
Jarek Kowalski
e710dcdff3
additional integration tests
2018-02-18 11:59:32 -08:00
Jarek Kowalski
b6ae16bfb6
changed how small files are stored to avoid creating very small data files by embedding them directly in the index packs
2018-02-16 18:18:47 -08:00
Jarek Kowalski
faa2625a5f
revamped CLI help to hide most commands
2018-02-15 19:49:51 -08:00
Jarek Kowalski
01ffec69fc
fixed output of 'storage list' for long block names
2018-02-11 08:12:20 -08:00
Jarek Kowalski
a350991762
reorganized CLI packages to be close to storage providers
2018-02-11 08:00:07 -08:00
Jarek Kowalski
06b4bec246
added 'storage sweep' subcommand which removes unreferenced storage blocks
2018-02-10 13:07:10 -08:00
Jarek Kowalski
4971671f3d
added pretty printing of block indexes
2018-02-10 13:07:10 -08:00
Jarek Kowalski
4564bc704f
breaking format change: removed support for groups in block manager, this produces bigger, tightly packed blocks
...
align pack blocks to 4096 bytes and insert random preamble
2018-02-10 13:07:10 -08:00
Jarek Kowalski
7e16471a21
added --flat option to filesystem and webdav providers
2018-02-10 13:06:34 -08:00
Jarek Kowalski
df35ebad73
added low-level storage viewing subcommands
2018-02-03 22:11:53 -08:00
Jarek Kowalski
7833626c4a
added error handling when unable to load block index
2018-01-24 11:44:26 +00:00
Jarek Kowalski
d7e1276a48
cleaned up storage.Storage API and documentation
2018-01-10 19:13:09 -08:00
Jarek Kowalski
090d97ba78
cli: reorganized all repo commands (connect/create/status) to top level
2018-01-10 19:13:09 -08:00
Jarek Kowalski
61abc8eee2
added support for S3 storage provider
2018-01-10 19:13:09 -08:00
Jarek Kowalski
b65a5c5689
added scrubber to remove passwords from 'repo status' output
2018-01-10 19:13:08 -08:00
Jarek Kowalski
bf4c0e694d
refactored CLI to use individual subcommands to connect to individual storage types, each with specialized flags and validation
2018-01-10 19:13:08 -08:00
Jarek Kowalski
7a97f22df4
switched to OS-specific locations for config and cache
2018-01-06 11:58:46 -08:00
Jarek Kowalski
f9f2c54993
added command to reconfigure caching on a repository that's already connected
2018-01-04 17:44:58 -08:00
Jarek Kowalski
7af93182a4
changed format of block index to be based on protobuf (using gogo), the deserialization is 10-12x faster than JSON, which matters for a large number of packs
2018-01-02 19:51:03 -08:00
Jarek Kowalski
10ca234633
switched to using zerolog for logging, by default log level=info, can be set to debug on commandline
2018-01-01 19:12:35 -08:00
Jarek Kowalski
98070064c8
index block list caching optimization - only load last 2 full compactions when scanning for active index blocks, this prevents the scan time from growing indefinitely
2018-01-01 17:11:41 -08:00
Jarek Kowalski
c288daef04
eradicated old metadata.Manager and replaced snapshot.Manager to use manifest.Manager instead
2017-11-27 19:15:02 -08:00
Jarek Kowalski
a644f2caa6
added 'manifest show' subcommand
2017-11-27 18:49:46 -08:00
Jarek Kowalski
30c11dc926
refactored block manager to support on-disk caching
2017-11-27 18:07:16 -08:00
Jarek Kowalski
a700508048
changed API for manifest manager, added filtering, sorting and better output to 'manifest ls'
2017-11-27 18:06:28 -08:00
