124 Commits

Author	SHA1	Message	Date
Jarek Kowalski	1c4e0e1e47	removed legacy oid format	2018-03-19 19:31:03 -07:00
Jarek Kowalski	2e61cc5688	code simplification, enabled gocyclo linter	2018-03-19 14:08:19 -07:00
Jarek Kowalski	fa5eae501a	removed cleanup for now	2018-03-19 12:26:28 -07:00
Jarek Kowalski	453bab3560	fixed some lint errors, mostly dead code and missing error checks	2018-03-19 12:26:28 -07:00
Jarek Kowalski	4657f16cc8	CLI improvement for automatic logging and log rotation	2018-03-16 22:21:57 -07:00
Jarek Kowalski	47b1e09ba9	added default CLI commands	2018-03-15 09:39:39 -07:00
Jarek Kowalski	54aeea8ee6	fixed block list for inline blocks	2018-03-14 19:27:52 -07:00
Jarek Kowalski	70ed5647e4	added parallel uploading support	2018-03-14 18:22:50 -07:00
Jarek Kowalski	a50546de0e	fixed previous manifest selection for incremental snapshots	2018-03-12 16:50:01 -07:00
Jarek Kowalski	772ef9a286	added 'ls' alias for 'snapshot list'	2018-03-04 18:17:18 -08:00
Jarek Kowalski	92330ac247	refactored policy manager to use SourceInfo	2018-02-19 13:20:32 -08:00
Jarek Kowalski	16e44cb37f	renamed policy.Manager as snapshot.PolicyManager and policy.Policy as snapshot.Policy	2018-02-19 10:59:15 -08:00
Jarek Kowalski	d97f49285f	policy CLI and API cleanup	2018-02-19 10:53:29 -08:00
Jarek Kowalski	e710dcdff3	additional integration tests	2018-02-18 11:59:32 -08:00
Jarek Kowalski	b6ae16bfb6	changed how small files are stored to avoid creating very small data files by embedding them directly in the index packs	2018-02-16 18:18:47 -08:00
Jarek Kowalski	faa2625a5f	revamped CLI help to hide most commands	2018-02-15 19:49:51 -08:00
Jarek Kowalski	01ffec69fc	fixed output of 'storage list' for long block names	2018-02-11 08:12:20 -08:00
Jarek Kowalski	a350991762	reorganized CLI packages to be close to storage providers	2018-02-11 08:00:07 -08:00
Jarek Kowalski	06b4bec246	added 'storage sweep' subcommand which removes unreferenced storage blocks	2018-02-10 13:07:10 -08:00
Jarek Kowalski	4971671f3d	added pretty printing of block indexes	2018-02-10 13:07:10 -08:00
Jarek Kowalski	4564bc704f	breaking format change: removed support for groups in block manager, this produces bigger, tightly packed blocks ... align pack blocks to 4096 bytes and insert random preamble	2018-02-10 13:07:10 -08:00
Jarek Kowalski	7e16471a21	added --flat option to filesystem and webdav providers	2018-02-10 13:06:34 -08:00
Jarek Kowalski	df35ebad73	added low-level storage viewing subcommands	2018-02-03 22:11:53 -08:00
Jarek Kowalski	7833626c4a	added error handling when unable to load block index	2018-01-24 11:44:26 +00:00
Jarek Kowalski	d7e1276a48	cleaned up storage.Storage API and documentation	2018-01-10 19:13:09 -08:00
Jarek Kowalski	090d97ba78	cli: reorganized all repo commands (connect/create/status) to top level	2018-01-10 19:13:09 -08:00
Jarek Kowalski	61abc8eee2	added support for S3 storage provider	2018-01-10 19:13:09 -08:00
Jarek Kowalski	b65a5c5689	added scrubber to remove passwords from 'repo status' output	2018-01-10 19:13:08 -08:00
Jarek Kowalski	bf4c0e694d	refactored CLI to use individual subcommands to connect to individual storage types, each with specialized flags and validation	2018-01-10 19:13:08 -08:00
Jarek Kowalski	7a97f22df4	switched to OS-specific locations for config and cache	2018-01-06 11:58:46 -08:00
Jarek Kowalski	f9f2c54993	added command to reconfigure caching on a repository that's already connected	2018-01-04 17:44:58 -08:00
Jarek Kowalski	7af93182a4	changed format of block index to be based on protobuf (using gogo), the deserialization is 10-12x faster than JSON, which matters for a large number of packs	2018-01-02 19:51:03 -08:00
Jarek Kowalski	10ca234633	switched to using zerolog for logging, by default log level=info, can be set to debug on commandline	2018-01-01 19:12:35 -08:00
Jarek Kowalski	98070064c8	index block list caching optimization - only load last 2 full compactions when scanning for active index blocks, this prevents the scan time from growing indefinitely	2018-01-01 17:11:41 -08:00
Jarek Kowalski	c288daef04	eradicated old metadata.Manager and replaced snapshot.Manager to use manifest.Manager instead	2017-11-27 19:15:02 -08:00
Jarek Kowalski	a644f2caa6	added 'manifest show' subcommand	2017-11-27 18:49:46 -08:00
Jarek Kowalski	30c11dc926	refactored block manager to support on-disk caching	2017-11-27 18:07:16 -08:00
Jarek Kowalski	a700508048	changed API for manifest manager, added filtering, sorting and better output to 'manifest ls'	2017-11-27 18:06:28 -08:00
Jarek Kowalski	eebb04c0cb	added 'manifest ls' subcommand	2017-11-27 18:05:44 -08:00
Jarek Kowalski	6e83ad0b9b	moved policies to separate package ... refactored policy management into policy.Manager that uses manifest.Manager internally	2017-11-27 18:05:32 -08:00
Jarek Kowalski	d3b854ef8e	optimized format for pack index, added compacted block with suffix -zTIMESTAMP where TIMESTAMP is base16-encoded unix nanoseconds of all blocks that this block supersedes, this allows much more efficient loading of blocks without having to delete anything	2017-11-23 15:55:56 -08:00
Jarek Kowalski	40a367f79e	added 'block rm' subcommand	2017-11-12 22:43:21 -08:00
Jarek Kowalski	0f6c569435	refactored repository status	2017-10-29 10:35:06 -07:00
Jarek Kowalski	b6575a97ba	moved metadata manager to 'metadata' package and refactored initialization sequence	2017-10-29 10:29:22 -07:00
Jarek Kowalski	3498f78d91	dead code cleanup	2017-10-28 22:50:11 -07:00
Jarek Kowalski	e76f1f8a72	refactored various show command into one	2017-10-24 21:59:40 -07:00
Jarek Kowalski	a48d54f00b	added 'block show' similar to 'object show', but operating on virtual blocks	2017-10-24 21:15:22 -07:00
Jarek Kowalski	b4e6a70e09	renamed 'repo optimize' to 'block reindex', added repacking method and unit tests	2017-10-24 21:05:59 -07:00
Jarek Kowalski	2aa44eb454	added listing group blocks and summary to 'block list'	2017-10-17 19:24:16 -07:00
Jarek Kowalski	a45f557c4b	changed object.ManagerOptions into a struct	2017-10-14 16:38:27 -07:00