110 Commits

Author	SHA1	Message	Date
Jarek Kowalski	b6ae16bfb6	changed how small files are stored to avoid creating very small data files by embedding them directly in the index packs	2018-02-16 18:18:47 -08:00
Jarek Kowalski	faa2625a5f	revamped CLI help to hide most commands	2018-02-15 19:49:51 -08:00
Jarek Kowalski	01ffec69fc	fixed output of 'storage list' for long block names	2018-02-11 08:12:20 -08:00
Jarek Kowalski	a350991762	reorganized CLI packages to be close to storage providers	2018-02-11 08:00:07 -08:00
Jarek Kowalski	06b4bec246	added 'storage sweep' subcommand which removes unreferenced storage blocks	2018-02-10 13:07:10 -08:00
Jarek Kowalski	4971671f3d	added pretty printing of block indexes	2018-02-10 13:07:10 -08:00
Jarek Kowalski	4564bc704f	breaking format change: removed support for groups in block manager, this produces bigger, tightly packed blocks ... align pack blocks to 4096 bytes and insert random preamble	2018-02-10 13:07:10 -08:00
Jarek Kowalski	7e16471a21	added --flat option to filesystem and webdav providers	2018-02-10 13:06:34 -08:00
Jarek Kowalski	df35ebad73	added low-level storage viewing subcommands	2018-02-03 22:11:53 -08:00
Jarek Kowalski	7833626c4a	added error handling when unable to load block index	2018-01-24 11:44:26 +00:00
Jarek Kowalski	d7e1276a48	cleaned up storage.Storage API and documentation	2018-01-10 19:13:09 -08:00
Jarek Kowalski	090d97ba78	cli: reorganized all repo commands (connect/create/status) to top level	2018-01-10 19:13:09 -08:00
Jarek Kowalski	61abc8eee2	added support for S3 storage provider	2018-01-10 19:13:09 -08:00
Jarek Kowalski	b65a5c5689	added scrubber to remove passwords from 'repo status' output	2018-01-10 19:13:08 -08:00
Jarek Kowalski	bf4c0e694d	refactored CLI to use individual subcommands to connect to individual storage types, each with specialized flags and validation	2018-01-10 19:13:08 -08:00
Jarek Kowalski	7a97f22df4	switched to OS-specific locations for config and cache	2018-01-06 11:58:46 -08:00
Jarek Kowalski	f9f2c54993	added command to reconfigure caching on a repository that's already connected	2018-01-04 17:44:58 -08:00
Jarek Kowalski	7af93182a4	changed format of block index to be based on protobuf (using gogo), the deserialization is 10-12x faster than JSON, which matters for a large number of packs	2018-01-02 19:51:03 -08:00
Jarek Kowalski	10ca234633	switched to using zerolog for logging, by default log level=info, can be set to debug on commandline	2018-01-01 19:12:35 -08:00
Jarek Kowalski	98070064c8	index block list caching optimization - only load last 2 full compactions when scanning for active index blocks, this prevents the scan time from growing indefinitely	2018-01-01 17:11:41 -08:00
Jarek Kowalski	c288daef04	eradicated old metadata.Manager and replaced snapshot.Manager to use manifest.Manager instead	2017-11-27 19:15:02 -08:00
Jarek Kowalski	a644f2caa6	added 'manifest show' subcommand	2017-11-27 18:49:46 -08:00
Jarek Kowalski	30c11dc926	refactored block manager to support on-disk caching	2017-11-27 18:07:16 -08:00
Jarek Kowalski	a700508048	changed API for manifest manager, added filtering, sorting and better output to 'manifest ls'	2017-11-27 18:06:28 -08:00
Jarek Kowalski	eebb04c0cb	added 'manifest ls' subcommand	2017-11-27 18:05:44 -08:00
Jarek Kowalski	6e83ad0b9b	moved policies to separate package ... refactored policy management into policy.Manager that uses manifest.Manager internally	2017-11-27 18:05:32 -08:00
Jarek Kowalski	d3b854ef8e	optimized format for pack index, added compacted block with suffix -zTIMESTAMP where TIMESTAMP is base16-encoded unix nanoseconds of all blocks that this block supersedes, this allows much more efficient loading of blocks without having to delete anything	2017-11-23 15:55:56 -08:00
Jarek Kowalski	40a367f79e	added 'block rm' subcommand	2017-11-12 22:43:21 -08:00
Jarek Kowalski	0f6c569435	refactored repository status	2017-10-29 10:35:06 -07:00
Jarek Kowalski	b6575a97ba	moved metadata manager to 'metadata' package and refactored initialization sequence	2017-10-29 10:29:22 -07:00
Jarek Kowalski	3498f78d91	dead code cleanup	2017-10-28 22:50:11 -07:00
Jarek Kowalski	e76f1f8a72	refactored various show command into one	2017-10-24 21:59:40 -07:00
Jarek Kowalski	a48d54f00b	added 'block show' similar to 'object show', but operating on virtual blocks	2017-10-24 21:15:22 -07:00
Jarek Kowalski	b4e6a70e09	renamed 'repo optimize' to 'block reindex', added repacking method and unit tests	2017-10-24 21:05:59 -07:00
Jarek Kowalski	2aa44eb454	added listing group blocks and summary to 'block list'	2017-10-17 19:24:16 -07:00
Jarek Kowalski	a45f557c4b	changed object.ManagerOptions into a struct	2017-10-14 16:38:27 -07:00
Jarek Kowalski	a715522c34	improved block stats output	2017-10-14 16:38:27 -07:00
Jarek Kowalski	b5d0f2d120	more renames	2017-10-14 16:38:27 -07:00
Jarek Kowalski	0e48800f1a	removed ObjectIDSection support, no longer needed because of packs	2017-10-14 16:38:22 -07:00
Jarek Kowalski	c3f31d744f	removed stuttering	2017-10-14 16:38:22 -07:00
Jarek Kowalski	31e5bce1cb	moved object management to kopia/kopia/object	2017-10-14 16:38:22 -07:00
Jarek Kowalski	f33ae14573	renamed 'blob.Storage' to 'storage.Storage'	2017-10-14 16:38:09 -07:00
Jarek Kowalski	cc3e590136	moved block manager and formatter to a separate package	2017-10-09 06:42:30 -07:00
Jarek Kowalski	a6a37b709a	added pack info output to 'block list'	2017-10-08 20:49:25 -07:00
Jarek Kowalski	ce5d9b2247	output pack group when listing blocks	2017-10-08 17:40:37 -07:00
Jarek Kowalski	444b6306ca	refactored object manager to use block manager, exposed on repository ... added CLI commands to manipulate blocks	2017-10-08 12:44:53 -07:00
Jarek Kowalski	10b9b4b451	fixed repo optimize to remove unused logical blocks	2017-10-02 22:28:00 -07:00
Jarek Kowalski	a412e45f1f	cleanup: bump worker count to 32	2017-10-01 19:15:32 -07:00
Jarek Kowalski	f96dfe471e	support for array pack indexes instead of maps	2017-10-01 16:07:55 -07:00
Jarek Kowalski	b2d9e07b75	fix 'object list' output.	2017-09-24 12:38:21 -07:00