111 Commits

Author	SHA1	Message	Date
Jarek Kowalski	e710dcdff3	additional integration tests	2018-02-18 11:59:32 -08:00
Jarek Kowalski	b6ae16bfb6	changed how small files are stored to avoid creating very small data files by embedding them directly in the index packs	2018-02-16 18:18:47 -08:00
Jarek Kowalski	faa2625a5f	revamped CLI help to hide most commands	2018-02-15 19:49:51 -08:00
Jarek Kowalski	01ffec69fc	fixed output of 'storage list' for long block names	2018-02-11 08:12:20 -08:00
Jarek Kowalski	a350991762	reorganized CLI packages to be close to storage providers	2018-02-11 08:00:07 -08:00
Jarek Kowalski	06b4bec246	added 'storage sweep' subcommand which removes unreferenced storage blocks	2018-02-10 13:07:10 -08:00
Jarek Kowalski	4971671f3d	added pretty printing of block indexes	2018-02-10 13:07:10 -08:00
Jarek Kowalski	4564bc704f	breaking format change: removed support for groups in block manager, this produces bigger, tightly packed blocks ... align pack blocks to 4096 bytes and insert random preamble	2018-02-10 13:07:10 -08:00
Jarek Kowalski	7e16471a21	added --flat option to filesystem and webdav providers	2018-02-10 13:06:34 -08:00
Jarek Kowalski	df35ebad73	added low-level storage viewing subcommands	2018-02-03 22:11:53 -08:00
Jarek Kowalski	7833626c4a	added error handling when unable to load block index	2018-01-24 11:44:26 +00:00
Jarek Kowalski	d7e1276a48	cleaned up storage.Storage API and documentation	2018-01-10 19:13:09 -08:00
Jarek Kowalski	090d97ba78	cli: reorganized all repo commands (connect/create/status) to top level	2018-01-10 19:13:09 -08:00
Jarek Kowalski	61abc8eee2	added support for S3 storage provider	2018-01-10 19:13:09 -08:00
Jarek Kowalski	b65a5c5689	added scrubber to remove passwords from 'repo status' output	2018-01-10 19:13:08 -08:00
Jarek Kowalski	bf4c0e694d	refactored CLI to use individual subcommands to connect to individual storage types, each with specialized flags and validation	2018-01-10 19:13:08 -08:00
Jarek Kowalski	7a97f22df4	switched to OS-specific locations for config and cache	2018-01-06 11:58:46 -08:00
Jarek Kowalski	f9f2c54993	added command to reconfigure caching on a repository that's already connected	2018-01-04 17:44:58 -08:00
Jarek Kowalski	7af93182a4	changed format of block index to be based on protobuf (using gogo), the deserialization is 10-12x faster than JSON, which matters for a large number of packs	2018-01-02 19:51:03 -08:00
Jarek Kowalski	10ca234633	switched to using zerolog for logging, by default log level=info, can be set to debug on commandline	2018-01-01 19:12:35 -08:00
Jarek Kowalski	98070064c8	index block list caching optimization - only load last 2 full compactions when scanning for active index blocks, this prevents the scan time from growing indefinitely	2018-01-01 17:11:41 -08:00
Jarek Kowalski	c288daef04	eradicated old metadata.Manager and replaced snapshot.Manager to use manifest.Manager instead	2017-11-27 19:15:02 -08:00
Jarek Kowalski	a644f2caa6	added 'manifest show' subcommand	2017-11-27 18:49:46 -08:00
Jarek Kowalski	30c11dc926	refactored block manager to support on-disk caching	2017-11-27 18:07:16 -08:00
Jarek Kowalski	a700508048	changed API for manifest manager, added filtering, sorting and better output to 'manifest ls'	2017-11-27 18:06:28 -08:00
Jarek Kowalski	eebb04c0cb	added 'manifest ls' subcommand	2017-11-27 18:05:44 -08:00
Jarek Kowalski	6e83ad0b9b	moved policies to separate package ... refactored policy management into policy.Manager that uses manifest.Manager internally	2017-11-27 18:05:32 -08:00
Jarek Kowalski	d3b854ef8e	optimized format for pack index, added compacted block with suffix -zTIMESTAMP where TIMESTAMP is base16-encoded unix nanoseconds of all blocks that this block supersedes, this allows much more efficient loading of blocks without having to delete anything	2017-11-23 15:55:56 -08:00
Jarek Kowalski	40a367f79e	added 'block rm' subcommand	2017-11-12 22:43:21 -08:00
Jarek Kowalski	0f6c569435	refactored repository status	2017-10-29 10:35:06 -07:00
Jarek Kowalski	b6575a97ba	moved metadata manager to 'metadata' package and refactored initialization sequence	2017-10-29 10:29:22 -07:00
Jarek Kowalski	3498f78d91	dead code cleanup	2017-10-28 22:50:11 -07:00
Jarek Kowalski	e76f1f8a72	refactored various show command into one	2017-10-24 21:59:40 -07:00
Jarek Kowalski	a48d54f00b	added 'block show' similar to 'object show', but operating on virtual blocks	2017-10-24 21:15:22 -07:00
Jarek Kowalski	b4e6a70e09	renamed 'repo optimize' to 'block reindex', added repacking method and unit tests	2017-10-24 21:05:59 -07:00
Jarek Kowalski	2aa44eb454	added listing group blocks and summary to 'block list'	2017-10-17 19:24:16 -07:00
Jarek Kowalski	a45f557c4b	changed object.ManagerOptions into a struct	2017-10-14 16:38:27 -07:00
Jarek Kowalski	a715522c34	improved block stats output	2017-10-14 16:38:27 -07:00
Jarek Kowalski	b5d0f2d120	more renames	2017-10-14 16:38:27 -07:00
Jarek Kowalski	0e48800f1a	removed ObjectIDSection support, no longer needed because of packs	2017-10-14 16:38:22 -07:00
Jarek Kowalski	c3f31d744f	removed stuttering	2017-10-14 16:38:22 -07:00
Jarek Kowalski	31e5bce1cb	moved object management to kopia/kopia/object	2017-10-14 16:38:22 -07:00
Jarek Kowalski	f33ae14573	renamed 'blob.Storage' to 'storage.Storage'	2017-10-14 16:38:09 -07:00
Jarek Kowalski	cc3e590136	moved block manager and formatter to a separate package	2017-10-09 06:42:30 -07:00
Jarek Kowalski	a6a37b709a	added pack info output to 'block list'	2017-10-08 20:49:25 -07:00
Jarek Kowalski	ce5d9b2247	output pack group when listing blocks	2017-10-08 17:40:37 -07:00
Jarek Kowalski	444b6306ca	refactored object manager to use block manager, exposed on repository ... added CLI commands to manipulate blocks	2017-10-08 12:44:53 -07:00
Jarek Kowalski	10b9b4b451	fixed repo optimize to remove unused logical blocks	2017-10-02 22:28:00 -07:00
Jarek Kowalski	a412e45f1f	cleanup: bump worker count to 32	2017-10-01 19:15:32 -07:00
Jarek Kowalski	f96dfe471e	support for array pack indexes instead of maps	2017-10-01 16:07:55 -07:00