Commit Graph

  • 14ea44d02d fixed S3 test by recreating bucket and using per-host bucket Jarek Kowalski 2018-02-03 17:19:37 -08:00
  • 7833626c4a added error handling when unable to load block index Jarek Kowalski 2018-01-24 11:44:26 +00:00
  • d7e1276a48 cleaned up storage.Storage API and documentation Jarek Kowalski 2018-01-10 18:40:31 -08:00
  • 090d97ba78 cli: reorganized all repo commands (connect/create/status) to top level Jarek Kowalski 2018-01-10 18:38:05 -08:00
  • 61abc8eee2 added support for S3 storage provider Jarek Kowalski 2018-01-10 18:10:15 -08:00
  • b6611d2131 removed depedendency on actual providers from 'repo' Jarek Kowalski 2018-01-10 18:10:56 -08:00
  • b65a5c5689 added scrubber to remove passwords from 'repo status' output Jarek Kowalski 2018-01-10 18:09:27 -08:00
  • bf4c0e694d refactored CLI to use individual subcommands to connect to individual storage types, each with specialized flags and validation Jarek Kowalski 2018-01-10 18:09:13 -08:00
  • 9bf5d4a8c2 added colorized logging on Windows, minor tweaks Jarek Kowalski 2018-01-06 12:04:39 -08:00
  • 7a97f22df4 switched to OS-specific locations for config and cache Jarek Kowalski 2018-01-06 11:58:46 -08:00
  • 393e7a8d59 disabled ANSI escape sequences when logging on windows Jarek Kowalski 2018-01-04 17:45:15 -08:00
  • f9f2c54993 added command to reconfigure caching on a repository that's already connected Jarek Kowalski 2018-01-04 17:44:58 -08:00
  • 7af93182a4 changed format of block index to be based on protobuf (using gogo), the deserialization is 10-12x faster than JSON, which matters for a large number of packs Jarek Kowalski 2018-01-01 22:47:35 -08:00
  • 10ca234633 switched to using zerolog for logging, by default log level=info, can be set to debug on commandline Jarek Kowalski 2018-01-01 19:12:35 -08:00
  • ef2ac1b5eb dead code Jarek Kowalski 2018-01-01 17:16:51 -08:00
  • 9bd93ee787 stale comment Jarek Kowalski 2018-01-01 17:12:56 -08:00
  • 98070064c8 index block list caching optimization - only load last 2 full compactions when scanning for active index blocks, this prevents the scan time from growing indefinitely Jarek Kowalski 2018-01-01 17:10:32 -08:00
  • 6dda6ca5e5 prefix each index block name with an inverse timestamp, that decreases over time an can be lexicographically sorted to produce newest blocks first when listing by prefix P Jarek Kowalski 2017-12-31 13:52:25 -08:00
  • 596c2b122a disk cache: remove list after a write fix close Jarek Kowalski 2017-12-31 11:19:49 -08:00
  • 51c1c77fb8 map initialization tweak Jarek Kowalski 2017-11-27 19:40:23 -08:00
  • 7f73ca2e52 added missing comments, cleanup Jarek Kowalski 2017-11-27 19:34:07 -08:00
  • c288daef04 eradicated old metadata.Manager and replaced snapshot.Manager to use manifest.Manager instead Jarek Kowalski 2017-11-27 19:15:02 -08:00
  • a644f2caa6 added 'manifest show' subcommand Jarek Kowalski 2017-11-27 18:49:33 -08:00
  • 30c11dc926 refactored block manager to support on-disk caching Jarek Kowalski 2017-11-27 18:07:16 -08:00
  • a0cc09b673 removed caching storage Jarek Kowalski 2017-11-26 10:38:55 -08:00
  • a700508048 changed API for manifest manager, added filtering, sorting and better output to 'manifest ls' Jarek Kowalski 2017-11-26 10:12:03 -08:00
  • 1968e2fa10 fix support for encrypted blocks with dash suffix Jarek Kowalski 2017-11-23 23:21:16 -08:00
  • eebb04c0cb added 'manifest ls' subcommand Jarek Kowalski 2017-11-23 23:15:24 -08:00
  • 6e83ad0b9b moved policies to separate package refactored policy management into policy.Manager that uses manifest.Manager internally Jarek Kowalski 2017-11-23 22:21:00 -08:00
  • de02941442 plumbed through key time in map storage Jarek Kowalski 2017-11-23 16:43:34 -08:00
  • d3b854ef8e optimized format for pack index, added compacted block with suffix -zTIMESTAMP where TIMESTAMP is base16-encoded unix nanoseconds of all blocks that this block supersedes, this allows much more efficient loading of blocks without having to delete anything Jarek Kowalski 2017-11-23 15:55:56 -08:00
  • 29ed48f734 manifest manager compaction Jarek Kowalski 2017-11-23 10:49:06 -08:00
  • 74195e316e switched buzhash implementation to github.com/silvasur/buzhash, turns out previous library was used incorrectly, this one has simpler API, produces different split points Jarek Kowalski 2017-11-14 22:45:15 -08:00
  • 15602e98f5 output version and build info Jarek Kowalski 2017-11-12 22:43:36 -08:00
  • 40a367f79e added 'block rm' subcommand Jarek Kowalski 2017-11-12 22:43:21 -08:00
  • 585b6ddcc1 additional test for race involving blockmanager.delete Jarek Kowalski 2017-11-12 20:43:33 -08:00
  • c0d71714e1 fixed a case where we would write a pack entry not corresponding to any block Jarek Kowalski 2017-11-12 18:52:32 -08:00
  • 50398692fe added support for block manager delete Jarek Kowalski 2017-11-11 10:06:27 -08:00
  • 5b287815ad add snapshot manifest to manifest.Manager at the end of upload Jarek Kowalski 2017-11-10 20:18:57 -08:00
  • 9c5c51dda7 deployment work Jarek Kowalski 2017-11-08 22:23:46 -08:00
  • 8982c985c9 travis creds Jarek Kowalski 2017-11-08 22:09:06 -08:00
  • 4967f6d406 fixed releasing Jarek Kowalski 2017-11-08 21:10:26 -08:00
  • 032e3cb737 merged 'format' and 'repo' blocks into 'kopia.repository' Jarek Kowalski 2017-11-08 20:09:06 -08:00
  • 2f25d8b9e8 initial manifest manager, not used yet Jarek Kowalski 2017-11-08 18:58:41 -08:00
  • 48a88eee09 removed dead code from metadata manage Jarek Kowalski 2017-11-04 09:36:00 -07:00
  • 113b1f55c7 cleanup Jarek Kowalski 2017-10-29 19:38:07 -07:00
  • 7c41f45e9f extracted block formatting options into a struct under 'block' Jarek Kowalski 2017-10-29 13:00:24 -07:00
  • 49a806154a code reorg Jarek Kowalski 2017-10-29 12:03:49 -07:00
  • 677f0535ed split repo connect/initialize/open into separate files Jarek Kowalski 2017-10-29 11:48:45 -07:00
  • e98a3b3325 tweaks Jarek Kowalski 2017-10-29 11:45:53 -07:00
  • f70d2af03d cleanup Jarek Kowalski 2017-10-29 11:42:11 -07:00
  • 0f6c569435 refactored repository status Jarek Kowalski 2017-10-29 10:35:06 -07:00
  • b6575a97ba moved metadata manager to 'metadata' package and refactored initialization sequence Jarek Kowalski 2017-10-29 10:29:22 -07:00
  • 769c3739bd refactored repository config out of metadata manager Jarek Kowalski 2017-10-29 08:44:53 -07:00
  • ac94dd1705 refactored repo.Connect to avoid calling connect() as most of what it does is not actually needed Jarek Kowalski 2017-10-28 23:40:39 -07:00
  • 3498f78d91 dead code cleanup Jarek Kowalski 2017-10-28 22:50:11 -07:00
  • e76f1f8a72 refactored various show command into one Jarek Kowalski 2017-10-24 21:59:40 -07:00
  • a48d54f00b added 'block show' similar to 'object show', but operating on virtual blocks Jarek Kowalski 2017-10-24 21:15:22 -07:00
  • b4e6a70e09 renamed 'repo optimize' to 'block reindex', added repacking method and unit tests Jarek Kowalski 2017-10-24 21:05:59 -07:00
  • 2aa44eb454 added listing group blocks and summary to 'block list' Jarek Kowalski 2017-10-17 19:24:16 -07:00
  • 42a560fbab block.Manager: Jarek Kowalski 2017-10-16 18:17:45 -07:00
  • 3f20c83e75 avoid storing auth.KeyMaster in metadata.Manager Jarek Kowalski 2017-10-15 09:09:10 -07:00
  • f7a1fb37b0 added auth.KeyManager Jarek Kowalski 2017-10-15 09:00:53 -07:00
  • 44f8940a8f removed support for object ID prefixes Jarek Kowalski 2017-10-14 15:59:23 -07:00
  • a45f557c4b changed object.ManagerOptions into a struct Jarek Kowalski 2017-10-14 15:49:32 -07:00
  • 59bb3ef943 additional test for block manager Jarek Kowalski 2017-10-14 15:43:42 -07:00
  • a715522c34 improved block stats output Jarek Kowalski 2017-10-14 11:19:56 -07:00
  • b5d0f2d120 more renames Jarek Kowalski 2017-10-11 05:47:05 -07:00
  • 0e48800f1a removed ObjectIDSection support, no longer needed because of packs Jarek Kowalski 2017-10-10 22:15:59 -07:00
  • c3f31d744f removed stuttering Jarek Kowalski 2017-10-10 22:09:53 -07:00
  • 31e5bce1cb moved object management to kopia/kopia/object Jarek Kowalski 2017-10-10 22:04:51 -07:00
  • f33ae14573 renamed 'blob.Storage' to 'storage.Storage' Jarek Kowalski 2017-10-09 07:06:20 -07:00
  • cc3e590136 moved block manager and formatter to a separate package Jarek Kowalski 2017-10-09 06:42:30 -07:00
  • a6a37b709a added pack info output to 'block list' Jarek Kowalski 2017-10-08 20:49:25 -07:00
  • ce5d9b2247 output pack group when listing blocks Jarek Kowalski 2017-10-08 17:40:37 -07:00
  • 444b6306ca refactored object manager to use block manager, exposed on repository added CLI commands to manipulate blocks Jarek Kowalski 2017-10-07 15:06:15 -07:00
  • 06efbfbe7a additional test case Jarek Kowalski 2017-10-07 07:24:52 -07:00
  • ff0ac7b8f1 refactored block manager added concurrency and compaction test for block manager Jarek Kowalski 2017-10-06 22:33:06 -07:00
  • c108dbf706 fixed degenerate case of saving index with zero items, added timing info Jarek Kowalski 2017-10-06 19:22:58 -07:00
  • a7818a9a4b fixed handling of zero-byte packs and block manager index flushing order Jarek Kowalski 2017-10-06 06:40:43 -07:00
  • 7071d0b01b refactored block manager, added unit tests Jarek Kowalski 2017-10-05 22:12:02 -07:00
  • 300008c98d renamed pack manager to block manager moved block reading and writing to blockManager Jarek Kowalski 2017-10-03 22:02:23 -07:00
  • 10b9b4b451 fixed repo optimize to remove unused logical blocks Jarek Kowalski 2017-10-02 22:28:00 -07:00
  • a412e45f1f cleanup: bump worker count to 32 Jarek Kowalski 2017-10-01 19:15:32 -07:00
  • 45381eff48 cleanup Jarek Kowalski 2017-10-01 19:15:05 -07:00
  • 903386fb33 removed block size cache, relying on pack manager data instead Jarek Kowalski 2017-10-01 16:05:32 -07:00
  • f96dfe471e support for array pack indexes instead of maps Jarek Kowalski 2017-09-30 17:59:10 -07:00
  • 79440ab969 fixed locking of pack manager, changed how it writes pack and index objects to avoid using object writer, which prevents certain cycles Jarek Kowalski 2017-09-24 21:35:19 -07:00
  • b2d9e07b75 fix 'object list' output. Jarek Kowalski 2017-09-24 12:37:42 -07:00
  • 1366f967f4 pack manager: Jarek Kowalski 2017-09-23 11:49:52 -07:00
  • 1e0ef911fa added 'object list' subcommand Jarek Kowalski 2017-09-23 11:49:04 -07:00
  • 4b77119736 made packing the default, replaced FinishPacking with Flush() Jarek Kowalski 2017-09-23 10:55:57 -07:00
  • 1b548b1ad5 store unpacked blocks in a pack index Jarek Kowalski 2017-09-22 23:01:09 -07:00
  • 4259227eb8 additional diagnostics output Jarek Kowalski 2017-09-18 22:29:24 -07:00
  • 89ddf98564 added 'object verify' subcommand to verify that the given object (file or directory) has all required blocks present removed GetStorageBlock(), replaced with Verify() that also returns storage blocks added blockSizeCache.put() so that object writer can update the cache Jarek Kowalski 2017-09-16 10:47:53 -07:00
  • 32c603362c removed support for inline object IDs now that packs are supported everywhere Jarek Kowalski 2017-09-16 09:27:32 -07:00
  • b5a7d05adf refactored Repository API Jarek Kowalski 2017-09-15 18:35:27 -07:00
  • 91e43b6842 changed how pack files get split by using the same splitter mechanism (static or dynamic) in use for regular objects Jarek Kowalski 2017-09-14 19:17:19 -07:00
  • 0d3dce175c small compaction fix Jarek Kowalski 2017-09-12 06:27:32 -07:00
  • e07c134297 typo Jarek Kowalski 2017-09-11 21:13:43 -07:00