Commit Graph

  • 3155aefa59 changed how pack indexes are flushed to avoid prematurely flushing directory packs to make bigger objects Jarek Kowalski 2017-09-11 21:10:29 -07:00
  • 29e0153eff fixed index data corruption during optimize Jarek Kowalski 2017-09-11 20:53:52 -07:00
  • 4010f35a30 fixed deadlock caused by attempt to write asynchronous pack data Jarek Kowalski 2017-09-10 23:08:27 -07:00
  • 207dff1b9f also close current pack when time-based flushing happens Jarek Kowalski 2017-09-10 12:22:28 -07:00
  • 4cfa1d5706 added transparent gunzip to 'obj show' Jarek Kowalski 2017-09-10 12:01:58 -07:00
  • 405ba139ce periodically flush pack indexes every 10 minutes Jarek Kowalski 2017-09-10 11:35:16 -07:00
  • bb64238eba added support for pack index compaction (not the pack file compaction, which is not implemented yet). it is combining multiple pack index objects P* into one large P which loads more quickly Jarek Kowalski 2017-09-09 09:30:06 -07:00
  • 6ef7005396 wait for Ctrl-C when mounting Jarek Kowalski 2017-09-06 18:48:23 -07:00
  • c72d493634 cleanup Jarek Kowalski 2017-09-07 20:01:59 -07:00
  • a7e9f995de removed dependency between pack manager and MetadataManager - all pack indexs are stored as objects with prefix P Jarek Kowalski 2017-09-07 19:58:38 -07:00
  • 976f3a8819 when saving pack index, also save an object with "P" prefix. Jarek 2017-09-06 19:16:54 -07:00
  • fa80f40185 added support for mounting snapshots as drive letters and opening them in explorer on Windows. Jarek 2017-09-05 19:01:51 -07:00
  • 82657f957d extracted timeFormat constant for printing dates in CLI Jarek Kowalski 2017-09-05 18:21:17 -07:00
  • 76b712c2fb moved creation of repofs entries to snapshot.Manager Jarek Kowalski 2017-09-04 20:47:51 -07:00
  • 371fc15694 merged 'repofs' into 'snapshot' package Jarek Kowalski 2017-09-04 20:26:25 -07:00
  • 2b032b1748 refactored FS caching flags Jarek Kowalski 2017-09-04 17:49:36 -07:00
  • b08a35e92b refactored FS cache into reusable wrapper that can be applied on top of any filesystem abstraction Jarek Kowalski 2017-09-04 16:27:46 -07:00
  • 8435ed4c80 beginnings of end-to-end test, cleaned up stdout vs stderr output in a few cases Jarek Kowalski 2017-09-02 19:37:45 -07:00
  • 0c98537d30 Fixed handle leak in filesystem storage GetBlock() that was causing test failures on Windows Jarek 2017-09-03 15:02:17 -07:00
  • e8847b65cc mounting webdav works, changed how FS directory caching is implemented to be based solely on ObjectIDs Jarek Kowalski 2017-09-02 08:29:59 -07:00
  • b5973aa998 added support for Ctrl-C unmounting of fuse filesystem Jarek Kowalski 2017-09-02 07:37:51 -07:00
  • 2c6213239a added cache expiration to fscache.Cache, so that entries based on ObjectID are cached for a very long time and others expire almost immediately. This allows 'kopia mount all' to see newly created snapshots. Jarek Kowalski 2017-08-29 21:37:36 -07:00
  • 4b1547fec9 periodically refresh the metadata manager cache when mounting Jarek Kowalski 2017-08-29 21:34:52 -07:00
  • 998ed1ddac simplified MetadataManager API for listing, added caching of metadata items (unbounded for now) Jarek Kowalski 2017-08-29 21:10:08 -07:00
  • 42c825c4f4 travis: don't do a shallow clone Jarek Kowalski 2017-08-27 19:52:50 -07:00
  • 4cb1339236 fixed windows build Jarek Kowalski 2017-08-27 18:25:37 -07:00
  • 2df77e8b73 add go 1.8 back Jarek Kowalski 2017-08-27 18:20:17 -07:00
  • 7b766a7681 fixed liter warnings not detected by old linter Jarek Kowalski 2017-08-27 15:16:27 -07:00
  • 2a37618243 moved to go 1.9 Jarek Kowalski 2017-08-27 15:04:02 -07:00
  • d613d99f7e beginnings of WebDAV mounting support, not really working yet, need to do have better caching Jarek Kowalski 2017-08-27 15:02:56 -07:00
  • e01fcff9a4 fixed sorting of synthetic fs.Entries Jarek Kowalski 2017-08-27 14:49:15 -07:00
  • ebe00e9405 fixup Jarek Kowalski 2017-08-27 09:00:32 -07:00
  • 2b887b0b2e changed fs.Reader to be a Seeker too Jarek Kowalski 2017-08-27 08:58:59 -07:00
  • c3064f56b0 added experimental support for WebDAV backends (use http:// or https://) Jarek Kowalski 2017-08-21 22:11:26 -07:00
  • b769dc1863 emit a warning instead of failing when unsupported directory entries are encountered Jarek Kowalski 2017-08-20 13:54:54 -07:00
  • 72174c4200 added 'snapshot estimate' command which counts files to snapshot while applying the current policy and estimates the time to upload Jarek Kowalski 2017-08-20 13:54:28 -07:00
  • 6f06e04c9a moved genpasswd to another tool Jarek Kowalski 2017-08-20 08:08:19 -07:00
  • 70c9d49232 reorganized more top-level CLI commands into subcommands Jarek Kowalski 2017-08-20 08:00:01 -07:00
  • f606ab4347 reorganized more top-level CLI commands into subcommands Jarek Kowalski 2017-08-20 07:51:24 -07:00
  • 13e328014b moved 'expire' to 'snapshot expire' Jarek Kowalski 2017-08-20 07:42:49 -07:00
  • 81f060a2c3 CLI: renamed 'backup' to 'snapshot create' and 'backups' to 'snapshot list' Jarek Kowalski 2017-08-19 23:44:51 -07:00
  • d2105aea40 added files policy include/exclude configuration and checking during Upload Jarek Kowalski 2017-08-19 23:36:49 -07:00
  • 61cc2b46be tweaks to upload behavior: default to ignoring errors, do not hashcache recently changed files Jarek Kowalski 2017-08-19 21:07:38 -07:00
  • ff487d1e85 added ability to mount special object 'all' that exposes the whole set of snapshots grouped by user@host, path and time Jarek Kowalski 2017-08-19 19:49:18 -07:00
  • 484db94e23 removed support for legacy bundles, now that packs are supported everywhere Jarek Kowalski 2017-08-19 18:06:15 -07:00
  • fa7c558aaf cleanup indirect object creation logic, which has historically been quite messy, with the new code structure it's much more readable Jarek Kowalski 2017-08-19 14:29:39 -07:00
  • f0fcc60374 now that packing is transparent, inline object IDs are not needed, removed support for them and simplified ObjectWriter.Result() to no longer require a parameter Jarek Kowalski 2017-08-19 14:23:09 -07:00
  • afc9a4fa4d typo Jarek Kowalski 2017-08-19 14:10:22 -07:00
  • cd2fcfeb4f refactored ObjectWriter to allow parallel hashing of parts of large files Jarek Kowalski 2017-08-19 14:08:42 -07:00
  • 5428657404 made packing transparent to ObjectID, that means previous object Pfoo@packID is now accessible as Dfoo, indistinguishable from real block-based 'foo' Jarek Kowalski 2017-08-19 09:28:59 -07:00
  • 1f8df162c3 simplified how upload code is structured + how hash cache is emitted Jarek Kowalski 2017-08-18 23:00:03 -07:00
  • 15e0d59c33 added backup --force-hashing-percentage=0..100 which can be used to re-hash all or part of source files when uploading Jarek Kowalski 2017-08-18 21:13:47 -07:00
  • 3df0fc0445 added pack support to GetStorageBlocks() Jarek Kowalski 2017-08-18 20:41:25 -07:00
  • 0c597de0fe added --trace-object-manager flag for debugging repository operation. Jarek Kowalski 2017-08-18 20:40:44 -07:00
  • 3559776c5f changed internal representation of indirect object IDs to be recursive, to allow storing I objects in packs Jarek Kowalski 2017-08-18 20:13:52 -07:00
  • 5fead928e0 fixed backups and expire command output Jarek Kowalski 2017-08-16 20:26:06 -07:00
  • 9634a798a8 additional detail output for 'backups' command Jarek Kowalski 2017-08-13 15:59:45 -07:00
  • 715f6cf937 change block size cache prefetching to be sharded based on only one character Jarek Kowalski 2017-08-13 11:19:02 -07:00
  • 4b1b7832e2 fixed logging storage output Jarek Kowalski 2017-08-13 11:18:34 -07:00
  • 3e36ce748f removed conditional PutBlock, which was causing more trouble that it's worth Jarek Kowalski 2017-08-13 08:53:12 -07:00
  • 0b5a195c26 added block size cache to speed up BlockSize() before uploading a block Jarek Kowalski 2017-08-13 08:48:36 -07:00
  • 8dda3bf2c5 added option to read portions of blob storage blocks Jarek Kowalski 2017-08-12 23:11:01 -07:00
  • 097cf2d0b3 added option to ignore file read errors for backup and migration Jarek Kowalski 2017-08-12 23:10:29 -07:00
  • 8f373c3aa4 travis - switched to 1.8 Jarek Kowalski 2017-08-12 17:36:22 -07:00
  • fe70b1e8b0 re-added conditional PutObject Jarek Kowalski 2017-08-12 16:36:12 -07:00
  • 9d1fb5032d improved GCS exponential backoff Jarek Kowalski 2017-08-12 16:28:01 -07:00
  • 60654e125b added 'migrate' command that can move all snapshot data between repositories. It's meant to be used in a cloud VM to (relatively) quickly re-create the contents of all snapshots using new repository format. Copying is done using snapshot.Uploader on top of a virtual filesystem from the source repository. Jarek Kowalski 2017-08-12 15:23:51 -07:00
  • 94f5420557 re-added support for bundles (read-only) to aid data migration Jarek Kowalski 2017-08-12 14:07:22 -07:00
  • 634302701f changed default pack file size to 20MB and pack item to 4MB Jarek Kowalski 2017-08-10 22:31:13 -07:00
  • ce4d59d8bb added support for cancelling upload with Ctrl-C which marks the snapshot as incomplete added --include-incomplete to 'backups' command improved upload API, added support for better cancellation and upload limits changed hashcache Reader and Writer to use interfaces Jarek Kowalski 2017-08-10 18:44:45 -07:00
  • a153c80064 tweaks to pack index format Jarek Kowalski 2017-08-10 18:30:09 -07:00
  • 2c86e54c81 removed support for bundles now that packs are looking much better and more flexible Jarek Kowalski 2017-08-10 18:24:22 -07:00
  • a0b8a75133 changed kopia metadata show to pretty-print JSON by default since all data in the vault is JSON Jarek Kowalski 2017-08-10 18:12:53 -07:00
  • e76243e550 introduced multiple packs per upload - directory data is stored together, which allows is to be cached together speeding up browsing of backups Jarek Kowalski 2017-08-10 18:08:34 -07:00
  • 036ce15a88 move throttling to GCS storage options and create/connect flags Jarek Kowalski 2017-08-10 17:29:45 -07:00
  • e3e095c347 swapped order of packed object IDs Jarek Kowalski 2017-08-09 20:13:00 -07:00
  • 9ba224854e fixed throttling in GCS provider Jarek Kowalski 2017-08-09 19:47:14 -07:00
  • 19339345e9 cleaned up connect/create CLI options Jarek Kowalski 2017-08-09 16:52:47 -07:00
  • f9a0a5dfaf fix for create incorrectly handling store errors Jarek Kowalski 2017-08-09 08:14:10 -07:00
  • 4912dc0f79 simplified pack manager Jarek Kowalski 2017-08-08 08:03:43 +02:00
  • cbb492ea09 removed buffer manager Jarek Kowalski 2017-08-08 07:56:05 +02:00
  • 65d188ec59 removed unnecessary optimization to avoid hashing directories, this is now unnecessary since they end up being packed most of the time so they don't incur the cost of a round-trip Jarek Kowalski 2017-08-07 15:29:57 +02:00
  • 22fe31044c enable packing when uploading Jarek Kowalski 2017-08-07 15:12:01 +02:00
  • dbeabbb1ef deprecated inline object IDs, which cause significant slowdowns to the backup process, will be much more efficient with pack files Jarek Kowalski 2017-08-07 15:11:01 +02:00
  • 8d714b48da parsing of pack IDs Jarek Kowalski 2017-08-07 14:31:28 +02:00
  • 6821b8def4 Initial support for creating pack objects, which are identified by Jarek Kowalski 2017-08-04 13:12:55 +02:00
  • 2dfcd23ea9 added hashcache database, based on BoltDB which caches results of computing ObjectID for any given os.FileInfo. Jarek Kowalski 2017-08-02 07:23:38 +02:00
  • 21a1cbd0bb renamed auth.Options to auth.SecurityOptions Jarek Kowalski 2017-08-01 10:15:31 +02:00
  • df83cbea27 added status subcommand Jarek Kowalski 2017-08-01 09:57:09 +02:00
  • cfbef13aff Major refactoring to how repository connections are established and managed to simplify programmatic API usage and CLI. Removed unnecessary and confusing options, improved the API. Jarek Kowalski 2017-07-18 15:40:50 +02:00
  • 095fc4aba4 removed 'client' package and merged with 'repo', names still not final Jarek Kowalski 2017-07-18 14:48:10 +02:00
  • 4dcdb69dcc replaced use of Vault with Repository Jarek Kowalski 2017-07-18 13:39:26 +02:00
  • 50c2cd86a3 refactored credentials to separate package Jarek Kowalski 2017-07-18 12:55:20 +02:00
  • d8b2ba4c24 moved most methods from Repository to casManager Jarek Kowalski 2017-07-18 12:37:44 +02:00
  • 5721ba8140 renamed repository.go to cas_manager.go in preparation for name flip Jarek Kowalski 2017-07-18 10:58:41 +02:00
  • ef71993cde stale comment Jarek Kowalski 2017-07-18 10:51:14 +02:00
  • cda3086dea cleanup: moved writeback to separate file Jarek Kowalski 2017-07-18 10:45:34 +02:00
  • c3f0d5ddc4 moved all Vault types to 'repo' package Jarek Kowalski 2017-07-18 10:30:07 +02:00
  • cab2723443 removed vault object ID storage, which was really only there to make convergent encryption UI not suck as much Jarek Kowalski 2017-07-18 10:13:25 +02:00
  • 989bb27f06 renamed vault.Format to vault.VaultFormat in preparation for merging with repo Jarek Kowalski 2017-07-18 10:07:51 +02:00