Commit Graph

  • 210d165148 changed how cache sweep is performed to use heap instead of loading and sorting everything in-memory Jarek Kowalski 2018-09-14 19:10:42 -07:00
  • 09408a5782 updating timestamp on block cache access to have actual LRU semantics Jarek Kowalski 2018-09-14 18:51:56 -07:00
  • 08d62d60f1 revamped progress bar by including hashing and uploading part separately Jarek Kowalski 2018-09-13 21:41:11 -07:00
  • 636837317b added 'block verify' subcommand Jarek Kowalski 2018-09-13 20:14:07 -07:00
  • a922c2b568 s3: fixed off-by-one error in s3_storage.GetBlock(), added more tests added progress reporting to s3 provider Jarek Kowalski 2018-09-13 20:11:41 -07:00
  • 69782d2abe refactored cache commands Jarek Kowalski 2018-09-13 19:32:56 -07:00
  • 906b8eaee3 plumbed through context to all manifest manager methods and switched to lazy initialization Jarek Kowalski 2018-09-12 19:15:53 -07:00
  • f2c7c29a57 fixed travis build Jarek Kowalski 2018-09-10 19:31:57 -07:00
  • 3d0ab23c03 switched release process to using goreleaser Jarek Kowalski 2018-09-08 06:58:39 -07:00
  • 93e03635fe added dist/ to .gitignore Jarek Kowalski 2018-09-09 18:24:12 -07:00
  • 3b82eadaa6 lint: disabled gosec linter until issues are fixed Jarek Kowalski 2018-09-09 18:23:47 -07:00
  • 59a3f4beec switched to stable version of gopkg.in/cheggaaa/pb.v1 Jarek Kowalski 2018-09-09 16:17:08 -07:00
  • 4cf73fdd0f moved storagetesting to kopia/internal Jarek Kowalski 2018-09-08 11:08:51 -07:00
  • 1b014c875a simplified repository API password handling. Jarek Kowalski 2018-09-04 21:50:53 -07:00
  • 1bbc169c0d added missing package godoc, fixed test paths Jarek Kowalski 2018-08-31 19:12:58 -07:00
  • 6d2d663cac moved CLI storage support to cli/ Jarek Kowalski 2018-08-30 22:08:55 -07:00
  • 91066f2469 reorganized low-level repository packages by moving them all under kopia/kopia/repo/ Jarek Kowalski 2018-08-30 21:59:16 -07:00
  • 6484a55d7c removed policy.Manager and snapshot.Manager and switched to package-level function interface instead Jarek Kowalski 2018-08-30 06:47:58 -07:00
  • a44c9554c0 refactored upload to separate package Jarek Kowalski 2018-08-30 06:21:40 -07:00
  • 3ef4593b03 refactored policy management into separate policy/ package Jarek Kowalski 2018-08-29 23:08:40 -07:00
  • 9060e7d310 moved repository VFS nodes into fs/repofs/ package Jarek Kowalski 2018-08-29 22:56:23 -07:00
  • 0f20965143 added 'blockindex recover' tests Jarek Kowalski 2018-08-25 16:29:13 -07:00
  • 5dd755fcec added support for index recovery from pack files, work in progress Jarek Kowalski 2018-08-25 15:04:07 -07:00
  • 98aa71c02a svg image, deleted png images Jarek Kowalski 2018-08-20 22:30:16 -07:00
  • a8a609c816 svg image Jarek Kowalski 2018-08-20 22:26:27 -07:00
  • 30c1c05aa0 updated readme and added logo Jarek Kowalski 2018-08-20 22:16:30 -07:00
  • 8850c69f2e updated readme and added logo Jarek Kowalski 2018-08-20 22:08:08 -07:00
  • de53519c66 Fixed a race condition between two manifest managers, where the first one compacts blocks and second one is in the middle of listing and loading all blocks. Jarek Kowalski 2018-08-20 20:53:52 -07:00
  • 6587ade1f9 cli: added 'policy edit' command which launches an editor and allows user to edit policy JSON directly Jarek Kowalski 2018-08-20 19:07:44 -07:00
  • f79571b8b3 upload: use runtime.NumCPU() for parallel upload Jarek Kowalski 2018-08-19 11:48:32 -10:00
  • 864b763cc0 progress: output the # of files in each directory Jarek Kowalski 2018-08-19 11:40:48 -10:00
  • 1ff003c820 added internal lib for gitignore-style pattern matching added 'ignorefs' which transparently hides all files according to .kopiaignore, which is very similar to .gitignore Jarek Kowalski 2018-08-19 11:33:20 -10:00
  • 5d73d18347 removed Parent from fs.Entry Jarek Kowalski 2018-08-18 16:12:52 -10:00
  • 5386370906 emit snapshot size Jarek Kowalski 2018-08-04 09:57:10 -07:00
  • db6991cb1a switched to stable gometalinter.v2 Jarek Kowalski 2018-07-27 18:37:17 -07:00
  • 9a3a2da5fa fixed stdout output of 'expire' Jarek Kowalski 2018-07-27 18:28:58 -07:00
  • 8fc77bd5b1 changed repository example to use filesystem instead of GCS Jarek Kowalski 2018-07-22 12:16:44 -07:00
  • 9543c85e5a introduced side-effecting import 'kopia/kopia/storage/providers' which registers all storage providers, by default no storage providers are dependencies of 'repo' Jarek Kowalski 2018-07-22 12:09:42 -07:00
  • be32f6b717 enabled more comprehensive repository stress test after resolving remaining races: skipped stress tests during short testing changed default make target to run short tests Jarek Kowalski 2018-07-20 18:29:57 -08:00
  • 00b18b4da4 switched to consistent listing of block indexes Jarek Kowalski 2018-07-20 18:26:14 -08:00
  • 14feae8193 replaced silent return with t.Skip() in s3_storage_test.go Jarek Kowalski 2018-07-20 17:28:34 -08:00
  • bc835955d5 added utility functions to fetch all matching list results without a callback until a self-consistent snapshot has been reached Jarek Kowalski 2018-07-20 17:28:07 -08:00
  • 306ab5888d changed source manager to handle remote hosts differently Jarek Kowalski 2018-07-19 15:03:48 -08:00
  • 8b00bdb284 fixed 'coverage' makefile target Jarek Kowalski 2018-07-19 15:02:45 -08:00
  • 14471f124d disable s3 test if the endpoint is not available Jarek Kowalski 2018-07-19 15:02:16 -08:00
  • 6af6914b40 reject invalid offset/length in map storage Jarek Kowalski 2018-07-19 15:01:53 -08:00
  • a91c46e9b4 added block cache tests Jarek Kowalski 2018-07-19 15:01:17 -08:00
  • b2b34c1dea reacted to a change in gometalinter that failed the build Jarek Kowalski 2018-07-10 06:19:22 -07:00
  • 09c80ed0e9 disabled repository stress Jarek Kowalski 2018-07-09 22:17:11 -07:00
  • a299bdb6f9 fixed linter errors Jarek Kowalski 2018-07-09 21:51:24 -07:00
  • ec66349eeb Fixed nasty bug where if the storage fails to write a block we would clobber pending block data in memory by encrypting it in place. This was happening a lot when backing up to cloud and closing the laptop whihle the upload was going on. Added regression test and a rudimentary mechanism for fault injection in storage. Jarek Kowalski 2018-07-09 21:04:35 -07:00
  • 801b6eb5eb cleaned up block manager to make enforcement of invariants easier Jarek Kowalski 2018-07-09 21:00:19 -07:00
  • f44234ef3a improved logging Jarek Kowalski 2018-07-04 08:18:11 -07:00
  • 2cd426eba0 additional logging for block manager Jarek Kowalski 2018-07-03 21:44:47 -07:00
  • c57531b751 switched Go logging library to github.com/op/go-logging which lets us override log levels per module Jarek Kowalski 2018-07-03 20:18:07 -07:00
  • 1c99dae4b1 added new type of stress test that utilizes multiple repository features Jarek Kowalski 2018-07-02 20:43:24 -07:00
  • 4a11ca5f07 increase the number of log files to retain Jarek Kowalski 2018-07-02 20:42:56 -07:00
  • 32b83be0b4 additional logging for block manager Jarek Kowalski 2018-07-02 20:42:12 -07:00
  • bd7fa874b0 write list cache file atomically Jarek Kowalski 2018-07-02 20:38:48 -07:00
  • fe52c87a58 better error reporting on data corruption Jarek Kowalski 2018-07-01 13:17:23 -07:00
  • 00db1abb35 added --html option to 'kopia server' which allows colocating HTML apps on the same port Jarek Kowalski 2018-07-01 13:16:58 -07:00
  • 75da8f3f31 removed dead code Jarek Kowalski 2018-06-24 15:04:58 -07:00
  • cbcdfa2eb6 switched coverage to native go test Jarek Kowalski 2018-06-23 16:32:47 -07:00
  • 94965997e8 switched storage logging to be on by default Jarek Kowalski 2018-06-23 13:50:11 -07:00
  • ba6419339a refactored storage.ListBlocks() to be callback based which simplifies usage significantly Jarek Kowalski 2018-06-23 13:47:43 -07:00
  • 83036f1a0d upload progress Jarek Kowalski 2018-06-20 16:26:50 -07:00
  • 91c51f4807 refactored 'block gc' subcommand Jarek Kowalski 2018-06-23 13:05:40 -07:00
  • f4424bd79c parallelized 'block rewrite' Jarek Kowalski 2018-06-23 12:30:59 -07:00
  • 1defc18300 refactored command_snapshot_expire Jarek Kowalski 2018-06-23 12:30:45 -07:00
  • c7d898a674 fixed block manager lock which triggered asserts during parallel read/write Jarek Kowalski 2018-06-23 12:30:19 -07:00
  • 173c5613c1 fixed aliasing of read buffers in block manager Jarek Kowalski 2018-06-18 19:16:13 -07:00
  • 85b4bb7cd9 better fix for aliasing issue Jarek Kowalski 2018-06-18 18:23:37 -07:00
  • 23318a461d fixed rare data corruption bug due to buffer aliasing Jarek Kowalski 2018-06-18 17:57:21 -07:00
  • 2f1cf8129b added skeleton of CLI actions for manipulating server sources Jarek Kowalski 2018-06-17 12:49:06 -07:00
  • 4faf3cd9d0 finalized CLI to manipulate policies Jarek Kowalski 2018-06-16 12:17:58 -07:00
  • 70ecebb769 added HTTP API to list policies Jarek Kowalski 2018-06-16 08:46:56 -07:00
  • a6f2e8c3f3 added option to disable block manager flush to ensure that a sequence of writes/deletes is atomically observable Jarek Kowalski 2018-06-14 17:11:00 -07:00
  • 5333a9cfc1 reorganized dynamic splitter logic to be slightly faster and ignore hash value of zero, which introduced blocks smaller than expected on average Jarek Kowalski 2018-06-13 22:56:17 -07:00
  • 6b79ab79d0 fixed --no-list-caching Jarek Kowalski 2018-06-13 22:37:45 -07:00
  • d3cf01fc45 present legacy object IDs without the 'D' Jarek Kowalski 2018-06-13 22:35:46 -07:00
  • 7dea2359c2 removed 'D' prefix from direct objects Jarek Kowalski 2018-06-12 20:39:53 -07:00
  • e21a5ce650 refactored committed block index to separate cache from the lookup Jarek Kowalski 2018-06-12 20:22:37 -07:00
  • df3129682c Added repo.Repository.Refresh{,Periodically} Jarek Kowalski 2018-06-10 21:42:22 -07:00
  • 2be6041d96 rewrote manifest manager to use distinct committed and pending layers Jarek Kowalski 2018-06-10 21:29:58 -07:00
  • 22b0766570 API server, work in progress Jarek Kowalski 2018-06-10 20:36:49 -07:00
  • 4313d5f6ab added block.Manager.Refresh method Jarek Kowalski 2018-06-10 20:35:32 -07:00
  • b105cca96c added 'kopia server' which runs simple daemon with simple HTTP API server, which will be used to build a web UI Jarek Kowalski 2018-06-10 17:24:42 -07:00
  • e9c11ea106 refactored block cache and list cache Jarek Kowalski 2018-06-10 12:55:42 -07:00
  • 846a46b879 changed how caching is controlled in block manager by using context instead of options Jarek Kowalski 2018-06-10 10:20:52 -07:00
  • 9a17179615 reorganized local cache to have 256 shards instead of tens of thousands, this improves scan speed and startup time with lots of cached items Jarek Kowalski 2018-06-10 08:15:07 -07:00
  • 6838d820c8 revamped object verification to be parallelized, added option to randomly verify a percentage of object contents Jarek Kowalski 2018-06-09 21:25:28 -07:00
  • 19dcda1f47 removed lock while loading a block that made block manager unnecessarily single-threaded Jarek Kowalski 2018-06-09 18:36:59 -07:00
  • e5e5577cfd fixed linter warnings Jarek Kowalski 2018-06-09 14:06:43 -07:00
  • fc618b0c66 changed block.PhysicalBlockID to a string Jarek Kowalski 2018-06-07 18:42:22 -07:00
  • 6ee5c81e21 renamed 'pack block ID' to 'pack file' everywhere Jarek Kowalski 2018-06-07 07:52:17 -07:00
  • f869996a1c aligned 'ls' output Jarek Kowalski 2018-06-06 21:38:32 -07:00
  • 9ea74b2862 when reporting stats to GA do it asynchronously at start and do not block for more than 1s on exit Jarek Kowalski 2018-06-06 21:38:08 -07:00
  • 7738fa536f revamped 'object verify' to support verifying one or more sources easily Jarek Kowalski 2018-06-06 21:18:44 -07:00
  • 8d751cb7bf added 'k' prefix to all directory blocks and 'h' prefix to all hash-cache blocks Jarek Kowalski 2018-06-06 21:15:52 -07:00
  • 985980925f removed support for v0 block format Jarek Kowalski 2018-06-04 23:03:38 -07:00