Jarek Kowalski
cd2fcfeb4f
refactored ObjectWriter to allow parallel hashing of parts of large files
2017-08-19 14:08:42 -07:00
Jarek Kowalski
5428657404
made packing transparent to ObjectID, that means previous object Pfoo@packID is now accessible as Dfoo, indistinguishable from real block-based 'foo'
2017-08-19 09:42:06 -07:00
Jarek Kowalski
1f8df162c3
simplified how upload code is structured + how hash cache is emitted
2017-08-18 23:00:03 -07:00
Jarek Kowalski
15e0d59c33
added backup --force-hashing-percentage=0..100 which can be used to re-hash all or part of source files when uploading
2017-08-18 21:13:47 -07:00
Jarek Kowalski
3df0fc0445
added pack support to GetStorageBlocks()
2017-08-18 20:41:25 -07:00
Jarek Kowalski
0c597de0fe
added --trace-object-manager flag for debugging repository operation.
2017-08-18 20:40:57 -07:00
Jarek Kowalski
3559776c5f
changed internal representation of indirect object IDs to be recursive, to allow storing I objects in packs
2017-08-18 20:13:52 -07:00
Jarek Kowalski
5fead928e0
fixed backups and expire command output
2017-08-16 20:26:06 -07:00
Jarek Kowalski
9634a798a8
additional detail output for 'backups' command
2017-08-13 15:59:45 -07:00
Jarek Kowalski
715f6cf937
change block size cache prefetching to be sharded based on only one character
2017-08-13 11:19:02 -07:00
Jarek Kowalski
4b1b7832e2
fixed logging storage output
2017-08-13 11:18:34 -07:00
Jarek Kowalski
3e36ce748f
removed conditional PutBlock, which was causing more trouble that it's worth
...
with unconditional rewrites any partial block writes are self-correcting on subsequent updates,
without it they can't complete at all.
2017-08-13 08:54:50 -07:00
Jarek Kowalski
0b5a195c26
added block size cache to speed up BlockSize() before uploading a block
2017-08-13 08:48:36 -07:00
Jarek Kowalski
8dda3bf2c5
added option to read portions of blob storage blocks
2017-08-12 23:11:01 -07:00
Jarek Kowalski
097cf2d0b3
added option to ignore file read errors for backup and migration
2017-08-12 23:10:29 -07:00
Jarek Kowalski
8f373c3aa4
travis - switched to 1.8
2017-08-12 17:36:22 -07:00
Jarek Kowalski
fe70b1e8b0
re-added conditional PutObject
2017-08-12 16:42:13 -07:00
Jarek Kowalski
9d1fb5032d
improved GCS exponential backoff
2017-08-12 16:28:01 -07:00
Jarek Kowalski
60654e125b
added 'migrate' command that can move all snapshot data between repositories. It's meant to be used in a cloud VM to (relatively) quickly re-create the contents of all snapshots using new repository format. Copying is done using snapshot.Uploader on top of a virtual filesystem from the source repository.
2017-08-12 15:23:51 -07:00
Jarek Kowalski
94f5420557
re-added support for bundles (read-only) to aid data migration
2017-08-12 14:07:22 -07:00
Jarek Kowalski
634302701f
changed default pack file size to 20MB and pack item to 4MB
2017-08-10 22:31:13 -07:00
Jarek Kowalski
ce4d59d8bb
added support for cancelling upload with Ctrl-C which marks the snapshot as incomplete
...
added --include-incomplete to 'backups' command
improved upload API, added support for better cancellation and upload limits
changed hashcache Reader and Writer to use interfaces
2017-08-10 20:27:31 -07:00
Jarek Kowalski
a153c80064
tweaks to pack index format
2017-08-10 18:30:09 -07:00
Jarek Kowalski
2c86e54c81
removed support for bundles now that packs are looking much better and more flexible
2017-08-10 18:24:22 -07:00
Jarek Kowalski
a0b8a75133
changed kopia metadata show to pretty-print JSON by default since all data in the vault is JSON
2017-08-10 18:12:53 -07:00
Jarek Kowalski
e76243e550
introduced multiple packs per upload - directory data is stored together, which allows is to be cached together speeding up browsing of backups
2017-08-10 18:08:34 -07:00
Jarek Kowalski
036ce15a88
move throttling to GCS storage options and create/connect flags
2017-08-10 17:29:45 -07:00
Jarek Kowalski
e3e095c347
swapped order of packed object IDs
2017-08-09 20:13:00 -07:00
Jarek Kowalski
9ba224854e
fixed throttling in GCS provider
2017-08-09 19:47:14 -07:00
Jarek Kowalski
19339345e9
cleaned up connect/create CLI options
2017-08-09 16:59:53 -07:00
Jarek Kowalski
f9a0a5dfaf
fix for create incorrectly handling store errors
2017-08-09 08:14:10 -07:00
Jarek Kowalski
4912dc0f79
simplified pack manager
2017-08-08 08:03:43 +02:00
Jarek Kowalski
cbb492ea09
removed buffer manager
2017-08-08 07:56:39 +02:00
Jarek Kowalski
65d188ec59
removed unnecessary optimization to avoid hashing directories, this is now unnecessary since they end up being packed most of the time so they don't incur the cost of a round-trip
...
disabled bundling of entries, since support for bundles will be removed soon
2017-08-08 07:56:28 +02:00
Jarek Kowalski
22fe31044c
enable packing when uploading
2017-08-07 15:12:01 +02:00
Jarek Kowalski
dbeabbb1ef
deprecated inline object IDs, which cause significant slowdowns to the backup process, will be much more efficient with pack files
2017-08-07 15:11:01 +02:00
Jarek Kowalski
8d714b48da
parsing of pack IDs
2017-08-07 14:31:28 +02:00
Jarek Kowalski
6821b8def4
Initial support for creating pack objects, which are identified by
...
P<pack-id>.<block-id> and store smaller files in big objects.
For each pack there is a corresponding metadata object that ObjectID to (offset,length) within a pack object.
2017-08-07 14:18:00 +02:00
Jarek Kowalski
2dfcd23ea9
added hashcache database, based on BoltDB which caches results of computing ObjectID for any given os.FileInfo.
2017-08-04 13:19:28 +02:00
Jarek Kowalski
21a1cbd0bb
renamed auth.Options to auth.SecurityOptions
2017-08-01 17:47:28 +02:00
Jarek Kowalski
df83cbea27
added status subcommand
2017-08-01 17:47:27 +02:00
Jarek Kowalski
cfbef13aff
Major refactoring to how repository connections are established and managed to simplify programmatic API usage and CLI.
...
Removed unnecessary and confusing options, improved the API.
2017-08-01 08:35:14 +02:00
Jarek Kowalski
095fc4aba4
removed 'client' package and merged with 'repo', names still not final
2017-08-01 08:29:42 +02:00
Jarek Kowalski
4dcdb69dcc
replaced use of Vault with Repository
2017-08-01 08:29:41 +02:00
Jarek Kowalski
50c2cd86a3
refactored credentials to separate package
2017-08-01 08:29:41 +02:00
Jarek Kowalski
d8b2ba4c24
moved most methods from Repository to casManager
2017-08-01 08:29:41 +02:00
Jarek Kowalski
5721ba8140
renamed repository.go to cas_manager.go in preparation for name flip
2017-08-01 08:29:41 +02:00
Jarek Kowalski
ef71993cde
stale comment
2017-08-01 08:29:41 +02:00
Jarek Kowalski
cda3086dea
cleanup: moved writeback to separate file
2017-08-01 08:29:41 +02:00
Jarek Kowalski
c3f0d5ddc4
moved all Vault types to 'repo' package
2017-08-01 08:29:41 +02:00
Jarek Kowalski