Jarek Kowalski
10d5102cd6
localfs: improved performance and memory usage when scanning large directories
...
- parallelized os.Lstat() x 16 (dramatically improves speed)
- discarded unused portions of os.FileInfo (uses 60% less RAM on macOS)
BEFORE:
10:47:03.670 [kopia/localfs] listed 200000 entries in 43.871211686s using 79126528 bytes of heap
After:
10:49:12.439 [kopia/localfs] listed 200000 entries in 1.953018184s using 30515200 bytes of heap
2019-08-24 11:51:16 -07:00
Jarek Kowalski
ae1510b65c
Deprecate hashcache and replace it with simply walking previous snapshot directory tree.
...
This cleans up the code a lot and removes many ugly hacks.
The performance is pretty reasonable and with separate metadata cache it's likely to stay that way.
2019-06-08 18:01:47 -07:00
Jarek Kowalski
72520029b0
golangci-lint: added more linters
...
Also fixed pre-existing lint errors.
2019-06-02 22:56:57 -07:00
Jarek Kowalski
1a7a02ddbe
cleanup imports by grouping all local imports together
2019-06-01 10:57:55 -07:00
Jarek Kowalski
63303904e1
switched remaining fmt.Errorf to errors.Wrap()
2019-06-01 10:57:05 -07:00
Jarek Kowalski
0c41d41276
Fixed up paths after merge
2019-05-27 15:48:39 -07:00
Jarek Kowalski
a6a153b22e
switched fmt.Errorf() to errors.Wrap()
2019-05-11 12:34:14 -07:00
Jarek Kowalski
fdfd336d6e
lint: fixed various linter issues
2019-04-01 18:59:48 -07:00
Jarek Kowalski
242d012a54
removed fs.EntryMetadata and replaced fs.Entry with os.FileInfo + small additions
2018-11-03 11:28:06 -07:00
Jarek Kowalski
327d8317d8
refactored repo/ into separate github.com/kopia/repo/ git repository
2018-10-26 20:40:57 -07:00
Jarek Kowalski
c999c0fba8
moved fs/repofs to snapshot/snapshotfs
2018-10-20 11:36:14 -07:00
Jarek Kowalski
906b8eaee3
plumbed through context to all manifest manager methods and switched to lazy initialization
2018-09-13 15:34:29 -07:00
Jarek Kowalski
91066f2469
reorganized low-level repository packages by moving them all under kopia/kopia/repo/
2018-08-30 22:01:05 -07:00
Jarek Kowalski
6484a55d7c
removed policy.Manager and snapshot.Manager and switched to package-level function interface instead
2018-08-30 06:47:58 -07:00
Jarek Kowalski
9060e7d310
moved repository VFS nodes into fs/repofs/ package
2018-08-29 22:56:23 -07:00
Jarek Kowalski
6587ade1f9
cli: added 'policy edit' command which launches an editor and allows user to edit policy JSON directly
2018-08-20 19:07:44 -07:00
Jarek Kowalski
1ff003c820
added internal lib for gitignore-style pattern matching
...
added 'ignorefs' which transparently hides all files according to .kopiaignore, which is very similar to .gitignore
2018-08-19 11:48:55 -10:00
Jarek Kowalski
5d73d18347
removed Parent from fs.Entry
2018-08-19 11:32:17 -10:00
Jarek Kowalski
c57531b751
switched Go logging library to github.com/op/go-logging which lets us override log levels per module
2018-07-09 21:28:12 -07:00
Jarek Kowalski
e36d02c59b
emit per-dir incomplete reason
...
store entire rootEntry on snapshot nmanifest
format 'snapshot list' consistently for both root and nested items
2018-05-19 22:00:14 -07:00
Jarek Kowalski
12c28e537d
simplified ObjectID code and changed it to a string
2018-05-09 22:22:37 -07:00
Jarek Kowalski
d8201229d8
plumbed through ctx in storage.Storage APIs and all uses
2018-04-03 17:39:54 -07:00
Jarek Kowalski
1c4e0e1e47
removed legacy oid format
2018-03-19 19:31:03 -07:00
Jarek Kowalski
453bab3560
fixed some lint errors, mostly dead code and missing error checks
2018-03-19 12:26:28 -07:00
Jarek Kowalski
10ca234633
switched to using zerolog for logging, by default log level=info, can be set to debug on commandline
2018-01-01 19:12:35 -08:00
Jarek Kowalski
31e5bce1cb
moved object management to kopia/kopia/object
2017-10-14 16:38:22 -07:00
Jarek Kowalski
371fc15694
merged 'repofs' into 'snapshot' package
2017-09-04 20:26:25 -07:00
Jarek Kowalski
b08a35e92b
refactored FS cache into reusable wrapper that can be applied on top of any filesystem abstraction
2017-09-04 17:42:50 -07:00
Jarek Kowalski
998ed1ddac
simplified MetadataManager API for listing, added caching of metadata items (unbounded for now)
...
disabled go <1.9 since we're now using t.Helper(), which is awesome
2017-08-29 21:46:51 -07:00
Jarek Kowalski
e01fcff9a4
fixed sorting of synthetic fs.Entries
2017-08-27 14:49:15 -07:00
Jarek Kowalski
2b887b0b2e
changed fs.Reader to be a Seeker too
2017-08-27 08:58:59 -07:00
Jarek Kowalski
b769dc1863
emit a warning instead of failing when unsupported directory entries are encountered
2017-08-20 13:54:54 -07:00
Jarek Kowalski
ff487d1e85
added ability to mount special object 'all' that exposes the whole set of snapshots grouped by user@host, path and time
2017-08-19 19:49:18 -07:00
Jarek Kowalski
81fd499bae
moved upload functionality to snapshot package
2017-01-30 20:07:02 -08:00
Jarek Kowalski
0195d2740a
moved directory JSON format handling to internal/dir
2017-01-30 20:04:13 -08:00
Jarek Kowalski
4b89c95721
moved hashcache to internal
2017-01-29 07:50:49 -08:00
Jarek Kowalski
bfd978a589
moved Snapshot and SnasphotSourceInfo to separate package
2017-01-28 17:33:47 -08:00
Jarek Kowalski
65bc50de5d
changed how paths are interpreted for 'kopia backups' and 'kopia expire'
2016-11-23 23:49:29 -08:00
Jarek Kowalski
a821a0053d
fixed bundling
2016-10-15 16:32:33 -07:00
Jarek Kowalski
f5cd6285d4
chanegd bundling algorithm to avoid small bundles where one would be ok
2016-10-15 16:27:37 -07:00
Jarek Kowalski
36e29f9249
added upload progress
2016-10-15 13:17:21 -07:00
Jarek Kowalski
4efc5825fa
added 'kopia cleanup' and 'kopia expire' subcommands
2016-10-04 22:14:56 -07:00
Jarek Kowalski
b21a53bccd
added backup size output
2016-10-01 10:19:38 -07:00
Jarek Kowalski
d5a78812b7
additional stats to repofs.Upload()
2016-09-26 19:23:48 -07:00
Jarek Kowalski
e322b39711
plumbed through context to repofs.Upload()
2016-09-25 21:10:39 -07:00
Jarek Kowalski
5959aa354e
added support for symlink reads to repofs
2016-09-25 10:16:58 -07:00
Jarek Kowalski
175c05d547
more repofs.Upload() API tweaks
2016-09-25 09:42:29 -07:00
Jarek Kowalski
ffda769607
replaced repofs.Uploader with static Upload() method
2016-09-24 15:27:49 -07:00
Jarek Kowalski
259c173233
merged backup.Generator and repofs.Uploader
2016-09-24 15:17:53 -07:00
Jarek Kowalski
88e3366a41
Added support for upload and download throttling for GCS.
2016-09-19 20:38:25 -07:00
Jarek Kowalski