mirror/kopia
1
0
Fork 0
mirror of https://github.com/kopia/kopia.git synced 2026-03-04 22:37:07 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
c8d1b221e2c93bbb81c9d7a3f09dabbcd684931e
kopia/internal
History
Jarek Kowalski c8d1b221e2 refactor(repository): added fs.DirectoryIterator (#3365) 
* refactor(repository): added fs.DirectoryIterator

This significantly reduces number of small allocations while
taking snapshots of lots of files, which leads to faster snapshots.

```
$ runbench --kopia-exe ~/go/bin/kopia \
   --compare-to-exe ~/go/bin/kopia-baseline --min-duration 30s \
   ./snapshot-linux-parallel-4.sh
DIFF duration: current:5.1 baseline:5.8 change:-13.0 %
DIFF repo_size: current:1081614127.6 baseline:1081615302.8 change:-0.0 %
DIFF num_files: current:60.0 baseline:60.0 change:0%
DIFF avg_heap_objects: current:4802666.0 baseline:4905741.8 change:-2.1 %
DIFF avg_heap_bytes: current:737397275.2 baseline:715263289.6 change:+3.1 %
DIFF avg_ram: current:215.0 baseline:211.5 change:+1.6 %
DIFF max_ram: current:294.8 baseline:311.4 change:-5.3 %
DIFF avg_cpu: current:167.3 baseline:145.3 change:+15.1 %
DIFF max_cpu: current:227.2 baseline:251.0 change:-9.5 %
```

* changed `Next()` API

* mechanical move of the iterator to its own file

* clarified comment

* pr feedback

* mechanical move of all localfs dependencies on os.FileInfo to a separate file

* Update fs/entry.go

Co-authored-by: ashmrtn <3891298+ashmrtn@users.noreply.github.com>

* Update fs/entry_dir_iterator.go

Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>

* doc: clarified valid results from Next()

---------

Co-authored-by: ashmrtn <3891298+ashmrtn@users.noreply.github.com>
Co-authored-by: Julio Lopez <1953782+julio-lopez@users.noreply.github.com>
2023-10-05 02:45:44 +00:00
..
acl
feat(repository): apply retention policies server-side (#3249)
2023-09-02 18:23:21 -07:00
apiclient
feat(server): add server support for unix domain sockets (#3216)
2023-08-20 17:17:20 -07:00
atomicfile
chore(ci): upgraded linter to 1.48.0 (#2294)
2022-08-09 06:07:54 +00:00
auth
feat(repository): apply retention policies server-side (#3249)
2023-09-02 18:23:21 -07:00
bigmap
chore(ci): upgrade golang to 1.19.2 and linter to 1.50.1 (#2526)
2022-10-28 11:02:47 -07:00
blobcrypto
fix(general): correct spelling mistakes (#2684)
2023-01-21 07:37:15 -08:00
blobtesting
fix(providers): fixed list/get caching with rclone providers (#3284)
2023-09-09 13:07:53 -07:00
cache
feat(repository): added hard size limit to the on-disk cache (#3238)
2023-08-24 09:38:56 -07:00
cachedir
refactor(repository): moved format blob management to separate package (#2245)
2022-07-30 14:13:52 -07:00
cacheprot
feat(repository): added hard size limit to the on-disk cache (#3238)
2023-08-24 09:38:56 -07:00
clock
fix(general): correct spelling mistakes (#2684)
2023-01-21 07:37:15 -08:00
completeset
chore(ci): upgraded linter to 1.48.0 (#2294)
2022-08-09 06:07:54 +00:00
connection
refactor(repository): replaced atomic values with Go 1.19 atomic wrappers (#2590)
2022-11-19 18:39:04 +00:00
crypto
refactor(repository): Move encryption into a common package in preparation to handle secrets (#3351)
2023-09-24 17:44:50 -07:00
ctxutil
chore(ci): upgraded linter to 1.48.0 (#2294)
2022-08-09 06:07:54 +00:00
diff
refactor(repository): added fs.DirectoryIterator (#3365)
2023-10-05 02:45:44 +00:00
dirutil
feat(providers): improved sharded directory creation (#1665)
2022-01-23 14:56:35 -08:00
editor
chore(ci): upgraded linter to 1.53.3 (#3079)
2023-06-18 13:26:01 -07:00
epoch
fix(repository): skip epoch compaction and advancement on read-only storage (#3225)
2023-08-21 15:29:13 -07:00
faketime
refactor(repository): replaced atomic values with Go 1.19 atomic wrappers (#2590)
2022-11-19 18:39:04 +00:00
fault
fix(deps-dev): fixup mutex for make check-locks (#2920)
2023-04-11 15:58:20 -04:00
feature
feat(repository): added required features to the repository (#2220)
2022-07-29 09:31:17 -07:00
freepool
test(general): added unit tests for freepool (#2930)
2023-04-13 18:58:16 -07:00
fshasher
chore(ci): fixes for ARM64 runner on AWS (#3179)
2023-07-27 11:30:40 -07:00
fusemount
refactor(repository): added fs.DirectoryIterator (#3365)
2023-10-05 02:45:44 +00:00
gather
feat(providers): improve performance of SFTP uploads (#3235)
2023-08-19 18:07:43 -07:00
grpcapi
feat(repository): apply retention policies server-side (#3249)
2023-09-02 18:23:21 -07:00
hmac
chore(ci): upgraded linter to 1.48.0 (#2294)
2022-08-09 06:07:54 +00:00
impossible
cache: improved test coverage (#1582)
2021-12-13 14:25:10 -08:00
indextest
chore(ci): upgraded linter to 1.48.0 (#2294)
2022-08-09 06:07:54 +00:00
iocopy
test(general): added tests for iocopy (#2932)
2023-04-14 16:57:57 -07:00
listcache
fix(general): correct spelling mistakes (#2684)
2023-01-21 07:37:15 -08:00
logfile
Potential fix for TestLogFileMaxTotalSize. (#3140)
2023-07-11 20:31:24 +00:00
metricid
feat(repository): added internal/metricid package (#2626)
2022-12-05 04:41:42 +00:00
metrics
chore(general): fixed remaining checklocks violations (#2939)
2023-04-13 20:11:36 -07:00
mockfs
refactor(repository): added fs.DirectoryIterator (#3365)
2023-10-05 02:45:44 +00:00
mount
chore(ci): upgraded linter to 1.53.3 (#3079)
2023-06-18 13:26:01 -07:00
osexec
feat(providers): ensure Ctrl-C is not passed to rclone (#1951)
2022-05-16 05:28:57 +00:00
ospath
fix(general): Fixed ordering in function "isAbs" in ospath (#2850)
2023-03-31 04:19:14 +00:00
ownwrites
chore(ci): upgraded linter to 1.48.0 (#2294)
2022-08-09 06:07:54 +00:00
parallelwork
test(general): fixed TestOnNthCompletion test flake, which was 1-5/10000 runs (#3263)
2023-09-02 16:04:36 -07:00
passwordpersist
chore(ci): upgraded linter to 1.53.3 (#3079)
2023-06-18 13:26:01 -07:00
providervalidation
fix(providers): fixed list/get caching with rclone providers (#3284)
2023-09-09 13:07:53 -07:00
releasable
chore(ci): upgraded linter to 1.48.0 (#2294)
2022-08-09 06:07:54 +00:00
remoterepoapi
feat(repository): apply retention policies server-side (#3249)
2023-09-02 18:23:21 -07:00
repolog
refactor(repository): extract parts repo/content into packages (#2651)
2022-12-17 16:19:12 +00:00
repotesting
fix(repository): remove metrics.Registry from repo.Repository (#2944)
2023-04-14 19:16:10 -07:00
retry
chore(ci): upgraded linter to 1.53.3 (#3079)
2023-06-18 13:26:01 -07:00
scheduler
feat(server): introduced generic scheduler (#3293)
2023-09-12 07:37:41 -07:00
scrubber
scrubber: added unit tests (#1581)
2021-12-13 13:33:15 -08:00
server
fix(cli): Convert --run-missed from bool to Enum (#3337)
2023-09-22 10:18:19 -07:00
serverapi
feat(ui): Allow specifying whether local or all snapshots are shown by default in the UI (#3289)
2023-09-11 22:49:36 -07:00
servertesting
test(repository): server testability refactoring (#2612)
2022-12-01 06:27:52 +00:00
sparsefile
chore(ci): upgraded linter to 1.51.1, minor style fixes (#2748)
2023-02-09 06:47:36 +00:00
stat
chore(ci): upgraded linter to 1.53.3 (#3079)
2023-06-18 13:26:01 -07:00
stats
refactor(repository): replaced atomic values with Go 1.19 atomic wrappers (#2590)
2022-11-19 18:39:04 +00:00
tempfile
feat(general): always try O_TMPFILE on Linux when creating temporary files (#2407)
2022-09-14 09:48:02 -07:00
testlogging
chore(ci): upgraded linter to 1.48.0 (#2294)
2022-08-09 06:07:54 +00:00
testutil
feat(server): add server support for unix domain sockets (#3216)
2023-08-20 17:17:20 -07:00
timestampmeta
chore(ci): upgrade golang to 1.19.2 and linter to 1.50.1 (#2526)
2022-10-28 11:02:47 -07:00
timetrack
refactor(repository): replaced atomic values with Go 1.19 atomic wrappers (#2590)
2022-11-19 18:39:04 +00:00
tlsutil
chore(ci): upgraded linter to 1.53.3 (#3079)
2023-06-18 13:26:01 -07:00
uitask
feat(server): fixed server logging file & console (#3262)
2023-09-02 18:10:18 -07:00
units
feat(cli): Support displaying storage values in base-2 [#2492] (#2502)
2022-10-24 19:00:36 -07:00
user
feat(repository): added RepositoryWriter.ReplaceManifests (#2747)
2023-03-12 00:44:28 +00:00
wcmatch
fix(general): correct spelling mistakes (#2684)
2023-01-21 07:37:15 -08:00
webdavmount
refactor(repository): added fs.DirectoryIterator (#3365)
2023-10-05 02:45:44 +00:00
workshare
refactor(repository): replaced atomic values with Go 1.19 atomic wrappers (#2590)
2022-11-19 18:39:04 +00:00
zaplogutil
chore(ci): upgraded linter to 1.48.0 (#2294)
2022-08-09 06:07:54 +00:00