Compare commits

...

586 Commits

Author SHA1 Message Date
Jakob Borg
5f8c0ca932 cmd/syncthing: Fix incorrect shadowing preventing first startup (fixes #4471) 2017-10-28 21:15:32 +02:00
Simon Frei
2953fe40d1 lib/model: Add initial noop watch cancel func (fixes #4464)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4465
2017-10-26 13:50:30 +02:00
Audrius Butkevicius
ff0a83fe5b vendor: Fix kcp deadlock
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4461
2017-10-25 21:57:51 +02:00
Simon Frei
dc42db444b lib/model, lib/config: Refactor folder health/error handling (fixes #4445, fixes #4451)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4455
LGTM: AudriusButkevicius, calmh
2017-10-24 07:58:55 +00:00
Audrius Butkevicius
a9c221189b lib/connections: Stun resolves server adress beforehand (fixes #4453)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4454
2017-10-22 18:48:06 +00:00
Audrius Butkevicius
b2966957e0 lib/connections: Add KCP blacklist period 2017-10-22 13:56:52 +01:00
Audrius Butkevicius
0d30166357 lib/connections: Use own KCP fork, move listener setup earlier (ref #4446)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4452
2017-10-22 12:36:36 +00:00
Jakob Borg
d65f1fb08a lib/config: Improve debug logging around folder marker 2017-10-22 00:04:51 +02:00
Audrius Butkevicius
622b614f31 all: Ignore Sync errors on directories (fixes #4432)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4449
2017-10-21 22:00:46 +00:00
Audrius Butkevicius
b1ade6d0c0 gui: Do not prompt for UR changes if disabled (fixes #4444)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4448
2017-10-21 18:46:07 +00:00
Simon Frei
46becc5338 build: Handle split GOPATH
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4447
2017-10-21 17:37:06 +00:00
Simon Frei
20fac4bb80 main: Improve logging for initial config loading (ref #4431)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4436
2017-10-21 09:00:24 +00:00
Simon Frei
d84b6bb822 build: Warn when being executed outside of gopath
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4425
2017-10-20 23:10:55 +00:00
Simon Frei
55b63941b8 cmd/syncthing: Add fswatcher and remove useAPIKey stats
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4438
2017-10-20 16:25:20 +00:00
Simon Frei
e70003737b lib/fs: make watcher tests even more darwin slowness resistant
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4439
2017-10-20 15:59:18 +00:00
Michael Ploujnikov
f98c21b68e all: Add filesystem notification support
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3986
2017-10-20 14:52:55 +00:00
Simon Frei
c704ba9ef9 gui: "versioner" -> "versioning" and add missing translations
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4435
2017-10-20 13:55:22 +00:00
Jakob Borg
dfad6a2aa9 gui, man: Update docs & translations 2017-10-18 07:45:20 +02:00
Audrius Butkevicius
fb7264a663 cmd/syncthing: Enable KCP by default
Also, use upstream library, as my changes have been merged.
2017-10-17 23:17:10 +01:00
Audrius Butkevicius
889814a1af gui: Add uncamel filter
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4428
LGTM: imsodin, calmh
2017-10-17 07:56:36 +00:00
Simon Frei
694a7de59d build: Correct import paths for vet tools
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4420
2017-10-15 10:39:50 +00:00
Audrius Butkevicius
059185b325 cmd/syncthing: Expand usage stats even more (ref #3628)
Also add diffing functionality

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4422
2017-10-15 07:45:15 +00:00
Audrius Butkevicius
1e9e9cbebb cmd/syncthing: Uptime should be an integer 2017-10-14 18:22:30 +01:00
Felix Ableitner
cdbb32d0f0 build: Support passing -pkgdir
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4417
2017-10-13 09:24:38 +00:00
Audrius Butkevicius
becbb3b123 lib/model: Fix tests not to require Go 1.9 2017-10-12 22:36:20 +01:00
Jakob Borg
06da67e6fc issue_template: Clarify what to post and not post
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4416
2017-10-12 07:14:21 +00:00
Jakob Borg
2f08f8021f lib/beacon: Don't exit after a single write failure (fixes #4414)
With VPNs and stuff we can get a single failure on an interface that
supposedly supports broadcasts without it being fatal.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4415
2017-10-12 07:13:44 +00:00
Audrius Butkevicius
9d3f3847ed lib/model: Fix removal of paused folders, improve tests (fixes #4405) 2017-10-12 08:23:33 +02:00
Audrius Butkevicius
74c8d34805 lib/model: Centralize error reporting, modified files are errors (fixes #4392) 2017-10-12 08:23:33 +02:00
Audrius Butkevicius
5ec1490be0 lib/fs: Ignore directory fsync failures 2017-10-12 08:22:29 +02:00
Audrius Butkevicius
2760d032ca cmd/syncthing: Add more stats to usage reports (ref #3628)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4347
2017-10-12 06:16:46 +00:00
Jakob Borg
813e6ddf83 gui, man: Update docs & translations 2017-10-11 07:45:19 +02:00
Matteo Ruina
6ffb95f6c8 etc: Add FreeBSD rc.d script
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4408
2017-10-08 17:29:51 +00:00
Jakob Borg
0b5c11bf93 gui, man: Update docs & translations 2017-10-04 07:45:19 +02:00
Tobias Tom
5aade9a4a5 Add -device-id command line option (fixes #4387)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4390
2017-09-25 06:05:21 +00:00
Jakob Borg
9717c3d292 authors: Add tobiastom 2017-09-25 07:51:21 +02:00
Jakob Borg
a365ae51c4 lib/model: Hide temporary files on Windows while they are in use (fixes #4382)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4394
2017-09-23 13:47:51 +00:00
Jakob Borg
97222797a0 lib/config: Make folder marker change non fatal 2017-09-23 15:29:55 +02:00
Simon Frei
e588bb29b9 lib/model: remove unused folderFs member from Model
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4389
2017-09-21 09:03:36 +00:00
Jakob Borg
2dd9450793 lib/protocol, vendor: Import luhn code directly
I've changed it incompatibly to fix a correctness bug. Nonetheless, we
should remain incorrect indefinitely.
2017-09-20 21:34:32 +02:00
Jakob Borg
3ee12464b4 lib/config, lib/model: Make sure to hide our special files (fixes #4382)
The folder marker conversion forgot to hide the .stfolder. This adds
that, for those who have not yet been converted.

Also adds Hide() calls to the folder start, to mend historical
unhidedness. (I'm sure this will upset someone who is manually managing
their .stignores in the other direction...)

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4384
2017-09-20 06:49:04 +00:00
Jakob Borg
59ebcea356 gui, man: Update docs & translations 2017-09-20 07:45:17 +02:00
Sly_tom_cat
27d4896a13 gui: Fix title attribute for popups on shared folders / devices (fixes #4377)
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4378
2017-09-19 15:37:26 +00:00
Simon Frei
1088eb12ea lib/model: Fix logging inconsistencies (fixes #4375)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4376
2017-09-18 11:56:19 +00:00
Jakob Borg
f40d219370 gui, man: Update docs & translations 2017-09-13 07:45:17 +02:00
Jakob Borg
429cc20eb7 cmd/syncthing: Add some common security releated HTTP headers (fixes #4360)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4361
2017-09-10 08:28:12 +00:00
Audrius Butkevicius
e85ce7c94e lib/model: Support removing paused folders (fixes #4357)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4358
LGTM: imsodin, calmh
2017-09-09 15:08:59 +00:00
Jakob Borg
283c8d95e2 lib/protocol: Comment typo 2017-09-08 15:25:14 +02:00
wangguoliang
a9aa375109 lib/fs: Comment typo
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4354
2017-09-07 09:40:46 +00:00
Simon Frei
f7d2c58783 lib/model: Deduplicate folder loops
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4352
LGTM: AudriusButkevicius, calmh
2017-09-07 06:17:47 +00:00
Jakob Borg
4d3e0de4ba cmd/syncthing, lib/sha256: Skip CPU benchmarks when user decided (fixes #4348)
When STHASHING is set, don't benchmark as it's already decided. If weak
hashing isn't set to "auto", don't benchmark that either.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4349
2017-09-06 06:55:47 +00:00
Simon Frei
9dbc509996 lib/ignore: Consistent behaviour for nil *Matcher
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4310
2017-09-06 06:39:18 +00:00
Jakob Borg
baec3f909c gui, man: Update docs & translations 2017-09-06 07:45:19 +02:00
Simon Frei
c41aaad3bb lib/ignore: Ignore duplicate lines in .stignore
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4350
LGTM: AudriusButkevicius, calmh
2017-09-04 12:46:19 +00:00
Jakob Borg
9682bbfbda lib/protocol: Optimize luhn and chunk functions
These functions were very naive and slow. We haven't done much about
them because they pretty much don't matter at all for Syncthing
performance. They are however called very often in the discovery server
and these optimizations have a huge effect on the CPU load on the
public discovery servers.

The code isn't exactly obvious, but we have good test coverage on all
these functions.

benchmark                 old ns/op     new ns/op     delta
BenchmarkLuhnify-8        12458         1045          -91.61%
BenchmarkUnluhnify-8      12598         1074          -91.47%
BenchmarkChunkify-8       10792         104           -99.04%

benchmark                 old allocs     new allocs     delta
BenchmarkLuhnify-8        18             1              -94.44%
BenchmarkUnluhnify-8      18             1              -94.44%
BenchmarkChunkify-8       44             2              -95.45%

benchmark                 old bytes     new bytes     delta
BenchmarkLuhnify-8        1278          64            -94.99%
BenchmarkUnluhnify-8      1278          64            -94.99%
BenchmarkChunkify-8       42552         128           -99.70%

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4346
2017-09-03 10:26:12 +00:00
Audrius Butkevicius
9e6a1fdcd4 vendor: Update kcp, removes closeConn (fixes #4343) 2017-09-02 16:11:48 +02:00
Jakob Borg
49bddfbe53 cmd/syncthing: Add test for truncate behavior of log file (ref #4255)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4342
2017-09-02 06:56:35 +00:00
Simon Frei
55e0ac3e24 lib/model: Make puller retrying and logging less aggressive
Currently all errors during pulling and the first of these errors again on
finishing are logged to info. Besides that the errors logged when finishing
are stored in f.errors. This PR moves all logging during pulling to the debug
channel (they might still be relevant in some obscure debugging case) and
uses the stored errors to log the main error per fail when all pulling
iterations are done and failed.

Additional instead of trying 11 times it now only tries 3 times.

This is the first part of what is discussed here:
https://forum.syncthing.net/t/reduce-verboseness-of-puller/10261

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4338
2017-09-02 06:05:55 +00:00
Audrius Butkevicius
cbcc3ea132 lib/connections: Use our own fork of kcp (fixes #4063)
This updates kcp and uses our own fork which:

1. Keys sessions not just by remote address, but by remote address +
conversation id 2. Allows not to close connections that were passed directly
to the library. 3. Resets cache key if the session gets terminated.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4339
LGTM: calmh
2017-09-02 06:04:35 +00:00
Audrius Butkevicius
ab132ff6fe lib: Folder marker is now a folder
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4341
LGTM: calmh
2017-09-02 05:52:38 +00:00
Jakob Borg
19e52a10df readme: Update badges for new build server 2017-09-01 14:23:17 +02:00
Jakob Borg
a8145e187f lib/model: Fixup because I suck and shouldn't commit directly to master 2017-08-31 10:49:17 +02:00
Jakob Borg
e33fa10115 lib/model: Use same batch size constants in db updater as for protocol 2017-08-31 10:47:39 +02:00
Jakob Borg
4b6e7e7867 lib/tlsutil: Remove undesired bufio from UnionedConnection (ref #4245)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4335
2017-08-31 07:34:48 +00:00
Jakob Borg
70d121a94b cmd/strelaysrv: Smaller, adjustable network buffer 2017-08-30 18:52:28 +02:00
Jakob Borg
33ffb07d31 cmd/strelaysrv: Don't leak tickers 2017-08-30 18:46:50 +02:00
Jakob Borg
7aaa92ac47 cmd/strelaysrv: Add profiling support, default disabled 2017-08-30 16:07:15 +02:00
Jakob Borg
5883eb9a25 gui, man: Update docs & translations 2017-08-30 10:50:19 +02:00
MaximAL
e60efa2b3d gui: Correct quotes in title attribute
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4332
2017-08-30 06:02:46 +00:00
Simon Frei
ddf6d64faa lib/model: Create folders via newFolder
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4329
2017-08-25 19:47:01 +00:00
Simon Frei
c7221b035d gui: Round down in devices completion (consistent with folders)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4325
2017-08-24 04:26:12 +00:00
Audrius Butkevicius
a69ba18f62 lib/model: Some platforms do not support usage checks (fixes #4321)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4322
2017-08-22 18:13:58 +00:00
Jakob Borg
b31611a8d1 gui, man: Update docs & translations
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4320
2017-08-22 09:00:52 +00:00
Simon Frei
0ca0e3e9bd lib/model: GetIgnores: Don't return error for no .stignore file
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4311
2017-08-22 06:48:25 +00:00
Audrius Butkevicius
0a96a1150b lib/model, lib/ignores: Properly handle out of folder ignores and free space checks (fixes #4313, fixes #4314)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4318
2017-08-22 06:45:00 +00:00
Audrius Butkevicius
b8c249cddc lib/model: Move stale scan check info finisher (ref #4305, fix #3742)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4317
2017-08-22 06:42:09 +00:00
Audrius Butkevicius
e8ba6d4771 lib/upnp: Fix build
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4316
2017-08-21 11:41:40 +00:00
Audrius Butkevicius
606fce09ca lib/upnp: Disable confusing messages
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4312
2017-08-21 10:03:25 +00:00
Audrius Butkevicius
3d8b4a42b7 all: Convert folders to use filesystem abstraction
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4228
2017-08-19 14:36:56 +00:00
Simon Frei
ab8c2fb5c7 lib/model: Fix race in GetIgnores (fixes #4300)
In addition this function returned an error when .stignore file was not
present, which is perfectly valid. Also removed inconsistent nil check in
ignores.go (only relevant for tests) and adjusted walk.go to do what it says
in comments (check if Matcher is nil) to prevent nil deref in tests.

Originally reported in:
https://forum.syncthing.net/t/reason-for-panic-maybe-too-little-ram/10346

Regression from #3996

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4301
2017-08-12 17:10:43 +00:00
Simon Frei
77578e8aac gui: Don't set default path editing existing folders without label (fixes #4297)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4298
LGTM: calmh
2017-08-10 15:31:25 +00:00
Jakob Borg
1fc2ab444b lib/model: Remove ineffective symlink recovery attempt 2017-08-08 15:30:28 +02:00
Jakob Borg
fa5c890ff6 lib/versioner: Clean the versions dir of symlinks, not the full folder
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4289
2017-08-08 13:13:08 +00:00
Jakob Borg
a3c17f8f81 lib/model: Disable symlink attack test on Windows 2017-08-08 08:05:24 +02:00
Jakob Borg
f1f21bf220 lib/model, lib/versioner: Prevent symlink attack via versioning (fixes #4286)
Prior to this, the following is possible:

- Create a symlink "foo -> /somewhere", it gets synced
- Delete "foo", it gets versioned
- Create "foo/bar", it gets synced
- Delete "foo/bar", it gets versioned in "/somewhere/bar"

With this change, versioners should never version symlinks.
2017-08-07 07:57:10 +02:00
MaximAL
54155cb42d gui: Add title attributes for shared devices/folders
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4266
LGTM: AudriusButkevicius
2017-07-27 17:31:14 +00:00
Jakob Borg
414c58174b build: Move -installsuffix behind an explicit option (fixes #4272) 2017-07-27 12:55:07 +02:00
Audrius Butkevicius
94acc20dd6 cmd/strelaysrv: Fix a few connection and routine leaks (fixes #4245)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4273
2017-07-26 19:18:00 +00:00
NoLooseEnds
8e9119eedf assets: Add Mac folder icon
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4238
2017-07-20 13:23:54 +00:00
Simon Frei
a04b92332f gui, lib/config: Add default path for new folders (fixes #2157)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4192
2017-07-20 13:16:54 +00:00
HairyFotr
0ad10b0fee all: Typos
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4263
2017-07-20 13:10:46 +00:00
Ross Smith II
0ca2ed7ad7 build: Use maximum compression when archiving
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4229
2017-07-17 13:20:13 +00:00
Jakob Borg
fa4226cae4 authors: Add rasa 2017-07-17 15:06:47 +02:00
Jakob Borg
cc63236a2e build: Use Go's default GOPATH if it's valid
As of Go 1.8 it's valid to not have a GOPATH set. We ask the Go tool
what the GOPATH seems to be, and if we find ourselves (or a valid copy
of ourselves...) in that location we do not fiddle with the GOPATH.
2017-07-16 21:39:29 +01:00
Jakob Borg
6623657ef3 build: The default build step already includes linting 2017-07-15 17:06:22 +02:00
Jose Manuel Delicado
4405117bea gui: HTML accessibility updates
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4258
LGTM: calmh
2017-07-15 09:54:37 +00:00
Jakob Borg
e371800878 authors: Fixup jmdaweb 2017-07-14 07:53:18 +02:00
Jakob Borg
7a47646534 authors: Add jmdaweb 2017-07-14 07:49:08 +02:00
Jakob Borg
d475ad7ce1 gui, man: Update docs & translations 2017-07-13 08:55:12 +02:00
Jakob Borg
7c8418f493 build: Support builds outside of GOPATH
This adds support for building with the source placed anywhere and no
GOPATH set. The build script handles this by creating a temporary GOPATH
in the system temp dir (or another specified location) and mirroring the
source there before building. The resulting binaries etc still end up in
the same place as usual, meaning at least the "build", "install", "tar",
"zip", "deb", "snap", "test", "vet", "lint", "metalint" and "clean"
commands work without a GOPATH. To this end these commands internally
use fully qualified package paths like
"github.com/syncthing/syncthing/cmd/..." instead of "./cmd/..." like
before.

There is a new command "gopath" that prepares and echoes the directory
of the temporary GOPATH. This can be used to run other non-build go
commands:

export GOPATH=$(go run build.go gopath)  // GOPATH is now set
go test -v -race github.com/syncthing/syncthing/cmd/...

There is a new option "-no-build-gopath" that prevents the
check-and-copy step, instead assuming the temporary GOPATH is already
created and up to date. This is a performance optimization for build
servers running multiple builds commands in sequence:

go run build.go gopath // creates a temporary GOPATH
go run build.go -no-build-gopath -goos=... tar // reuses GOPATH
go run build.go -no-build-gopath -goos=... tar // reuses GOPATH

The temporary GOPATH is placed in the system temporary directory
(os.TempDir()) unless overridden by the STTMPDIR variable. It is named
after the hash of the current directory where build.go is run. The
reason for this is that the name should be unique to a source checkout
without risk for conflict, but still persistent between runs of
build.go.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4253
LGTM: AudriusButkevicius, imsodin
2017-07-11 07:57:58 +00:00
Jakob Borg
200a7fc844 meta: Move metadata checks into meta directory, make them tests
This moves a few things from script/ to a new directory meta/, and makes
them real Go tests. These are the authors, copyright, metalint and gofmt
checks. That means that they can now be run by

go test -v ./meta

and optionally filtered by the usual -run thing to go test. Also -short
will cut down on the metalint stuff and exclude the authors check (which
is slow because it runs git lots of times).

Mainly this makes everything easier on things like build servers where
we can now just run tests instead of do a bunch of scripting.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4252
2017-07-07 20:43:26 +00:00
Jakob Borg
5a38e0ba3f script: Trivial lint error in changelog.go 2017-07-07 21:56:12 +02:00
Jakob Borg
c77490c32d authors: Fixup author email mistakes 2017-07-07 21:42:50 +02:00
Simon Frei
b75c9f2bbb lib/ignores: Don't add text from includes to lines (fixes #4249)
Otherwise all the lines from includes will be shown in the web UI instead of
just the #include ... line. This problem was introduced in #3996.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4248
LGTM: calmh
2017-07-06 11:44:11 +00:00
Siyuan Liu
322bedbb04 gui: Show remaining bytes in remote device panel (fixes #4227)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4246
LGTM: AudriusButkevicius, calmh
2017-07-05 09:19:29 +00:00
Jakob Borg
487655b365 gui, man: Update docs & translations 2017-07-05 07:45:22 +02:00
Siyuan Liu
03c678a810 lib/versioner: Interpret versions path relative to folder path (fixes #4188)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4243
2017-07-03 14:50:51 +00:00
Jakob Borg
b79f8aceb8 authors: Fixup liusy182 2017-07-03 16:40:02 +02:00
Jakob Borg
92e8c4303a authors: Add liusy182 2017-07-03 16:33:41 +02:00
Jakob Borg
e735a3a25c build: Builds from "release" branch are not branch builds 2017-06-30 14:02:16 +02:00
Simon Frei
8d13e01342 gui: Fix discovered devices list (follow-up to #4186)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4239
2017-06-29 13:47:42 +00:00
Jakob Borg
34f8cc8620 gui: Rename assets/lang/README-FIRST to README for greater visibility
README is rendered in the GitHub web UI
2017-06-28 18:58:41 +02:00
Jakob Borg
add10c98fa lib/db: Fix test for Go 1.9's smarter time.Time values 2017-06-28 14:36:04 +02:00
Jakob Borg
6503326073 gui, man: Update docs & translations 2017-06-28 07:45:20 +02:00
Jakob Borg
db1dc9985a lib/upgrade: 0.x to 1.0 is a minor upgrade
This removes the special handling of minor versions as major when the
actual major is zero, and adds the special case that upgrades from 0.x
to 1.x are considered minor. 0.x to 2.x or 1.x to 2.x etc are still
considered major.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4226
2017-06-25 14:17:43 +00:00
Jakob Borg
3641c97667 gui, man: Update docs & translations 2017-06-21 07:45:20 +02:00
Jakob Borg
8f214fe4a9 lib/config: Ignored folders that are added to the config should not remain ignored (fixes #4219)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4220
2017-06-17 18:40:28 +00:00
Simon Frei
98cfc204ca gui: Add dropdown menu to choose discovered devices to device modal (fixes #4157)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4186
LGTM: AudriusButkevicius, calmh
2017-06-16 15:27:09 +00:00
Jakob Borg
d0061c172c gui, man: Update docs & translations 2017-06-14 07:45:21 +02:00
Jakob Borg
93a04158fd vendor: Update github.com/gogo/protobuf, keeping all files
gvt fetch -a, because we need the protobuf files etc for regeneration
2017-06-14 05:24:09 +02:00
Jakob Borg
d862e79133 gui: Fix minor breakage from 9c417571 2017-06-12 08:54:08 +02:00
Jakob Borg
68c1a0b9b4 lib/ignore, lib/model: Use an interface to detect file changes, improving tests
This solves the erratic test failures on model.TestIgnores by ensuring
that the ignore patterns are reloaded even in the face of unchanged
timestamps.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4208
2017-06-11 10:27:12 +00:00
Jakob Borg
2a38d2a3d2 build: Build Debian packages for non-syncthing targets (fixes #4193)
Things like the package name "syncthing" was hardcoded, which is not
awesome. With this in place we can build debs called syncthing-discosrv,
syncthing-relaysrv and syncthing-relaypoolsrv. I don't actually intend
to build and publish the relaypoolsrv, but the others can be good. Using
the "syncthing-" prefix to make the obvious "apt-cache search syncthing"
actually show them etc.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4206
LGTM: AudriusButkevicius, calmh
2017-06-09 08:53:45 +00:00
Simon Frei
9c4175715a gui: Reset ignore input field when adding remote share (fixes #4203)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4204
LGTM: AudriusButkevicius, calmh
2017-06-09 05:41:19 +00:00
Audrius Butkevicius
d637148cca cmd/strelaypoolsrv: Show popup on row hover
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4201
2017-06-07 22:06:00 +00:00
snugghash
3395992abd gui: Make icons and directory information in local device summary consistent with folders (fixes #4100)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4200
2017-06-07 10:34:45 +00:00
Jakob Borg
7a15fef3b8 authors, nicks: Add snugghash 2017-06-07 12:23:16 +02:00
Audrius Butkevicius
9667a0a618 lib/connections: Fix race (fixes #4177) 2017-06-07 10:17:01 +02:00
Jakob Borg
7346113742 gui, man: Update docs & translations 2017-06-07 07:45:21 +02:00
Lars K.W. Gohlke
3f1fa04725 build, jenkins: Move test coverage stuff to jenkins/
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4178
2017-06-02 07:18:59 +00:00
Simon Frei
719c313b23 cmd/syncthing, lib/logger: Set debug flags on SetDebug for tests
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4187
2017-06-02 07:04:06 +00:00
Jakob Borg
3d11efc9e0 gui, man: Update docs & translations 2017-06-01 12:25:17 +02:00
Jakob Borg
3dca7cd694 build: Update prerelease command 2017-06-01 12:09:20 +02:00
Jakob Borg
803da92ca9 cmd/syncthing: Start CPU usage monitoring not from init (fixes #4183)
Starting stuff from init() is an antipattern, and the innerProcess
variable isn't 100% reliable. We should sort out the other uses of it as
well in due time.

Also removing the hack on innerProcess as I happened to see it and the
affected versions are now <1% users.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4185
2017-05-31 18:14:04 +00:00
nrm21
b49bbe82dd gui, lib/config, lib/model: Add ability to ignore folders offered by other nodes (fixes #3993)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4179
LGTM: AudriusButkevicius, calmh
2017-05-31 18:04:00 +00:00
Jakob Borg
3959eb26fb lib/model: Events should have "folder" key, not "folderID" 2017-05-30 08:57:18 +02:00
nrm21
1235cead35 lib/model: Add name of latest modifying device to conflict file (fixes #3524)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4161
LGTM: AudriusButkevicius, calmh
2017-05-25 10:26:41 +00:00
Simon Frei
dd6bb6d5fd gui: Reset warnings about nested folder roots (fixes #3433)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4173
2017-05-23 19:54:56 +00:00
Audrius Butkevicius
91d37f35bc lib/model: Use up to date device name, do not provide name to unknown devices (fixes #4164)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4168
2017-05-22 19:58:33 +00:00
Audrius Butkevicius
51518490c6 lib/connections: Fix KCP from locking up the service (fixes #4072) 2017-05-21 22:16:21 +02:00
Audrius Butkevicius
2c10beed0b gui: Add (?d) and (?i) to ignores guide 2017-05-21 22:16:21 +02:00
Jakob Borg
8f3f787a34 gui, man: Update docs & translations 2017-05-18 07:47:51 +02:00
Darshil Chanpura
3522d451df gui: Added a CSS rule for out of sync items modal
Removed text decoration on hover which was shown while hovering on span with eject icon.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4135
2017-05-16 08:56:54 +00:00
Jakob Borg
cf3114b56d authors: Amend dtchanpura 2017-05-16 10:49:48 +02:00
Jakob Borg
3d95135638 authors: Add dtchanpura 2017-05-16 10:47:29 +02:00
Jakob Borg
4db662e576 gui: Ensure failed items folder path contains separator (fixes #4143)
Windows specific, due to how we handle folder paths.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4149
2017-05-15 06:32:16 +00:00
Jakob Borg
1d15b8be9b cmd/syncthing: Warn when running as a super user (fixes #4123)
UID 0 on Unixes, SYSTEM SID on Windows.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4148
2017-05-15 05:42:21 +00:00
Jakob Borg
d25b15263a cmd/syncthing: Process handle should be closed 2017-05-14 18:06:27 +02:00
Wulf Weich
7931d956f7 gui: fix path breaking out of global changes modal (fixes #3895)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4146
2017-05-13 10:42:37 +00:00
Elliot Huffman
c9afabf09f etc: Update systemd docs URL
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4145
2017-05-13 08:01:53 +00:00
Jakob Borg
c262f48bfe Merge branch 'release'
* release:
  cmd/syncthing: Fix CPU usage reporting (tick time)
2017-05-06 17:37:16 +02:00
Jakob Borg
8c108b4d20 cmd/syncthing: Fix CPU usage reporting (tick time) 2017-05-06 17:36:12 +02:00
Jakob Borg
ec137c9522 cmd/syncthing: Fix CPU usage reporting (tick time) 2017-05-06 17:35:07 +02:00
Jakob Borg
b17d7d8126 Merge branch 'release'
* release:
  cmd/syncthing: Report correct CPU usage on Windows (fixes #4133)
2017-05-05 16:20:35 +02:00
Jakob Borg
43569d8d36 cmd/syncthing: Report correct CPU usage on Windows (fixes #4133) 2017-05-05 16:16:43 +02:00
Jakob Borg
c4b527e5e9 cmd/syncthing: Report correct CPU usage on Windows (fixes #4133) 2017-05-05 16:12:59 +02:00
Jakob Borg
6386d079b0 gui: Don't reload concurrently with saving config when changing theme (fixes #4127) 2017-05-05 16:12:59 +02:00
Ben S
d2699a20fc gui: Display global changes timestamps in 24h format
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4130
2017-05-04 10:18:47 +00:00
Simon Frei
0b854dff9d lib/ignore: Don't match root (".")
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4122
2017-05-01 16:58:08 +00:00
Simon Frei
9de6cdddfd script: Ignore units and allow translated strings as translate-values
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4120
2017-04-29 15:48:00 +00:00
Jakob Borg
5045842f4f lib/config: Don't XML serialize deprecated minDiskFreePct 2017-04-27 14:46:19 +09:00
Jakob Borg
e0405de5bf lib/protocol: More descriptive errors on device ID parse failures 2017-04-27 14:45:35 +09:00
Jakob Borg
d6fbfc3545 lib/fs, lib/model, lib/scanner: Make scans cancellable (fixes #3965)
The folder already knew how to stop properly, but the fs.Walk() didn't
and can potentially take a very long time. This adds context support to
Walk and the underlying scanning stuff, and passes in an appropriate
context from above. The stop channel in model.folder is replaced with a
context for this purpose.

To test I added an infiniteFS that represents a large amount of data
(not actually infinite, but close) and verify that walking it is
properly stopped. For that to be implemented smoothly I moved out the
Walk function to it's own type, as typically the implementer of a new
filesystem type might not need or want to reimplement Walk.

It's somewhat tricky to test that this actually works properly on the
actual sendReceiveFolder and so on, as those are started from inside the
model and the filesystem isn't easily pluggable etc. Instead I've tested
that part manually by adding a huge folder and verifying that pause,
resume and reconfig do the right things by looking at debug output.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4117
2017-04-26 00:15:23 +00:00
Jakob Borg
bdaef44765 gui, man: Update docs & translations 2017-04-26 09:13:12 +09:00
Jakob Borg
488444354b lib/db: Don't panic
So, when first implementing the database layer I added panics on every
unexpected error condition mostly to be sure to flush out bugs and
inconsistencies. Then it became sort of standard, and we don't seem to
have many bugs here any more so the panics are usually caused by things
like checksum errors on read. But it's not an optimal user experience to
crash all the time.

Here I've weeded out most of the panics, while retaining a few "can't
happen" ones like errors on marshalling and write that we really can't
recover from.

For the rest, I'm mostly treating any read error as "entry didn't
exist". This should mean we'll rescan the file and correct the info (if
scanning) or treat it as a new file and do conflict handling (when
pulling). In some cases things like our global stats may be slightly
incorrect until a restart, if a database entry goes suddenly missing
during runtime.

All in all, I think this makes us a bit more robust and friendly without
introducing too many risks for the user. If the database is truly toast,
probably many other things on the system will be toast as well...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4118
2017-04-25 22:52:37 +00:00
Jakob Borg
26654df48c gui: Restrict GUI listening ports to >1024, add help (fixes #4020)
This adds a pattern validator to the GUI listen port field that checks
for port numbers 1024 and above. Also adds a help link pointing to the
(new) page talking about GUI listen port numbers. That page has
information on how to work around the restriction, in general terms.

Also changes the header from "GUI Listen Addresses" to the singular
version, because we only support one listen address today.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4116
2017-04-24 07:19:28 +00:00
Jakob Borg
7f5e236dd7 gui: Wait for config commit before setting ignores (ref #4095)
Given the saveConfig() is async, it might not have happened before we
try to save the ignores and unpause. Likewise must wait for saving
ignores before unpausing or the scan might start before ignores are on
disk.

Javsacript <3
2017-04-24 08:57:07 +09:00
Simon Frei
914b09fd1f lib/model: Create root directory for paused folders (fixes #4094)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4095
2017-04-23 23:50:56 +00:00
Jakob Borg
e9f05d138f lib/db, lib/model: Always use reasonable sized batches (fixes #2250, fixes #4112)
Harmonize how we use batches in the model, using ProtoSize() to judge
the actual weight of the entire batch instead of estimating. Use smaller
batches in the block map - I think we might have though that batch.Len()
in the leveldb was the batch size in bytes, but it's actually number of
operations.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4114
2017-04-22 14:23:33 +00:00
Jakob Borg
10894695c6 cmd/syncthing: Use a 10 second EWMA for CPU usage, refactor a little
We have it lying around so might as well use it instead of inventing our
own thing.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4111
2017-04-20 08:33:09 +00:00
Simon Frei
6b188ebcf3 lib/model: Mark initial scan as finished even if failed and refactor (fixes #4103)
The mechanism to disallow manual scans before the initial scan completed
(#3996) , had the side effect, that if the initial scan failed, no further
scans are allowed. So this marks the initial scan as finished regardless of
whether it succeeded or not.

There was also redundant code in rofolder and a pointless check for folder
health in scanSubsIfHealthy (happens in internalScanFolderSubdirs as well).
This also moves logging from folder.go to ro/rw-folder.go to include the
information about whether it is send-only or send-receive

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4104
2017-04-20 00:20:34 +00:00
Jakob Borg
57e3f9e64b vendor: Update golang.org/x/sys/unix 2017-04-15 10:31:55 +02:00
Jakob Borg
b5694ca788 Merge branch 'release' (omitting the revert)
* release:
  Revert "lib/model, gui: Allow creating and editing ignores of paused folders (fixes #3608)"
2017-04-15 09:49:32 +02:00
Jakob Borg
bcfd18ceb1 Revert "lib/model, gui: Allow creating and editing ignores of paused folders (fixes #3608)"
This reverts commit 25b314f5f1.
2017-04-15 09:38:23 +02:00
Jonathan Cross
f689512a3f gui: Reduce height of textarea in ignores modal
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4102
2017-04-14 20:33:41 +00:00
Jakob Borg
dd1f7a5ab7 lib/connections: Allow negative ACL entries on devices (fixes #4096)
Prefix an entry with "!" to make it a negative entry. First match wins.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4097
2017-04-13 17:43:29 +00:00
Jakob Borg
d48e46a29c cmd/syncthing: Allow custom event subscriptions (fixes #1879)
This adds a parameter "events" to the /rest/events endpoint. It should
be a comma separated list of the events the consumer is interested in.
When not given it defaults to the current set of events, so it's
backwards compatible.

The API service then manages subscriptions, creating them as required
for each requested event mask. Old subscriptions are not "garbage
collected" - it's assumed that in normal usage the set of event
subscriptions will be small enough. Possibly lower than before, as we
will not set up the disk event subscription unless it's actually used.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4092
2017-04-13 17:14:34 +00:00
Jakob Borg
75460be98d Merge branch 'release'
* release:
  lib/protocol: Accept invalid files without blocks (fixes #4093)
2017-04-12 11:49:11 +02:00
Jakob Borg
ce0456b5ac lib/protocol: Accept invalid files without blocks (fixes #4093) 2017-04-12 11:48:27 +02:00
Jakob Borg
e3e028c988 lib/protocol: Accept invalid files without blocks (fixes #4093) 2017-04-12 11:28:12 +02:00
Jakob Borg
da34f27546 gui, lib/config, lib/model: Allow absolute values for minimum disk free space (fixes #3307)
This deprecates the current minDiskFreePct setting and introduces
minDiskFree. The latter is, in it's serialized form, a string with a
unit. We accept percentages ("2.35%") and absolute values ("250 k", "12.5
Gi"). Common suffixes are understood. The config editor lets the user
enter the string, and validates it.

We still default to "1 %", but the user can change that to an absolute
value at will.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4087
LGTM: AudriusButkevicius, imsodin
2017-04-12 09:01:19 +00:00
Jakob Borg
c205fdd77e cmd/syncthing: Handle pre-releases with zero auto upgrade interval (fixes #4090) 2017-04-10 15:56:52 +02:00
Jakob Borg
ae4206f362 gui: Add forgotten translation string 2017-04-07 09:16:55 +02:00
Keith Turner
391665e322 readme: Use logo as header
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4088
2017-04-07 05:21:11 +00:00
Simon Frei
5521759b23 lib/model: TestIgnores: Add ms sleep on all platforms (ref #3986 #3996)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4083
2017-04-06 09:55:54 +00:00
Jakob Borg
f0492c4eb3 gui: Per remote device transfer rates should follow setting (fixes #4082) 2017-04-06 10:42:45 +02:00
Jakob Borg
b1edf12257 gui, man: Update docs & translations 2017-04-05 17:10:31 +02:00
Jakob Borg
2579e8f715 build: Use tildes in Debian package versioning
Makes their version sorting closer to the semver we expect.
2017-04-05 16:59:37 +02:00
Jakob Borg
a1bcc15458 vendor: Mega update all dependencies
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4080
2017-04-05 14:34:41 +00:00
Jakob Borg
49c1527724 vendor: Update github.com/calmh/du (ref #4079) 2017-04-04 14:53:24 +02:00
Jakob Borg
da35820fd5 lib/model: Work around Mac filesystem precision in TestIgnores (ref #3996) 2017-04-03 15:47:23 +02:00
Adam Piggott
79eac61b09 gui: Re-word introducer text (fixes #1819)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3831
2017-04-01 11:06:13 +00:00
Jakob Borg
2ff08e6c84 lib/sync: Make some tests not depend on real clock
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4065
2017-04-01 11:03:11 +00:00
Simon Frei
25b314f5f1 lib/model, gui: Allow creating and editing ignores of paused folders (fixes #3608)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3996
LGTM: calmh, AudriusButkevicius
2017-04-01 09:58:06 +00:00
Jakob Borg
c5e0c47989 lib/connections, lib/model, gui: Specify allowed networks per device (fixes #219)
This adds a new config AllowedNetworks per device, which when set should
contain a list of network prefixes (192.168.0.0/126 etc) that are
allowed for the given device. The connection service will not attempt
connections to addresses outside of the given networks and incoming
connections will be rejected as well.

I've added the config to the normal device editor and shown it (when
set) in the device summary on the main screen.

There's a unit test for the IsAllowedNetwork method, I've done some
manual sanity testing on top of that.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4073
2017-04-01 09:52:31 +00:00
Jakob Borg
4253f22680 lib/scanner: Use fs.Filesystem for all operations
One more step on the path of the great refactoring. Touches rwfolder a
little bit since it uses the Lstat from fs as well, but mostly this is
just on the scanner as rwfolder is scheduled for a later refactor.

There are a couple of usages of fs.DefaultFilesystem that will in the
end become a filesystem injected from the top, but that comes later.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4070
LGTM: AudriusButkevicius, imsodin
2017-04-01 09:04:11 +00:00
Jakob Borg
bdb56d91b9 lib/model: Honor umask when creating folder directories on Unix (fixes #2519)
Doesn't change the behavior on Windows.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4076
2017-03-31 07:51:23 +00:00
Jakob Borg
c3820fbbf2 gui: Allow toggleable units for transfer rate (fixes #234)
Click the transfer rate to toggle between binary-exponent bytes (KiB/s,
MiB/s) and metric based bits (kb/s, Mb/s). The setting is persisted in
browser local storage (best effort).

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4074
2017-03-31 06:32:54 +00:00
Jakob Borg
cbdb036b69 gui: Handle slices in advanced config, show devices in advanced config (fixes #2267)
The ng-list directive makes angular handle lists by doing comma
separation in the roundtrip. We could simplify our normal config dialog
this way as well...

Also adds devices that were inexplicably not available in the advanced
config.

This reveals some other uglyness, as the "devices" config of a folder
now shows as "[object Object], [object Object]" - previously it was
invisible. I think that's fine for now.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4071
2017-03-30 14:32:58 +00:00
Jakob Borg
b75b4190c8 lib/protocol: Add some consistency checks on incoming index updates (fixes #4053)
With this change we will throw a protocol error on some kinds of
malformed index entries.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4064
2017-03-27 07:21:08 +00:00
Simon Frei
1ad547fb65 gui, script: Translate discovery popover and detect it in translate script
Skip-check: pr-solaris

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4061
2017-03-25 11:56:51 +00:00
Jakob Borg
ac46db78d7 gui, man: Update docs & translations 2017-03-23 20:04:35 +09:00
Jaya chithra
53a7c7bd49 gui: Sort languages alphabetically (fixes #3813)
This change sorts the language selection menu in Syncthing's home page so
that the languages are displayed alphabetically. The issue was discussed in
#3813. There were few ways of doing this. Sorting the language names in
transifix file did not work due to access control. So I sorted the languages
directly in languageSelectdirective.js by inverting and sorting the language
info retrieved from localeService.js.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4052
2017-03-22 05:15:32 +00:00
Jakob Borg
e667fcb472 authors: Add jayachithra 2017-03-22 14:02:46 +09:00
Sacheendra Talluri
ee92ee0190 gui: Move discovery failures to modal (fixes #2344)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4048
LGTM: calmh, AudriusButkevicius
2017-03-20 13:55:08 +00:00
Jakob Borg
3e49b73f70 authors: Add sacheendra 2017-03-20 16:03:22 +09:00
Jakob Borg
236e206764 top: Remove inadvertently commit binary 2017-03-20 14:28:13 +09:00
nov1n
ed771f5c64 gui: Show full failed item path
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4023
2017-03-18 11:43:24 +00:00
Jakob Borg
c6dd777fd6 authors: Add nov1n 2017-03-18 19:08:18 +09:00
kwhite17
1caa683ec1 lib/scanner: Stopped outputting rescan debug message if file doesn't exist locally (fixes #1350)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4028
2017-03-18 00:36:54 +00:00
Simon Frei
88dfd634e5 script, gui: Silence useless warnings in translation script
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4034
LGTM: AudriusButkevicius
2017-03-18 00:27:22 +00:00
Jakob Borg
5c27796471 lib/model, lib/scanner: Properly ignore symlinks on Windows (fixes #4035)
Adds a unit test to ensure we don't scan symlinks on Windows. For the
rwfolder, trusts that the logic in the invalid check is correct and that
the check is actually called from the need loop.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4042
2017-03-18 00:25:47 +00:00
Jakob Borg
a54365424e cmd/syncthing: Append build tags used to the long version string
$ go run build.go -race -no-upgrade
    $ syncthing -version
    syncthing v0.14.25-rc.2 "Dysprosium Dragonfly" (go1.8 darwin-amd64)
      jb@unu.kastelo.net 2017-03-16 23:06:21 UTC [noupgrade, race]

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4043
2017-03-16 23:40:27 +00:00
Benedikt Morbach
ef35a7a4cb etc: linux-systemd: Use sleep.target in syncthing-resume.service
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4037
2017-03-12 15:21:12 +00:00
Simon Frei
601a4fac1a cmd/syncthing: Accept absolute -home paths (regression from #3183)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4033
2017-03-09 14:57:12 +00:00
Jakob Borg
f4ccc69422 gui, man: Update docs & translations 2017-03-09 14:13:11 +01:00
Audrius Butkevicius
ceea5ebeb3 lib/connections, vendor: Change KCP mux to SMUX
Closes #4032
2017-03-09 14:03:09 +01:00
Jakob Borg
f35e1ac0c5 lib/config: Oops, update tests 2017-03-08 14:23:48 +01:00
Jakob Borg
0e76f9d93b lib/config: Update default update interval for KCP 2017-03-08 14:18:39 +01:00
Simon Frei
1b08176583 gui: Update translation strings
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4031
2017-03-08 12:38:25 +00:00
Jakob Borg
b3e2665a79 vendor: Update github.com/xtaci/kcp 2017-03-07 14:29:21 +01:00
Jakob Borg
81af29e3e2 lib/config, lib/connections: Configurables for KCP, disable by default
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4030
2017-03-07 12:55:50 +00:00
Audrius Butkevicius
0da0774ce4 lib/connections: Add KCP support (fixes #804)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3489
2017-03-07 12:44:16 +00:00
Jakob Borg
151004d645 vendor: Update github.com/syndtr/goleveldb 2017-03-07 11:57:45 +01:00
Jakob Borg
4e3fdfaeef build: Disable seemingly buggy staticcheck warning 2017-03-07 11:51:58 +01:00
Simon Frei
1c29a93013 lib/events, cmd/syncthing: Correct GlobalID in debug and mark "since" parameter optional
Skip-check: metalint

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4025
2017-03-07 05:44:47 +00:00
Jakob Borg
689f13f4f1 build: Don't check for supported Go version
One thing less to maintain, and you can discover for yourself if it
builds or not.
2017-03-05 10:31:56 +01:00
Jakob Borg
0558565a95 build, jenkins: Build for linux-mipsle 2017-03-05 10:13:07 +01:00
Jakob Borg
b84c4e1417 vendor: Update github.com/minio/sha256-simd 2017-03-05 10:12:54 +01:00
Simon Frei
416811a2a9 cmd/syncthing, lib/config: Pause/resume all devices whithout argument
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3989
LGTM: AudriusButkevicius, calmh
2017-03-04 07:54:13 +00:00
Simon Frei
c20d612736 cmd/syncthing, lib/model: Handle rel/abs paths for config/protected paths (fixes #3183)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3995
2017-03-04 07:49:48 +00:00
Jakob Borg
3475a9ab0a jenkins: Remove spurious 'parts' directory when cleaning 2017-03-04 08:22:35 +01:00
Jakob Borg
d7adee05a8 build, jenkins: Build for mips (fixes #3959) 2017-03-04 07:48:42 +01:00
Jakob Borg
73cbd17e17 vendor: Update github.com/minio/sha256-simd with patch for mips 2017-03-04 07:48:16 +01:00
Jakob Borg
7260629bc0 vendor: Update golang.org/x/net/ipv6 2017-03-04 07:28:11 +01:00
andresvia
566c348b00 build: Remove obsolete check for vendoring support
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4008
2017-02-26 15:12:20 +00:00
chucic
190f153b92 gui: Correct usage of folder vs directory
skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4007
2017-02-25 17:43:07 +00:00
HairyFotr
c56c48a777 all: Correct various typos
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4005
2017-02-25 08:12:13 +00:00
Wulf Weich
98d22b88a0 gui: Fix icons in pause / resume all buttons (fixes #4003)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4004
2017-02-24 08:42:32 +00:00
Jakob Borg
28449f9f5b gui, man: Update docs and translations 2017-02-23 07:47:07 +01:00
Jakob Borg
a0e2e7a962 gui: Translation strinsg for pause/resume all 2017-02-22 04:03:41 +01:00
Ben S
fb6d453c74 gui: Update button wording
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3982
2017-02-10 17:21:16 +00:00
Jakob Borg
89f8be40c6 lib/model: Use ignore patterns in remote completion calculation
Basically, if we don't care about the sync status of the file we should
not tag someone else out of sync because they don't have the latest
version. This solves *my* "Syncing - 100%" scenario at least.

The reason this happens seems to be like this, in my situation. I have
three devices, connected in a "line": A-B-C. A is a Mac and litters
.DS_Store files everywhere. I've ignored these, but some escaped into
the folders before I did so. I've also ignored them on B and C but at
different stages. B was flagging C as out of sync, because at the point
the ignores were introduced C had a lower version of .DS_Store than A.
Now none of them are sending updates about it any more since it's
ignored...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3981
2017-02-10 11:22:09 +00:00
Jakob Borg
96fba1b322 Merge branch 'release'
* release:
  cmd/syncthing: Environment handling for upgrade restarts (fixes #3970)
2017-02-10 11:58:26 +01:00
Niller303
0c68e1e510 GUI: Added "Pause All Folders" and "Resume All Folders buttons
As per scienmind's post (#3964) i though it was a good idea and wanted
such a button.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3973
LGTM: AudriusButkevicius, calmh
2017-02-09 20:31:23 +00:00
Simon Frei
22903df2c1 lib/model: Meaningful error messages for paused folders
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3977
LGTM: AudriusButkevicius, calmh
2017-02-09 20:29:56 +00:00
Jakob Borg
10618e80a3 authors: Add Niller303 2017-02-09 13:55:51 +01:00
Jakob Borg
161326c548 all: Weed out a few other http urls (ref #3976) 2017-02-09 08:04:16 +01:00
Jakob Borg
f7fc0c1d3e all: Update license url to https (ref #3976) 2017-02-09 08:04:16 +01:00
Audrius Butkevicius
120e6eab2c lib/sync: Fix a race in unlocker logging (fixes #3884)
Other routines use atomics, hence even if we are under a lock, we should
too.

We might atomically store with
Not sure how it happens, but it's between lines

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3974
2017-02-08 22:31:19 +00:00
Audrius Butkevicius
72de47df00 lib/model: Increase in-flight buffer from 2mb to 8mb, download 2 files at a time
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3972
2017-02-08 20:00:55 +00:00
Jakob Borg
dfe23e8d53 jenkins: Set BUILD_USER for Debian and Snap builds
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3971
2017-02-08 13:02:09 +00:00
Jakob Borg
2b1c942d56 cmd/syncthing: Environment handling for upgrade restarts (fixes #3970) 2017-02-07 21:36:58 +01:00
Jakob Borg
204f125ab3 cmd/syncthing: Environment handling for upgrade restarts (fixes #3970) 2017-02-07 21:25:33 +01:00
Jakob Borg
73f9c7d174 lib/versioner: Convert Staggered to a suture.Service (fixes #3820) 2017-02-07 14:33:33 +01:00
Jakob Borg
c4ba580cbb all: Remove symlink support on Windows, SymlinksEnabled config
After this change,

- Symlinks on Windows are always unsupported. Sorry.

- Symlinks are always enabled on other platforms. They are just a small
  file like anything else. There is no need to special case them. If you
  don't want to sync some symlinks, ignore them.

- The protocol doesn't differentiate between different "types" of
  symlinks. If that distinction ever does become relevant the individual
  devices can figure it out by looking at the destination when they
  create the link.

It's backwards compatible in that all the old symlink types are still
understood to be symlinks, and the new SYMLINK type is equivalent to the
old SYMLINK_UNKNOWN which was always a valid way to do it.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3962
LGTM: AudriusButkevicius
2017-02-07 08:34:24 +00:00
Jakob Borg
9fda9642d3 lib/events: Make test even less timing dependent 2017-02-07 08:57:39 +01:00
Jakob Borg
dfd2c464b6 lib/events: Overflow test should calculate average log time 2017-02-07 08:47:53 +01:00
Jakob Borg
5a1ee7f0b0 gui, man: Update docs & translations 2017-02-07 08:28:02 +01:00
Jakob Borg
fdcbd54cd7 lib/events: Be more resilient against dropping events (fixes #3952)
Instead of just immediately dropping the event if the subscription isn't
ready to receive it, give it 15 ms to catch up. The value 15 ms is
grabbed out of thin air - it just seems reasonable to me.

The timer juggling makes the event send pretty much exactly twice as
slow as it was before, but we're still under a microsecond. I think it's
negligible compared to whatever event that just happened that we're
interested in logging (usually a file operation of some kind).

	benchmark                  old ns/op     new ns/op     delta
	BenchmarkBufferedSub-8     475           950           +100.00%

	benchmark                  old allocs     new allocs     delta
	BenchmarkBufferedSub-8     4              4              +0.00%

	benchmark                  old bytes     new bytes     delta
	BenchmarkBufferedSub-8     104           117           +12.50%

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3960
2017-02-07 07:25:09 +00:00
Jakob Borg
e14741a58c cmd/syncthing: Speed up the CPU benchmark
Not the measured performance, just the wall clock time it takes to
complete. The random generation was dominating.
2017-02-06 13:42:39 +01:00
Audrius Butkevicius
67acef1794 lib/weakhash, lib/model, cmd/syncthing: Decide if to use weakhash on startup (fixes #3938)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3949
2017-02-06 10:27:11 +00:00
Simon Frei
237893ead3 cmd/syncthing: Only delay next scan (via REST) if the scan succeeds
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3927
2017-02-05 18:17:44 +00:00
Jakob Borg
2590536ef3 readme: Spelling error in bold in the first sentence
Nobody reads this, right?
2017-02-05 18:58:12 +01:00
Jakob Borg
dc91995475 readme: Go report card badge 2017-02-05 18:54:25 +01:00
Jakob Borg
3655c97850 cmd/syncthing, lib/fs, lib/sync: Spelling in comments 2017-02-05 18:51:52 +01:00
Jakob Borg
c0f3f06cfb lib/weakhash, script: gofmt -s 2017-02-05 18:49:57 +01:00
Jakob Borg
05450ca034 readme: More build badges 2017-02-05 18:46:17 +01:00
Jakob Borg
c005e61151 snapcraft: s/snap/prime (fixes #3955) 2017-02-05 15:03:36 +01:00
Jakob Borg
63e0b53e8b build: Set snap grade "stable" for release candidates 2017-02-05 14:42:29 +01:00
Jakob Borg
1f586c0fdd gui, man: Update docs & translations 2017-02-05 13:54:49 +01:00
Jakob Borg
f1a073501f lib/fs: A nil MtimeFS is valid (fixes #3958)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3961
2017-02-05 12:54:08 +00:00
Jakob Borg
a72f5379fb jenkins: Build natively on Solaris
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3957
2017-02-05 11:20:15 +00:00
Jakob Borg
8cccecceba lib/events: Speed up event polling loop slightly (ref #3952)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3954
2017-02-04 15:53:39 +00:00
janost
81418d724a jenkins: Add arm64 deb build
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3953
2017-02-04 14:18:54 +00:00
Jakob Borg
3eb7a9373a gui, lib/config: Add notification about new release channels
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3945
2017-02-04 09:45:17 +00:00
Antony Male
ac510b26e2 cmd/syncthing, lib/events, lib/sync: Add timeout to REST event API, remove Ping (fixes #3933)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3941
2017-01-31 12:04:29 +00:00
Jakob Borg
20f8b4fd57 gui: Wording of auto upgrade selections 2017-01-30 23:14:45 +01:00
Jakob Borg
1c9361a818 cmd/syncthing: Implement "release candidate" logic
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3943
LGTM: AudriusButkevicius
2017-01-30 21:33:07 +00:00
Jakob Borg
35e87e23fd cmd/syncthing, gui, lib/config, lib/upgrade: Add option to upgrade to pre-releases
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3939
2017-01-27 12:17:06 +00:00
benshep
e03be9158b gui: Remaining sync bytes in folder header (fixes #3908)
The progress indicator in the folder header in the GUI now shows the
remaining bytes to sync as well as the percentage.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3928
2017-01-26 09:39:48 +00:00
Jakob Borg
9833d13762 gui: Don't trim space on the password field (fixes #3935) 2017-01-26 08:13:41 +01:00
Jakob Borg
6ec7d711d8 authors: Add benshep 2017-01-25 19:20:52 +01:00
Jakob Borg
22a4d49ed0 cmd/syncthing: Handle -logfile again (fixes #3931)
The monitor process should not set STNORESTART as this indicates the
intention from the user. Setting STMONITORED is enough, as this tells
the next Syncthing instance that it is running under the monitor
process.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3932
2017-01-25 07:33:35 +00:00
Antony Male
ddca8d91fa cmd/syncthing: Send Ping events to the disk events API
The Ping event is important, as it means that requests complete within
a sensible time. The disk events API didn't have the Ping event, so
if there were no disk events, the request would keep taking forever.
Unless, of course, there's a reverse proxy which times the request out
after a suitably large interval (or something else aborts it), in which
case Syncthing isn't very happy.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3929
2017-01-24 17:32:34 +00:00
Jakob Borg
ee36e2d46d lib/weakhash: Limit number of hits for any given weakhash
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3925
2017-01-24 08:26:45 +00:00
Jakob Borg
de49ea594a lib/upnp: Wrong order of internal/external port after OnlyPermanentLeasesSupported (fixes #3924) 2017-01-23 23:17:02 +01:00
Jakob Borg
6d4fa27ea7 cmd/syncthing: Report real hashing performance, including weakhash
Instead of

    [I6KAH] 19:05:56 INFO: Single thread hash performance is 359 MB/s using minio/sha256-simd (354 MB/s using crypto/sha256).

it now says

    [I6KAH] 19:06:16 INFO: Single thread SHA256 performance is 359 MB/s using minio/sha256-simd (354 MB/s using crypto/sha256).
    [I6KAH] 19:06:17 INFO: Actual hashing performance is 299.01 MB/s

which is more informative. This is also the number it reports in usage
reporting.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3918
2017-01-23 21:56:43 +00:00
Jakob Borg
9587b89d9d gui, man: Update docs & translations 2017-01-23 22:03:08 +01:00
Jakob Borg
79c7f7193b lib/upnp: Remove unnecessary error allocation
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3923
2017-01-23 21:02:55 +00:00
Audrius Butkevicius
2c4b92d410 lib/connections: Fix rate limiting on arm64 (fixes #3921)
Skip-check: metalint

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3922
2017-01-23 20:55:00 +00:00
Audrius Butkevicius
dd78177ae0 scanner: Allow disabling weak hash in scanning (fixes #3891)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3905
2017-01-23 13:50:32 +00:00
Jakob Borg
bd55ec79d2 goals, readme: Add updated project goals 2017-01-19 18:12:40 +01:00
Jakob Borg
1313ba8c0a gui, vendor: Update license, copyright for github.com/chmduquesne/rollinghash 2017-01-19 15:59:39 +01:00
Jakob Borg
842e873a94 cmd/syncthing: Fix -logfile/-no-restart test on non-Windows. 2017-01-18 18:59:48 +01:00
Jakob Borg
d4c4b1fb4c vendor: Temporarily patch github.com/chmduquesne/rollinghash
To avoid allocations in the hasher. PR files, should be available for
update soon.
2017-01-18 18:45:29 +01:00
Jakob Borg
68f1c6ccab lib/scanner: Avoid per iteration allocations in Blocks()
Resetting the io.LimitReader is better than creating a new one on every
iteration.
2017-01-18 18:43:00 +01:00
Antony Male
4c8aa14e07 cmd/syncthing: Fail if -logfile and -no-restart passed together (fixes #3912)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3915
2017-01-18 12:19:22 +00:00
Jakob Borg
bd1c29ee32 lib/scanner, vendor: Fix previous commit
Can't do what I did, as the rolling function is not the same as the
non-rolling one. Instead this uses an improved version of the rolling
adler32 to accomplish the same thing. (PR filed on upstream, so should
be able to use that directly in the future.)
2017-01-18 11:57:01 +01:00
Jakob Borg
9b1c592fb7 lib/scanner: Speed up weak hash
The rolling version of adler32 is just a wrapper around the standard
hash/adler32 when used in a non-rolling fashion, but it's inefficient as
it allocates a new hash instance for every Write(). This uses the
default version instead in the block hasher, and adds a test to verify
the result is the same as they were before. It reduces allocations by
88% and increases speed about 5%.

	benchmark               old ns/op     new ns/op     delta
	BenchmarkHashFile-8     64434698      61303647      -4.86%

	benchmark               old MB/s     new MB/s     speedup
	BenchmarkHashFile-8     276.65       290.78       1.05x

	benchmark               old allocs     new allocs     delta
	BenchmarkHashFile-8     1238           150            -87.88%

	benchmark               old bytes     new bytes     delta
	BenchmarkHashFile-8     17877363      49292         -99.72%
2017-01-18 10:33:17 +01:00
Jakob Borg
f36f00e87b gui: Update translation base (lang-en) (fixes #3909) 2017-01-17 15:58:35 +01:00
Simon Frei
dbb3a34887 lib/ignore: Centralize handling of temporary filenames (fixes #3899)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3901
LGTM: calmh, AudriusButkevicius
2017-01-17 07:33:48 +00:00
KAMADA Ken'ichi
929a4d0c0c gui: Improve warnings when creating folder in a subdirectory (fixes #3197, fixes #3902)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3904
2017-01-14 12:18:48 +00:00
ProactiveServices
c953cdc375 gui: Package attribution and copyright bumps (fixes #3861)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3863
2017-01-10 07:50:11 +00:00
Jakob Borg
c20c17e3c6 gui, man: Update docs & translations 2017-01-10 08:41:14 +01:00
Audrius Butkevicius
1a1e35d998 lib/osutil: Replace IsDir with TraversesSymlink (fixes #3839)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3883
LGTM: calmh
2017-01-10 07:09:31 +00:00
Adel Qalieh
8d2a31e38e lib/model: Remove syncthing-specific files (fixes #3819)
Syncthing adds some hidden files when a folder is added, but there is currently
no equivalent cleanup procedure. This change is conservative as not to
accidentally cause data loss.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3874
2017-01-07 17:05:30 +00:00
Jakob Borg
fe377a166a authors: Update adelq 2017-01-07 17:56:46 +01:00
Jakob Borg
5770d80bf9 authors: Add adelq 2017-01-07 17:54:41 +01:00
Jakob Borg
7a16dbd31d lib/scanner: Fix previous commit: don't stop scan completely 2017-01-05 15:05:49 +01:00
Jakob Borg
926c88cfc4 lib/scanner: Never, ever descend into symlinks (ref #3857)
On Windows we would descend into SYMLINKD type links when we scanned
them successfully, as we would return nil from the walk function and the
filepath.Walk iterator apparently thought it OK to descend into the
symlinked directory.

With this change we always return filepath.SkipDir no matter what.

Tested on Windows 10 as admin, does what it should.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3875
2017-01-05 13:26:29 +00:00
Audrius Butkevicius
29d010ec0e lib/model, lib/weakhash: Hash using adler32, add heuristic in puller
Adler32 is much faster, and the heuristic avoid the obvious cases where it
will not help.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3872
2017-01-04 21:04:13 +00:00
Jakob Borg
920274bce4 lib/db: Don't panic on unknown folder in ListFolders (fixes #3584)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3869
2017-01-04 10:34:52 +00:00
Jakob Borg
2ebd6ad77f lib/scanner: Don't stop byte counter ticks before scan is done 2017-01-03 15:03:32 +01:00
Kudalufi
79dd6918f2 cmd/syncthing: Add support for -auditfile= (fixes #3859)
Adds support for -auditfile= where is "-" for stdout, "--" for stderr, or a
filename. It can be left blank (or left out entirely) for the original
behaviour of creating a timestamped filename.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3860
2017-01-03 08:54:28 +00:00
Jakob Borg
79f7f50c4d authors: Add Kudalufi 2017-01-03 09:24:29 +01:00
Jakob Borg
987718baf8 vendor: Update github.com/gogo/protobuf
Also tweaks the proto definitions:

 - [packed=false] on the block_indexes field to retain compat with
   v0.14.16 and earlier.

 - Uses the vendored protobuf package in include paths.

And, "build.go setup" will install the vendored protoc-gen-gogofast.
This should ensure that a proto rebuild isn't so dependent on whatever
version of the compiler and package the developer has installed...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3864
2017-01-03 00:16:21 +00:00
Jakob Borg
4fb9c143ac authors: Update ProactiveServices 2017-01-02 15:12:57 +01:00
Jakob Borg
ec62888539 lib/connections: Allow on the fly changes to rate limits (fixes #3846)
Also replaces github.com/juju/ratelimit with golang.org/x/time/rate as
the latter supports changing the rate on the fly.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3862
2017-01-02 11:29:20 +00:00
Jakob Borg
8c34a76f7a man: refresh.sh requires bash 2017-01-01 20:45:52 +01:00
Jakob Borg
6809d38cde lib/protocol: Revert protobuf encoder changes in v0.14.17 (fixes #3855)
The protobuf encoder now produces packed arrays for things like []int32,
which is actually correct according to the proto3 spec. However
Syncthing v0.14.16 and earlier doesn't support this. This reverts the
encoding change, but keeps the updated decoder so that we are both more
compatible with other proto3 implementations and can move to the updated
encoder in the future.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3856
2017-01-01 17:19:00 +00:00
Mark Pulford
69ae4aa024 cmd/syncthing: Avoid Keepalive/GUI refresh race
This avoids unnecessary browser request failures and retries. Eg:
- Browser reuses existing HTTP connection for GUI refresh request
- Server closes connection with request in flight
- Browser retries GET request.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3854
2017-01-01 12:38:31 +00:00
Jakob Borg
8e8b867fba authors: Add mpx 2017-01-01 13:28:33 +01:00
Jakob Borg
0a118d2979 lib/config, lib/model: Temporarily disable bad tests (ref #3834, #3843) 2017-01-01 13:27:18 +01:00
Nathan Morrison
8daaa5d0d2 gui: Populate global changes on load
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3848
2016-12-30 01:33:27 +00:00
Jakob Borg
eb14f85a57 vendor: Update github.com/syndtr/goleveldb 2016-12-28 12:19:14 +01:00
Jakob Borg
c69c3c7c36 lib/sha256: Smoke test the implementation on startup (hello OpenSUSE!) 2016-12-28 12:15:51 +01:00
Jakob Borg
54911d44c5 gui: s/foldersendonly.html/foldertypes.html 2016-12-27 11:29:12 +01:00
Jakob Borg
c765f7be8d gui, man: Update docs and translations 2016-12-26 14:23:55 +01:00
Jakob Borg
44bdaf3ac2 cmd/syncthing: Add -reset-deltas option to reset delta index IDs
Also rename and clarify the description of -reset-database (formerly
-reset).
2016-12-26 13:49:51 +01:00
Jakob Borg
fc16e49cb0 Merge branch 'v0.14.16-hotfix'
* v0.14.16-hotfix:
  gui, man: Update docs & translations
  lib/model: Allow empty subdirs in scan request (fixes #3829)
  lib/model: Don't send symlinks to old devices that can't handle them (fixes #3802)
  lib/model: Accept scan requests of paths ending in slash (fixes #3804)
  gui: Avoid pause between event polls (ref #3527)
2016-12-24 20:12:53 +01:00
Jakob Borg
f5a310ad64 Revert "lib/model: Handle filename conflicts on Windows."
This reverts commit 01e50eb3fa.
2016-12-23 11:10:58 +01:00
Unrud
01e50eb3fa lib/model: Handle filename conflicts on Windows.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3810
LGTM: calmh
2016-12-22 23:04:53 +00:00
Jakob Borg
722b81c6f0 gui, man: Update docs & translations 2016-12-21 19:46:28 +01:00
Jakob Borg
f0efa2b974 lib/model: Allow empty subdirs in scan request (fixes #3829)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3833
2016-12-21 19:45:38 +01:00
Audrius Butkevicius
bab7c8ebbf all: Add folder pause, make pauses permanent (fixes #3407, fixes #215, fixes #3001)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3520
2016-12-21 18:41:25 +00:00
Nathan Morrison
0725e3af38 all: Add a global change list
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3694
2016-12-21 16:35:20 +00:00
Jakob Borg
dd7bb6c4b8 lib/model: Fix tests, clean up pool usage in protocol 2016-12-21 14:53:45 +01:00
Jakob Borg
d41c131364 build: Enable gometalinter "gosimple" check, improve build.go 2016-12-21 14:53:45 +01:00
Jakob Borg
47f22ff3e5 build: Enable gometalinter "unconvert" check 2016-12-21 14:53:45 +01:00
Jakob Borg
744c2e82b5 build: Enable gometalinter "staticcheck" check 2016-12-21 14:53:45 +01:00
Jakob Borg
ead7281c20 build: Enable gometalinter "unused" check 2016-12-21 14:53:45 +01:00
AudriusButkevicius
aa3ef49dd7 lib/model: Fix lock order
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3823
2016-12-21 12:22:18 +00:00
Jakob Borg
5c067661f4 lib/model: Consistently show folder description in startup messages
Since we anyway need the folderConfig for this I'm skipping the copying
of all it's attributes that rwfolder did and just keeping the original
around instead.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3825
2016-12-21 11:23:20 +00:00
Jakob Borg
226da976dc lib/model: Allow empty subdirs in scan request (fixes #3829)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3833
2016-12-21 10:33:07 +00:00
AudriusButkevicius
ba17cc0a11 gui: Show introducedBy (fixes #3809) 2016-12-21 11:01:15 +01:00
AudriusButkevicius
9e0afb7d8a lib/connections: Support setting traffic class (fixes #3790) 2016-12-21 11:01:15 +01:00
AudriusButkevicius
9e7d50bc76 cmd/syncthing: Explain corruption panics (fixes #3689) 2016-12-21 11:01:15 +01:00
Jakob Borg
d7d5687faa lib/protocol: Warnln should have been Warnf 2016-12-21 10:43:12 +01:00
Jakob Borg
21eb098dd2 vendor: Update github.com/minio/sha256-simd, CPU detection (Linux) 2016-12-20 09:20:28 +01:00
Jakob Borg
2f770f8bfb lib/config, lib/protocol: Improve folder description with empty label 2016-12-19 10:12:06 +01:00
Jakob Borg
f09698d845 gui: Add missing strings to lang-en.json 2016-12-18 22:29:43 +01:00
Jakob Borg
3fbcb024a7 gui: Update link to send only documentation page 2016-12-18 22:29:43 +01:00
Jakob Borg
b8c1c0e048 cmd/syncthing: Enable better crypto, print negotiated cipher suite
This adds support for AES_256_GCM_SHA384 (in there since Go 1.5, a bit
of a shame we missed it) and ChaCha20-Poly1305 (if built with Go 1.8;
ignored on older Gos).

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3822
2016-12-18 21:07:44 +00:00
Jakob Borg
2d47242d54 jenkins: Don't clean out when building with old Go (nukes coverage data) 2016-12-18 19:58:46 +01:00
Jakob Borg
66a7829eee jenkins: Also try build with old Go version, if available 2016-12-18 19:25:27 +01:00
Jakob Borg
9c67bd2550 lib/connections: Fix port fixup in Go 1.8 (fixes #3817)
The test for the error string is fragile, and the error string changed
in Go 1.8 so the relevant part is no longer a prefix. This covers it
with a test though, so it should be fine in the future as well.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3818
2016-12-18 11:28:18 +00:00
Jakob Borg
f67c5a2fd6 lib/model: Don't send symlinks to old devices that can't handle them (fixes #3802)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3811
2016-12-17 19:48:33 +00:00
Jakob Borg
0437f6dd66 lib/model: Don't send symlinks to old devices that can't handle them (fixes #3802) 2016-12-17 12:39:22 +01:00
Jakob Borg
11b35d650d lib/model: Accept scan requests of paths ending in slash (fixes #3804)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3805
2016-12-17 12:39:22 +01:00
Jakob Borg
b279e261a1 gui: Avoid pause between event polls (ref #3527)
We lose important events due to the frequency of ItemStarted /
ItemFinished events.
2016-12-17 12:39:22 +01:00
Jakob Borg
263402f80a cmd/stcli: Add copyright headers to satisfy check 2016-12-17 12:28:59 +01:00
Jakob Borg
920a83ec7a cmd/stcli: Fix metalint ineffasign complaint 2016-12-17 10:51:48 +01:00
Jakob Borg
3c2ac3522c cmd/stcli: Update for new folder type 2016-12-17 01:44:18 +01:00
Jakob Borg
9fdaa637a8 vendor: Add github.com/AudriusButkevicius/cli 2016-12-17 01:33:17 +01:00
Jakob Borg
81a9d7f2b9 cmd/stcli: Make it build again 2016-12-17 01:33:17 +01:00
Audrius Butkevičius
d8d3f05164 cmd/stcli: Import from syncthing-cli repository 2016-12-17 01:33:17 +01:00
Jakob Borg
653be136ee gui: Update lang-en.json 2016-12-17 00:59:25 +01:00
Heiko Zuerker
398c356f22 lib/model: Clarify master terminology (fixes #2679)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3793
2016-12-16 22:23:35 +00:00
Audrius Butkevicius
542b76f687 lib/model: Moar sleep
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3807
2016-12-16 12:05:27 +00:00
Jakob Borg
abb8a1914a lib/model: Accept scan requests of paths ending in slash (fixes #3804)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3805
2016-12-16 11:21:22 +00:00
Jakob Borg
163d335078 gui: Update lang-en.json 2016-12-16 09:42:13 +01:00
Audrius Butkevicius
0582836820 lib/model, lib/scanner: Efficient inserts/deletes in the middle of the file
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3527
2016-12-14 23:30:29 +00:00
Jakob Borg
bb15776ae6 gui: Avoid pause between event polls (ref #3527)
We lose important events due to the frequency of ItemStarted /
ItemFinished events.
2016-12-14 10:31:16 +01:00
Jakob Borg
dde9d4c9eb lib/rc: Remove pause to aggregate events (ref #3527) 2016-12-14 10:24:36 +01:00
Jakob Borg
1ef75be1c6 gui, man: Update docs & translations 2016-12-13 11:29:40 +01:00
Jakob Borg
3582783972 lib/model, lib/osutil: Verify target directory before pulling / requesting
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3798
2016-12-13 10:24:10 +00:00
Jakob Borg
5070d52f2f lib/model: Add benchmark for model.Request() 2016-12-09 23:14:56 +01:00
Jakob Borg
7b07ed6580 lib/model, lib/protocol, lib/scanner: Include symlink target in index, pull symlinks synchronously
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3792
2016-12-09 18:02:18 +00:00
Han Boetes
f6a2b6252a gui: Tweak wording (fixes #3769)
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3796
2016-12-09 17:16:29 +00:00
Jakob Borg
a9b03de99a gui, lib/db: Correct space accounting of symlinks, for "out of sync" status 2016-12-09 10:38:36 +01:00
Jakob Borg
a7f7058636 vendor: Update github.com/gobwas/glob (bugfix) 2016-12-07 09:25:58 +01:00
Lars K.W. Gohlke
8ce9b026e9 lib/model: Minor cleanups
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3765
2016-12-06 08:54:04 +00:00
Audrius Butkevicius
0dcf2f1bc8 cmd/strelaysrv: Use legacy dial (fixes #3753)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3784
2016-12-02 22:45:08 +00:00
Audrius Butkevicius
99922feb3b gui: Disable device removal when we know it will be reintroduced
Skip-check: pr-build-windows

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3762
2016-12-02 21:07:02 +00:00
Jakob Borg
2fd1dca905 lib/connections: Fix odd logging, forgot to call function 2016-12-02 12:56:14 +01:00
Jakob Borg
e3cf718998 lib/ignore: Add central check for internal files, used in scanning, pulling and requests
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3779
2016-12-01 14:00:11 +00:00
Stefan Kuntz
7157917a16 etc: Updated ufw firewall application preset with default GUI port
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3774
2016-12-01 12:36:15 +00:00
Jakob Borg
3266aae1c3 lib/protocol: Apply input filtering on file names
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3775
2016-12-01 12:35:32 +00:00
Jakob Borg
63194a37f6 lib/model: Double check results in filepath.Join where needed
Wherever we have untrusted relative paths, make sure they are not
escaping their folder root.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3776
2016-12-01 12:35:11 +00:00
Unrud
cabe94552a lib/model: Prevent collisions in the progressemitter registry
Using filepath.Join can cause collisions. The folder ID could be something
like ".." or "../..".

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3778
2016-12-01 12:34:20 +00:00
Jakob Borg
48a229a0cd lib/model: Temp names from all platforms should be recognized as such
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3777
LGTM: AudriusButkevicius
2016-11-30 21:23:24 +00:00
Jakob Borg
e4db86836b lib/model: Locking in the request test 2016-11-30 13:11:06 +01:00
Jakob Borg
913a85c571 lib/model: Add simple file syncing test
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3772
2016-11-30 09:32:28 +00:00
Jakob Borg
ed4f6fc4b3 lib/connections, lib/model: Connection service should expose a single interface
Makes testing easier, which we'll need

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3771
2016-11-30 07:54:20 +00:00
Jakob Borg
9da422f1c5 gui, man: Update docs & translations 2016-11-29 11:56:02 +01:00
Stefan Tatschner
ab1739ba34 cmd/syncthing: Trigger usage message on extra CLI parameters
fixes: #3690

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3763
2016-11-27 11:21:05 +00:00
Jakob Borg
fc1430aa92 lib/fs: The interface and basicfs
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3748
2016-11-24 12:07:14 +00:00
Jakob Borg
3cde608eda lib/db: Fix ineffassign lint issue 2016-11-24 12:08:44 +01:00
Jakob Borg
2898552f4b build: Setup should insteall deadcode metalinter 2016-11-24 12:05:04 +01:00
Jakob Borg
911c148c71 build: Improve setup, add metalint ineffasign 2016-11-24 11:33:43 +01:00
Jakob Borg
91568a173a lib/model: Remove ineffectual assignment in test 2016-11-24 11:33:27 +01:00
Jakob Borg
e57f5499a1 lib/sync: Remove unused struct field 2016-11-24 11:30:55 +01:00
Jakob Borg
9abb7b71a9 lib/osutil: Fix lint warning on error formatting (fixes #3760) 2016-11-24 11:20:51 +01:00
Jakob Borg
724c354d62 cmd/stdiscosrv: Fix lint warning on Context keys (fixes #3760) 2016-11-24 11:20:51 +01:00
Jakob Borg
c44779094d build: Setup should download latest version of linters etc 2016-11-24 11:20:51 +01:00
Wulf Weich
eeedab4091 gui: bottom nav always behind dropdown (fixes #3758)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3759
2016-11-23 17:03:43 +00:00
Jakob Borg
8559e20237 lib/osutil: Don't chmod in atomic file creation (fixes #2472)
Instead, trust (and test) that the temp file has appropriate permissions
from the start. The only place where this changes our behavior is for
ignores which go from 0644 to 0600. I'm OK with that.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3756
2016-11-23 14:06:08 +00:00
Jakob Borg
26730eb083 lib/model: Fix test that relies on ignore reloading 2016-11-23 14:42:29 +01:00
Simon Frei
4160ce674d model: consistently use cfg when referring to config instance and not package
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3755
2016-11-22 23:14:20 +00:00
Jakob Borg
2dbeea21c4 lib/ignore: Don't slow down tests by sleeping 2016-11-22 22:44:04 +01:00
Jakob Borg
a2b8485a89 lib/ignore: Fast reload of unchanged ignores (fixes #3394)
This changes the "seen" map that we're anyway keeping around to track
the modtimes of loaded files instead. When doing a Load() we check that
1) the file we are loading is in the modtime set, and 2) that none of
the files in the modtime set have changed modtimes. If that's the case
we do a quick return without parsing anything or clearing the cache.

This required adding two one seconds sleeps in the tests to make sure
the modtimes were updated when we expect cache reloads, because I'm on a
crappy filesystem with one second timestamp granularity. That also
proves it works...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3754
2016-11-22 21:30:45 +00:00
Jakob Borg
5bb74ee61c gui, man: Update docs & translations 2016-11-22 09:32:57 +01:00
Jakob Borg
462fde5e7d cmd/syncthing: Make the default folder default again
The current way is quite confusing for new users - we create a default
folder, but it's not usable with the default folder created somewhere
else. Instead, when setting up for the first time with two devices, the
default folder must be removed and recreated on one of them. This comes
up on IRC and the forum now and then.

I think this matches expectactions better.

Another alternative would be to remove it entirely (not create a default
folder), but then we should also add some guidance in the UI on how to
proceed.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3751
2016-11-22 08:18:43 +00:00
Unrud
f1e83a57cd lib/osutil: Remove unnecessary fsync in Copy()
Fsyncing the file has a small performance penalty and seems unnecessary. The
file will be fsynced anyway, when the changes are commited to the database.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3749
2016-11-22 07:59:54 +00:00
Jakob Borg
cc9a9fb390 lib/model, lib/protocol: Add Folder.Description() for logging (ref #3741) 2016-11-22 08:36:14 +01:00
Jakob Borg
8fbcceb742 authors: Add further Unrud address 2016-11-22 08:14:22 +01:00
Roman Zaynetdinov
d3a251e6d9 lib/model: Log folder IDs and labels (fixes #3724)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3741
2016-11-21 20:09:18 +01:00
Jakob Borg
be80b26c18 authors: Add zaynetro 2016-11-21 20:08:31 +01:00
Unrud
1574b7d834 lib/model: Add fsync of files and directories, option to disable (fixes #3711) 2016-11-21 18:09:51 +01:00
Jakob Borg
51e10e344d Add Unrud 2016-11-21 17:59:44 +01:00
kwhite17
0d55d8c5b0 gui: Convert URLs in warning messages to HTML links (fixes #3241)
Skip-check: metalint (annoying timeout)

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3747
2016-11-21 08:27:44 +00:00
Jakob Borg
1392589d36 authors: Add kwhite17 2016-11-21 09:12:03 +01:00
Jakob Borg
548a324256 lib/connections: Slow down failing listeners
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3745
2016-11-19 12:37:14 +00:00
Jakob Borg
a8a0bc356a lib/model: Minor cleanup to not fondle cfg.Raw things in handleDeintroductions
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3739
2016-11-17 08:56:55 +00:00
Jakob Borg
faee1d5a8d lib/model: Fix locking around introduction handling (fixes #3737)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3738
2016-11-17 08:50:24 +00:00
Jakob Borg
3088dac33b lib/model: Clean up generateClusterConfig, fix spurious test failure by sorting 2016-11-17 07:45:45 +01:00
Jakob Borg
2641062c17 gui, man: Update docs & translations 2016-11-15 07:23:48 +01:00
Jakob Borg
95c738ea28 lib/protocol: Serialize the all zeroes device ID to the empty string
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3734
2016-11-15 06:22:36 +00:00
Jakob Borg
562d2f67a6 snapcraft: Point home and config dir towards non-versioned snap home (fixes #3730) 2016-11-14 19:06:05 +01:00
Ben Schulz
ba6aff4a1b gui: Use icons and tooltips for folder size info (fixes #3710)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3731
2016-11-13 13:56:07 +00:00
Audrius Butkevicius
bb23e3940e cmd/strelaysrv: Use listen address for outgoing HTTP requests (fixes #3682) 2016-11-13 09:32:05 +01:00
Audrius Butkevicius
94e4370c7e cmd/strelaysrv: Outbox will get GCed (fixes #3718) 2016-11-13 09:32:05 +01:00
Audrius Butkevicius
38d28c3f4a lib/relay: Close invitation channel in all error cases (fixes #3726) 2016-11-13 09:32:05 +01:00
Audrius Butkevicius
f60b424d70 lib/config: Raw() -> RawCopy() 2016-11-13 09:29:35 +01:00
Audrius Butkevicius
a1a91d5ef4 lib/model: Introducer can remove stuff it introduced (fixes #1015)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3522
2016-11-13 09:29:33 +01:00
Jakob Borg
bfb48b5dde jenkins: Clean should remove old snaps 2016-11-12 10:08:13 +01:00
Jakob Borg
2860813a8e build: Set snap grade to "stable" for releases 2016-11-12 09:47:57 +01:00
Jakob Borg
72538e350d build: Snap versions should not have initial "v"
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3728
2016-11-12 08:36:19 +00:00
Jakob Borg
59f3d1445f Revert "lib/model: Introducer can remove stuff it introduced (fixes #1015)"
This reverts commit 0b88cf1d03.
2016-11-12 08:38:29 +01:00
Audrius Butkevicius
0b88cf1d03 lib/model: Introducer can remove stuff it introduced (fixes #1015)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3522
2016-11-11 15:54:25 +01:00
Audrius Butkevicius
56e2ba29d0 lib/config: Subscribers get a copy of the config
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3722
2016-11-11 14:52:23 +00:00
Jakob Borg
6ec9b84674 test: Fix test config 2016-11-09 09:02:55 +08:00
Leo Arias
afd15392b1 build: Build snaps for ARM
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3717
2016-11-09 00:52:33 +00:00
Jakob Borg
ae4cc94a9d lib/model: Fix locking order in Availability() (fixes #3634)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3714
2016-11-08 06:38:50 +00:00
Jakob Borg
3f9b75b7b3 Revert "lib/model: Introducer can remove stuff it introduced (fixes #1015)"
This reverts commit ec2b097313.
2016-11-08 14:27:32 +08:00
Audrius Butkevicius
ec2b097313 lib/model: Introducer can remove stuff it introduced (fixes #1015)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3522
2016-11-08 00:40:48 +08:00
Audrius Butkevicius
caaab462bc lib/sync: Fix broken build 2016-11-05 02:31:52 +00:00
Audrius Butkevicius
da413b823b lib/sync: Add option for sasha-s/go-deadlock 2016-11-05 02:24:53 +00:00
Audrius Butkevicius
14937e7dd2 build: Fix proto builder on Windows 2016-11-03 22:06:51 +00:00
Audrius Butkevicius
3418497f3d lib/sync: Log everything... 2016-11-03 21:33:33 +00:00
Stefan Kuntz
e408f1061a etc: Added ufw firewall application preset
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3703
2016-11-03 15:46:25 +00:00
佛跳墙
c08fe4e2c5 gui: Remove erroneous right parenthesis
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3699
2016-11-02 13:03:57 +00:00
Jakob Borg
b7e21984a1 gui, man: Update docs & translations 2016-11-01 11:26:25 +01:00
Audrius Butkevicius
7fba8cf759 lib/sync: Print all lockers, add holder to RWMutex 2016-10-30 00:17:25 +01:00
Jakob Borg
0296c23685 lib/protocol: Use DeviceID in protocol messages, with custom marshalling
This makes the device ID a real type that can be used in the protobuf
schema. That avoids the juggling back and forth from []byte in a bunch
of places and simplifies the code.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3695
2016-10-29 21:56:24 +00:00
Jakob Borg
1cdfef4d6a script: Missed a newline in the commit-msg hook output 2016-10-27 21:47:14 +02:00
Jakob Borg
cead20ec91 script: Add commit message check hook 2016-10-27 21:42:05 +02:00
MikeLund
74dd051d51 all: Update docs.s.n links to use https
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3691
2016-10-27 17:02:19 +00:00
Jakob Borg
5473285010 gui: Add help link for sync protocol listen addresses 2016-10-26 21:16:53 +02:00
Jakob Borg
4b3adfa21c vendor: Update gobwas/glob to fix question mark handling 2016-10-23 15:47:31 +02:00
Simon Frei
7c37301c91 lib/ignore: Add directory separator to glob.Compile call
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3674
LGTM: calmh
2016-10-21 07:33:40 +00:00
Jakob Borg
d9040f8038 authors: Add imsodin 2016-10-21 15:23:17 +08:00
Jakob Borg
f41606c0b0 jenkins: Build snap
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3678
2016-10-20 09:31:07 +00:00
Leo Arias
31d9750579 build: Add build method for snapcraft
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3636
2016-10-20 09:16:30 +00:00
Jakob Borg
173fb97832 authors: Add elopio 2016-10-20 16:38:14 +08:00
Wulf Weich
81248c3f56 gui: resurrect old dark theme as black theme
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3676
2016-10-19 09:06:27 +00:00
Frank Isemann
2914a0a0a5 gui: Slightly lighten Text for Disconnected/Scanning in Dark Theme 2016-10-19 08:27:17 +08:00
Audrius Butkevicius
815588daba lib/sync, lib/model: Capture locker routine ID, print locker details on deadlock 2016-10-18 21:00:01 +01:00
Jakob Borg
6152eb6d6d gui: Hide "Failed Items" unless there is an actual failure (fixes #3647)
Since delta indexes it's perfectly normal for us to need files that are
currently unavailable due to devices being disconnected. This doesn't
imply a failure, so we should not show the "Failed Items" line and
corresponding eternal spinner (since it would never be filled in, since
there is no failure).

We still show state "Out of Sync" (correct) and the list of files we
need (correct).
2016-10-17 23:57:58 +02:00
Jakob Borg
ff0ebc196c gui: Improve display of local size, ignore pattern status (fixes #3623)
The discrepancy between global and local sizes is fine and expected in
the presence of ignores. This just moves the "we have ignore patterns"
indication to the actual local size metric, as an explanation of why it
may differ from the global size...
2016-10-17 23:57:58 +02:00
Jakob Borg
4e8c8d7e2c cmd/syncthing, lib/db, lib/model: Track more detailed file/dirs/links/deleted counts 2016-10-17 23:57:43 +02:00
Jakob Borg
b8a90b7eaa lib/model: Remove chatty delta index confirmation
This had its moment when making sure it worked initially, but nowadays
there is not point discussing the obvious on every folder on every
connect.
2016-10-17 09:21:58 +02:00
Jakob Borg
60e7ca4a4c gui, man: Update docs & translations 2016-10-17 09:12:16 +02:00
Dale Visser
3a3c8ec6b8 readme: Add Core Infrastructure Initiative Badge
Skip-check: authors, pr-build

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3515
2016-10-17 07:03:52 +00:00
Benny Ng
05c37e58c1 lib/osutil: Prevent infinite Glob recursion (fixes #3577)
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3665
2016-10-12 20:55:38 +00:00
Jakob Borg
d203dd4770 cmd/syncthing: go fmt traceback.go 2016-10-12 20:37:26 +02:00
Jakob Borg
29ccf10d0b cmd/syncthing: Only SetTraceback on Go 1.7+ (fixes #3664) 2016-10-10 17:16:18 +02:00
Jakob Borg
b49df09fec build: Trivial perf improvement of shouldRebuildAssets 2016-10-09 14:28:20 +02:00
Jakob Borg
ce3e117976 cmd/stdiscosrv: rm 'cmd/stdiscosrv/stdiscosrv' (fixes #3663) 2016-10-09 14:26:43 +02:00
Audrius Butkevicius
309795198d cmd/strelaypoolsrv: Remove hostnames from statusAddr 2016-10-08 10:03:53 +01:00
Audrius Butkevicius
7db00132b2 cmd/strelaysrv: Fix sorting zeros versus undefined 2016-10-07 21:24:47 +01:00
Audrius Butkevicius
76a2862b7a authors: Add Xavier O. 2016-10-07 21:24:47 +01:00
Jakob Borg
215503b4f7 cmd/syncthing: Delay browser start until the GUI is ready (fixes #3619) 2016-10-07 12:10:26 +09:00
Jakob Borg
54d4010f1a gui, man: Update docs and translations 2016-10-07 11:09:19 +09:00
Jakob Borg
cb1b53cfb5 gui: Update English base strings (fixes #3638) 2016-10-07 11:07:12 +09:00
Xav
96e8f94833 skip-check: authors
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3650
2016-10-05 19:13:47 +00:00
MikeLund
1e54a3e801 jenkins: use https when downloading docs (fixes #3651)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3652
2016-10-05 12:17:35 +00:00
Tim Howes
fe9c2b9857 lib/ignore: Match directory contents for patterns ending in / (fixes #3639)
Appends "**" to patterns with a terminal slash, so that directory
contents are ignored, but not the directory itself.
2016-10-04 08:12:55 +09:00
Jakob Borg
2a2177e7fa authors: Add timhowes 2016-10-04 08:11:57 +09:00
Jakob Borg
d1d565e58b cmd/syncthing: Localhost header comparison should be case insensitive 2016-10-03 17:34:13 +09:00
Peter Hoeg
891ff383ec etc/linux-systemd: Remove bogus dependency on networking for user unit
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3627
2016-10-03 03:49:00 +00:00
Nathan Morrison
d322ebd0b9 Add API service for local disk changes
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3626
LGTM: calmh, AudriusButkevicius
2016-09-28 15:54:13 +00:00
Peter Hoeg
50190236bb Ignore pkill error on resume
The ```syncthing-resume.service``` will show as a failed service in case
there are no syncthing processes running after resume but it can be
safely ignored because it makes no difference.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3630
2016-09-28 10:21:15 +00:00
Jakob Borg
d5a0f91cb4 cmd/syncthing: Restore useful levels of traceback on panic 2016-09-26 21:14:17 +02:00
Jakob Borg
467c1b26fb cmd/syncthing, lib/config: Log errors replacing or saving config (ref #3567) 2016-09-24 09:59:09 +02:00
Jakob Borg
3cabecda04 lib/upnp: Correct the result deduplication mechanism (fixes #3578)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3618
2016-09-24 07:33:56 +00:00
Jakob Borg
6d3160b0ab gui: Folder is out of sync when it needs deletes, too (fixes #3588) 2016-09-24 09:11:38 +02:00
Jakob Borg
d328e0fb75 cmd/syncthing: Add selectable sha256 package (fixes #3613, fixes #3614)
This adds autodetection of the fastest hashing library on startup, thus
handling the performance regression. It also adds an environment
variable to control the selection, STHASHING=standard (Go standard
library version, avoids SIGILL crash when the minio library has bugs on
odd CPUs), STHASHING=minio (to force using the minio version) or unset
for the default autodetection.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3617
2016-09-23 19:33:54 +00:00
Jakob Borg
5f01afb7ea build: No need for outdated go2xunit 2016-09-18 21:02:42 +02:00
fti7
6fe2fa5ff0 gui: Slightly lighten the dark theme
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3605
2016-09-18 13:47:36 +00:00
Jakob Borg
b371b1fe34 lib/versioner: Test both spaces and parens in ext versioner paths
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3610
2016-09-18 12:24:55 +00:00
Jakob Borg
90c0a39df8 lib/versioner: Test for external versioner
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3609
2016-09-17 20:34:50 +00:00
Lars K.W. Gohlke
70c5a5dff1 lib/versioner: Rename versioner_test to simple_test
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3603
2016-09-16 11:01:43 +00:00
Jakob Borg
da0b7cc7f2 lib/model: Correct lock taking order in ConnectionStats (fixes #3596)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3597
2016-09-14 19:38:55 +00:00
Jakob Borg
139e9b144e lib/config: Fix tests for changes in previous commit 2016-09-13 22:20:22 +02:00
Jakob Borg
77c0a19451 vendor: Update github.com/d4l3k/messagediff 2016-09-13 22:20:22 +02:00
Jakob Borg
58cbd19742 vendor: Update golang.org/cznic/... 2016-09-13 22:20:22 +02:00
Jakob Borg
9bf6917ae8 vendor: Update golang.org/x/crypto/... 2016-09-13 22:20:22 +02:00
Jakob Borg
897cca0a82 vendor: Update golang.org/x/net/... 2016-09-13 22:20:22 +02:00
Jakob Borg
6af09c61be vendor: Update github.com/thejerf/suture 2016-09-13 22:20:22 +02:00
Jakob Borg
c3c7798446 vendor: Update github.com/gobwas/glob 2016-09-13 22:20:22 +02:00
Jakob Borg
06dc91fadf vendor: Update github.com/syndtr/goleveldb 2016-09-13 22:20:22 +02:00
Jakob Borg
526cab538a jenkins: Don't fetch --prune unnecessarily, print build version on Windows 2016-09-13 22:18:55 +02:00
Jakob Borg
81d19a00aa vendor: Add github.com/cznic/lldb and friends (new recursive dependency) 2016-09-13 21:57:19 +02:00
Jakob Borg
ca755ec9e0 vendor: Add golang.org/x/net/bpf 2016-09-13 21:56:33 +02:00
Jakob Borg
4f6206cb2d build: Simpler creation of Debian packages
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3591
2016-09-12 12:21:07 +00:00
Jakob Borg
7fb53ec954 lib/config: Correct name of discovery-v6-4 server 2016-09-12 11:30:06 +02:00
Jakob Borg
d8b5070ca8 lib/config: Update default set of discovery servers 2016-09-12 09:55:45 +02:00
Jakob Borg
5e99d38412 all: Use github.com/minio/sha256-simd
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3581
2016-09-09 09:57:51 +00:00
Laurent Etiemble
3990014073 cmd/syncthing: Conditionally enable CORS
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3541
LGTM: AudriusButkevicius
2016-09-06 22:16:50 +00:00
Jakob Borg
3e51206a6b build, jenkins: Jenkins version tag should be same as when building manually 2016-09-06 13:02:17 +02:00
Aranjedeath
7569b75d61 cmd/strelaysrv: Correct go get command in README
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3564
2016-09-04 21:06:30 +00:00
Jakob Borg
8fcabac518 jenkins: Add batch file for Windows 2016-09-04 16:43:56 +02:00
Jakob Borg
abb0cfde72 jenkins: Add scripts for automated builds (Linux & Mac) 2016-09-04 15:30:16 +02:00
Jakob Borg
7990ffcc60 cmd/syncthing: Copy config on upgrade, instead of renaming (fixes #3525)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3560
2016-09-03 21:29:32 +00:00
Jakob Borg
49910a1d85 lib/config, cmd/syncthing: Enforce localhost only connections
When the GUI/API is bound to localhost, we enforce that the Host header
looks like localhost. This can be disabled by setting
insecureSkipHostCheck in the GUI config.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3558
2016-09-03 08:33:34 +00:00
Jakob Borg
46a143e80e lib/model: Handle deleted-then-ignored files (fixes #3502)
When files that were previously marked as deleted became ignored, we
used to do nothing at all. This changes that behavior to set the Invalid
bit (that we should rename to Ignored). This then becomes an update to
other devices that they should not trust our knowledge about the file in
question.

Read this diff without whitespace...

Tested by
- creating a bunch of files on s1
- letting them sync to s2
- shutting down s2
- deleting the files on s1 and rescanning
- adding the files to .stignore on s1 and rescanning
- starting up s2 and letting it sync
- observing the files are not deleted on s2, and it considers itself up
  to date.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3557
2016-09-02 13:23:24 +00:00
Jakob Borg
69b7f26e4c lib/model, cmd/syncthing: Also account for deleted files in folder summary events (ref #3496)
This should probably be reflected in the GUI somewhere as well...
2016-09-02 10:45:39 +02:00
Jakob Borg
5b37d0356c lib/model, gui: Correct completion percentages when there are lots of deletes (fixes #3496)
We used to consider deleted files & directories 128 bytes large. After
the delta indexes change a bug slipped in where deleted files would be
weighted according to their old non-deleted size. Both ways are
incorrect (but the latest change made it worse), as if there are more
files deleted than remaining data in the repo the needSize can be
greater than the globalSize, resulting in a negative completion
percentage.

This change makes it so that deleted items are zero bytes large, which
makes more sense. Instead we expose the number of files that we need to
delete as a separate field in the Completion() result, and hack the
percentage down to 95% complete if it was 100% complete but we need to
delete files. This latter part is sort of ugly, but necessary to give
the user some sort of feedback.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3556
2016-09-02 06:45:46 +00:00
Jakob Borg
1188ebbb7b gui, man: Update docs & translations 2016-08-23 10:42:09 +02:00
Audrius Butkevicius
76b903b2e0 lib/upgrade: Cleanup failed upgrades (fixes #3500, fixes #3530)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3535
2016-08-23 06:53:39 +00:00
Audrius Butkevicius
be38c2111f cmd/strelaysrv: Add uPNP support, ability to set listen protocol (fixes #3503, fixes #3505, fixes #3506) 2016-08-23 08:43:27 +02:00
Audrius Butkevicius
1de787fab8 cmd/strelaypoolsrv: Ability to select listen protocol 2016-08-23 08:42:57 +02:00
Audrius Butkevicius
81f683a61c cmd/stdiscosrv: Generate keys if missing on startup (fixes #3511) 2016-08-23 08:41:49 +02:00
Audrius Butkevicius
db6f68d031 cmd/stdiscosrv: Use UTC in database timestamps (fixes #3509) 2016-08-23 08:41:15 +02:00
Jakob Borg
d0a1c805e9 cmd/syncthing: uintptr may not be stored in a variable
Must do the whole uintptr(unsafe.Pointer(&whatever)) directly in the
call to sycall.Call, as per https://golang.org/pkg/unsafe/.
2016-08-22 18:24:26 +02:00
Jakob Borg
00a654845f cmd/syncthing: Remove old temp index dbs on startup (fixes #3529)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3532
2016-08-22 12:19:19 +00:00
Jakob Borg
04dad8485a authors: Add calmh email 2016-08-18 19:38:15 +02:00
Jakob Borg
0b1475169f lib/model: Correct virtual mtime handling (fixes #3516)
We previously set the mtime on the temp file, and then renamed it to the
real path. Unfortunately that means we'd save the real timestamp under
the under the temp name ".syncthing.foo.tmp" when the actual file that
we will look up on the next scan is "foo". This moves the Chtimes later,
ensuring that it gets recorded correctly under the right name.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3519
2016-08-16 18:22:19 +00:00
Audrius Butkevicius
6ec4fbc82b lib/model: Add minumum interval for progress emitter (fixes #3517)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3518
2016-08-16 18:22:01 +00:00
Jakob Borg
18cc7a663b lib: Remove osutil.Remove & osutil.RemoveAll (fixes #3513)
These are no longer required with Go 1.7. Change made by removing the
functions, doing a global s/osutil.Remove/os.Remove/.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3514
2016-08-16 10:01:58 +00:00
Jakob Borg
cf5febad47 build, cmd, lib: Minimum supported compiler version is Go 1.5 2016-08-15 08:37:32 +02:00
Jakob Borg
42849af5a8 cmd/syncthing: Default folder should also have lower case ID 2016-08-13 23:23:18 +02:00
Jakob Borg
e6364407a9 cmd/stdiscosrv: Fix index creation checks on startup 2016-08-12 11:39:10 +02:00
Jakob Borg
480b78f2c8 cmd/stdiscosrv: Longer address in schema 2016-08-12 11:38:37 +02:00
Jakob Borg
fa8f339478 gui: Fix division by zero in completion calc (ref #3493) 2016-08-12 08:49:16 +02:00
Jakob Borg
7776839c82 cmd/syncthing, gui: Improve completion calculation (fixes #3492)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3493
2016-08-12 06:41:43 +00:00
Jakob Borg
7114cacb85 gui, man: Update docs & translations 2016-08-10 11:41:46 +02:00
Jakob Borg
e52be3d83e lib/connections, lib/model: Refactor connection close handling (fixes #3466)
So there were some issues here. The main problem was that
model.Close(deviceID) was overloaded to mean "the connection was closed
by the protocol layer" and "i want to close this connection". That meant
it could get called twice - once *to* close the connection and then once
more when the connection *was* closed.

After this refactor there is instead a Closed(conn) method that is the
callback. I didn't need to change the parameter in the end, but I think
it's clearer what it means when it takes the connection that was closed
instead of a device ID. To close a connection, the new close(deviceID)
method is used instead, which only closes the underlying connection and
leaves the cleanup to the Closed() callback.

I also changed how we do connection switching. Instead of the connection
service calling close and then adding the connection, it just adds the
new connection. The model knows that it already has a connection and
makes sure to close and clean out that one before adding the new
connection.

To make sure to sequence this properly I added a new map of channels
that get created on connection add and closed by Closed(), so that
AddConnection() can do the close and wait for the cleanup to happen
before proceeding.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3490
2016-08-10 09:37:32 +00:00
Antoine Lamielle
c9cf01e0b6 gui: weighting % of devices according to folder size (fixes #1300)
The completion of remote devices was based only on the average of the percentages of all folders, which is irrelevant in case of two folders with very different sizes.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3481
LGTM: calmh, AudriusButkevicius
2016-08-09 19:58:44 +00:00
Jakob Borg
dcbf68e104 lib/versioner: Hack to make test coverage stable from run to run
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3485
2016-08-08 18:27:55 +00:00
Jakob Borg
c2d8c07137 lib/events: Hack to make test coverage stable from run to run
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3484
2016-08-08 18:09:40 +00:00
Jakob Borg
a4ed50ca85 build, lib: Correct total test coverage calculation
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3483
2016-08-08 16:29:32 +00:00
Jakob Borg
b3788c8ea0 authors: Fixup 0x010C 2016-08-08 08:34:36 +02:00
Jakob Borg
946c074a41 authors: Add 0x010C 2016-08-08 08:19:02 +02:00
Jakob Borg
19f79afb0f build: Setup should install golint 2016-08-07 21:58:27 +02:00
Audrius Butkevicius
af3b6f9c83 lib/model, lib/config: Support "live" device removal, folder unsharing and folder configuration changes
Furthermore:
1. Cleans configs received, migrates them as we receive them.
2. Clears indexes of devices we no longer share the folder with

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3478
2016-08-07 16:21:59 +00:00
Jakob Borg
fbe42c156d gui: Move "ignore patterns" away from "remove" in folder edit dialog 2016-08-07 14:26:32 +02:00
Jakob Borg
a1f6cbd354 lib/protocol: Clean away outdated files 2016-08-07 14:24:25 +02:00
Audrius Butkevicius
a4f052ad31 lib/connections: Fix connection switching
It seems that it would be impossible to drop down to relay after establishing a direct connection
Also, we should not drop the existing connection until after we've passed the validation steps,
and it seems it's being dropped in two places unnecesserily at the moment.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3480
2016-08-07 12:20:37 +00:00
Jakob Borg
ea87bcefd6 lib/protocol, lib/model: Implement high precision time stamps (fixes #3305)
This adds a new nanoseconds field to the FileInfo, populates it during
scans and sets the non-truncated time in Chtimes calls.

The actual file modification time is defined as modified_s seconds +
modified_ns nanoseconds. It's expected that the modified_ns field is <=
1e9 (that is, all whole seconds should go in the modified_s field) but
not really enforced. Given that it's an int32 the timestamp can be
adjusted += ~2.9 seconds by the modified_ns field...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3431
2016-08-06 13:05:59 +00:00
Jakob Borg
0655991a19 lib/db, lib/fs, lib/model: Introduce fs.MtimeFS, remove VirtualMtimeRepo
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3479
2016-08-05 17:45:45 +00:00
Jakob Borg
f368d2278f lib/config, lib/connections: Refactor handling of ignored devices (fixes #3470)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3471
2016-08-05 09:29:49 +00:00
Jakob Borg
1eb6db6ca8 cmd/syncthing, lib/...: Correctly handle ignores & invalid file names (fixes #3012, fixes #3457, fixes #3458)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3464
2016-08-05 07:13:52 +00:00
Jakob Borg
a25b63e2df cmd/syncthing: Delete old format indexes after a while (fixes #3468)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3469
2016-08-02 15:44:09 +00:00
Jakob Borg
ffe7a2fcd7 cmd/syncthing, lib/config: Enable HTTP CPU/heap profile collection for users
This adds a config to enable debug functions on the API server, which is
by default disabled. When enabled, the /rest/debug things become
available and become available without requiring a CSRF token (although
authentication is required if configured).

We also add a new endpoint /rest/debug/cpuprof?duration=15s (with the
duration being configurable, defaulting to 30s). This runs a CPU profile
for the duration and returns it as a file. It sets headers so that a
browser will save the file with an informative name.

The same is done for heap profiles, /rest/debug/heapprof, which does not
take any parameters.

The purpose of this is that any user can enable debugging under
advanced, then point their browser to the endpoint above and get a file
that contains a CPU or heap profile we can use, with the filename
telling us what version and architecture the profile is from.

On the command line, this becomes

    curl -O -J http://localhost:8082/rest/debug/cpuprof?duration=5s
    curl: Saved to filename
    'syncthing-cpu-darwin-amd64-v0.14.3+4-g935bcc0-110307.pprof'

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3467
2016-08-02 11:06:45 +00:00
Audrius Butkevicius
08b5a7908f gui: Add one-off notifications that need to be acked
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3254
2016-08-02 08:07:30 +00:00
derekriemer
a8cd9d0154 gui: Improve accessibility (fixes #3297)
skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3463
2016-07-31 22:59:44 +00:00
Jakob Borg
297240facf all: Rename LocalVersion to Sequence (fixes #3461)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3462
2016-07-29 19:54:24 +00:00
Jakob Borg
a022b0cfff gui, man: Update docs & translations 2016-07-28 13:15:14 +02:00
Jakob Borg
72026db599 lib/db, lib/model: Create temp sorting database in config dir (fixes #3449)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3454
2016-07-27 21:38:43 +00:00
Jakob Borg
aafc96f58f lib/model, lib/protocol: Sequence ClusterConfig properly (fixes #3448)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3452
2016-07-27 21:36:25 +00:00
Jakob Borg
7c7e8648ff lib/model: Trigger a puller iteration on connection (fixes #3451)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3453
2016-07-27 21:35:41 +00:00
Jakob Borg
24e2ce0764 build: Allow easy influencing build user and build host
To facilitate reproducible builds.
2016-07-27 23:27:47 +02:00
aviau
d7cb4d407b man: Include stdiscosrv and strelaysrv manpages
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3450
2016-07-27 15:00:10 +00:00
Jakob Borg
66a506e72b lib/scanner: Correctly scan symlinks (fixes #3445)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3446
2016-07-26 11:55:25 +00:00
Jakob Borg
25a7b0a6f8 gui, man: Update docs & translations 2016-07-26 10:53:00 +02:00
Jakob Borg
7aaa1dd8a3 lib/scanner: Recheck file size and modification time after hashing (ref #3440)
To catch the case where the file changed. Also make sure we never let a
size-vs-blocklist mismatch slip through.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3443
2016-07-26 08:51:39 +00:00
Jakob Borg
2a6f164923 lib/scanner: When scanning a file, stick to the size given by Lstat (fixes #3440)
Otherwise if the file grows during scanning the block list will be out
of sync with the stated size and things get confused. We could fixup the
size afterwards based on the block list, but then we might see other
inconsistencies as the mtime should have changed to reflect the new size
etc. Better stick to the original state and let the next scan pick up
the change.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3442
2016-07-25 19:16:49 +00:00
Jakob Borg
0f28626bb4 cmd/syncthing: Generate FolderCompletion events for folders shared with a connecting device (fixes #3436)
This used to happen by itself as the connecting device always sent an
Index message and we triggered on that. Nowadays there's no guarantee
for that, but we anyway need to send out one event to let listeners know
the state of folders shared with the device.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3438
2016-07-25 10:42:17 +00:00
Jakob Borg
6ed22d0885 lib/model: Stricter temporary file permissions
We could have a file to sync with permissions rw------- but we'd create
the temp file with rw-rw-rw- minus umask, usually rw-r--r--. This
potentially exposes private data while the file is being synced.

Similarly, when ignorePerms was set and we were reusing a temp files we
would set the permissions to rw-r--r-- explicitly, potentially
overriding a strict umask that would otherwise have had the file be
rw-------.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3437
2016-07-25 10:18:05 +00:00
Jakob Borg
6715b91a6c build: Remove unused docker-* commands 2016-07-25 08:10:46 +02:00
Jakob Borg
694da60659 lib/db: Reinstate database update locking
The previous commit loosened the locking around database updates.
Apparently that was not fine - what happens is that parallell updates
to the same file for different devices stomp on each others updates to
the global index, leaving it missing one of the two devices.
2016-07-23 20:32:15 +02:00
Jakob Borg
47fa4b0a2c cmd/syncthing, lib/db, lib/model, lib/protocol: Implement delta indexes (fixes #438)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3427
2016-07-23 12:46:31 +00:00
Jakob Borg
8ab6b60778 lib/model: Sort outgoing index updates by LocalVersion
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3411
2016-07-21 17:21:15 +00:00
2245 changed files with 500411 additions and 135415 deletions

2
.gitignore vendored
View File

@@ -5,6 +5,7 @@ stdiscosrv.exe
*.tar.gz
*.zip
*.asc
*.deb
.jshintrc
coverage.out
files/pidx
@@ -15,3 +16,4 @@ syncthing.sig
RELEASE
deb
lib/auto/gui.files.go
snapcraft.yaml

29
AUTHORS
View File

@@ -6,19 +6,22 @@
# The NICKS list is auto generated from this file.
Aaron Bieber (qbit) <qbit@deftly.net>
Adam Piggott (simplypeachy) <aD@simplypeachy.co.uk> <simplypeachy@users.noreply.github.com>
Adam Piggott (ProactiveServices) <aD@simplypeachy.co.uk> <simplypeachy@users.noreply.github.com> <ProactiveServices@users.noreply.github.com>
Adel Qalieh (adelq) <aqalieh95@gmail.com> <adelq@users.noreply.github.com>
Alessandro G. (alessandro.g89) <alessandro.g89@gmail.com>
Alexander Graf (alex2108) <register-github@alex-graf.de>
Alexandre Viau (aviau) <alexandre@alexandreviau.net> <aviau@debian.org>
Anderson Mesquita (andersonvom) <andersonvom@gmail.com>
Andrew Dunham (andrew-d) <andrew@du.nham.ca>
Andrey D (scienmind) <scintertech@cryptolab.net>
Antoine Lamielle (0x010C) <antoine.lamielle@0x010c.fr> <gh@0x010c.fr>
Antony Male (canton7) <antony.male@gmail.com>
Arthur Axel fREW Schmidt (frioux) <frew@afoolishmanifesto.com> <frioux@gmail.com>
Audrius Butkevicius (AudriusButkevicius) <audrius.butkevicius@gmail.com>
Bart De Vries (mogwa1) <devriesb@gmail.com>
Ben Curthoys (bencurthoys) <ben@bencurthoys.com>
Ben Schulz (uok) <ueomkail@gmail.com> <uok@users.noreply.github.com>
Ben Shepherd (benshep) <bjashepherd@gmail.com>
Ben Sidhom (bsidhom) <bsidhom@gmail.com>
Benny Ng (tpng) <benny.tpng@gmail.com>
Brandon Philips (philips) <brandon@ifup.org>
@@ -34,6 +37,7 @@ Colin Kennedy (moshen) <moshen.colin@gmail.com>
Daniel Bergmann (brgmnn) <dan.arne.bergmann@gmail.com> <brgmnn@users.noreply.github.com>
Daniel Harte (norgeous) <daniel@harte.me> <daniel@danielharte.co.uk> <norgeous@users.noreply.github.com>
Daniel Martí (mvdan) <mvdan@mvdan.cc>
Darshil Chanpura (dtchanpura) <dtchanpura@gmail.com> <dcprime314@gmail.com>
David Rimmer (dinosore) <dinosore@dbrsoftware.co.uk>
Denis A. (dva) <denisva@gmail.com>
Dennis Wilson (snnd) <dw@risu.io>
@@ -47,27 +51,35 @@ Felix Unterpaintner (bigbear2nd) <bigbear2nd@gmail.com>
Francois-Xavier Gsell (zukoo) <fxgsell@gmail.com>
Frank Isemann (fti7) <frank@isemann.name>
Gilli Sigurdsson (gillisig) <gilli@vx.is>
Heiko Zuerker (Smiley73) <heiko@zuerker.org>
Jaakko Hannikainen (jgke) <jgke@jgke.fi>
Jacek Szafarkiewicz (hadogenes) <szafar@linux.pl>
Jake Peterson (acogdev) <jake@acogdev.com>
Jakob Borg (calmh) <jakob@nym.se>
Jakob Borg (calmh) <jakob@nym.se> <jakob@kastelo.net>
James Patterson (jpjp) <jamespatterson@operamail.com> <jpjp@users.noreply.github.com>
Jaroslav Malec (dzarda) <dzardacz@gmail.com>
Jaya Chithra (jayachithra) <s.k.jayachithra@gmail.com>
Jens Diemer (jedie) <github.com@jensdiemer.de> <git@jensdiemer.de>
Jochen Voss (seehuhn) <voss@seehuhn.de>
Johan Vromans (sciurius) <jvromans@squirrel.nl>
Jose Manuel Delicado (jmdaweb) <jmdaweb@hotmail.com> <jmdaweb@users.noreply.github.com>
Karol Różycki (krozycki) <rozycki.karol@gmail.com>
Kelong Cong (kc1212) <kc04bc@gmx.com> <kc1212@users.noreply.github.com>
Ken'ichi Kamada (kamadak) <kamada@nanohz.org>
Kevin Allen (ironmig) <kma1660@gmail.com>
Kevin White, Jr. (kwhite17) <kevinwhite1710@gmail.com>
Kurt Fitzner (Kudalufi) <kurt@va1der.ca> <kurt.fitzner@gmail.com>
Lars K.W. Gohlke (lkwg82) <lkwg82@gmx.de>
Laurent Etiemble (letiemble) <laurent.etiemble@gmail.com> <laurent.etiemble@monobjc.net>
Leo Arias (elopio) <yo@elopio.net>
Liu Siyuan (liusy182) <liusy182@gmail.com> <liusy182@hotmail.com>
Lode Hoste (Zillode) <zillode@zillode.be>
Lord Landon Agahnim (LordLandon) <lordlandon@gmail.com>
Majed Abdulaziz (majedev) <majed.alhajry@gmail.com>
Marc Laporte (marclaporte) <marc@marclaporte.com> <marc@laporte.name>
Marc Pujol (kilburn) <kilburn@la3.org>
Marcin Dziadus (marcindziadus) <dziadus.marcin@gmail.com>
Mark Pulford (mpx) <mark@kyne.com.au>
Mateusz Naściszewski (mateon1) <matin1111@wp.pl>
Matt Burke (burkemw3) <mburke@amplify.com> <burkemw3@gmail.com>
Max Schulze (kralo) <max.schulze@online.de> <kralo@users.noreply.github.com>
@@ -75,24 +87,35 @@ Michael Jephcote (Rewt0r) <rewt0r@gmx.com> <Rewt0r@users.noreply.github.com>
Michael Ploujnikov (plouj) <ploujj@gmail.com>
Michael Tilli (pyfisch) <pyfisch@gmail.com>
Nate Morrison (nrm21) <natemorrison@gmail.com>
Niels Peter Roest (Niller303) <nielsproest@hotmail.com> <seje.niels@hotmail.com>
Pascal Jungblut (pascalj) <github@pascalj.com> <mail@pascal-jungblut.com>
Peter Hoeg (peterhoeg) <peter@speartail.com>
Philippe Schommers (filoozoom) <philippe@schommers.be>
Phill Luby (pluby) <phill.luby@newredo.com>
Piotr Bejda (piobpl) <piotrb10@gmail.com>
Robert Carosi (nov1n) <robert@carosi.nl>
Roman Zaynetdinov (zaynetro) <romanznet@gmail.com>
Ross Smith II (rasa) <ross@smithii.com>
Ryan Sullivan (KayoticSully) <kayoticsully@gmail.com>
Sacheendra Talluri (sacheendra) <sacheendra.t@gmail.com>
Scott Klupfel (kluppy) <kluppy@going2blue.com>
Sergey Mishin (ralder) <ralder@yandex.ru>
Simon Frei (imsodin) <freisim93@gmail.com>
Stefan Kuntz (Stefan-Code) <stefan.github@gmail.com> <Stefan.github@gmail.com>
Stefan Tatschner (rumpelsepp) <stefan@sevenbyte.org> <rumpelsepp@sevenbyte.org>
Suhas Gundimeda (snugghash) <suhas.gundimeda@gmail.com> <snugghash@gmail.com>
Tim Abell (timabell) <tim@timwise.co.uk>
Tim Howes (timhowes) <timhowes@berkeley.edu>
Tobias Nygren (tnn2) <tnn@nygren.pp.se>
Tobias Tom (tobiastom) <t.tom@succont.de>
Tomas Cerveny (kozec) <kozec@kozec.com>
Tully Robinson (tojrobinson) <tully@tojr.org>
Tyler Brazier (tylerbrazier) <tyler@tylerbrazier.com>
Unrud (Unrud) <unrud@openaliasbox.org> <Unrud@users.noreply.github.com>
Veeti Paananen (veeti) <veeti.paananen@rojekti.fi>
Victor Buinsky (buinsky) <vix_booja@tut.by>
Vil Brekin (Vilbrekin) <vilbrekin@gmail.com>
William A. Kennington III (wkennington) <william@wkennington.com>
Wulf Weich (wweich) <wweich@users.noreply.github.com> <wweich@gmx.de>
Wulf Weich (wweich) <wweich@users.noreply.github.com> <wweich@gmx.de> <wulf@weich-kr.de>
Xavier O. (damajor) <damajor@gmail.com>
Yannic A. (eipiminus1) <eipiminusone+github@gmail.com> <eipiminus1@users.noreply.github.com>

View File

@@ -33,12 +33,12 @@ latest info on Transifex.
Every contribution is welcome. If you want to contribute but are unsure
where to start, any open issues are fair game! See the [Contribution
Guidelines](http://docs.syncthing.net/dev/contributing.html) for the full
Guidelines](https://docs.syncthing.net/dev/contributing.html) for the full
story on committing code.
## Contributing Documentation
Updates to the [documentation site](http://docs.syncthing.net/) can be
Updates to the [documentation site](https://docs.syncthing.net/) can be
made as pull requests on the [documentation
repository](https://github.com/syncthing/docs).

83
GOALS.md Normal file
View File

@@ -0,0 +1,83 @@
# The Syncthing Goals
Syncthing is a **continuous file synchronization program**. It synchronizes
files between two or more computers. We strive to fulfill the goals below.
The goals are listed in order of importance, the most important one being
the first.
> "Syncing files" here is precise. It means we specifically exclude things
> that are not files - calendar items, instant messages, and so on. If those
> are in fact stored as files on disk, they can of course be synced as
> files.
Syncthing should be:
### 1. Safe From Data Loss
Protecting the user's data is paramount. We take every reasonable precaution
to avoid corrupting the user's files.
> This is the overriding goal, without which synchronizing files becomes
> pointless. This means that we do not make unsafe trade offs for the sake
> of performance or, in some cases, even usability.
### 2. Secure Against Attackers
Again, protecting the user's data is paramount. Regardless of our other
goals we must never allow the user's data to be susceptible to eavesdropping
or modification by unauthorized parties.
> This should be understood in context. It is not necessarily reasonable to
> expect Syncthing to be resistant against well equipped state level
> attackers. We will however do our best. Note also that this is different
> from anonymity which is not, currently, a goal.
### 3. Easy to Use
Syncthing should be approachable, understandable and inclusive.
> Complex concepts and maths form the base of Synchting's functionality.
> This should nonetheless be abstracted or hidden to a degree where
> Syncthing is usable by the general public.
### 4. Automatic
User interaction should be required only when absolutely necessary.
> Specifically this means that changes to files are picked up without
> prompting, conflicts are resolved without prompting and connections are
> maintained without prompting. We only prompt the user when it is required
> to fulfill one of the (overriding) Secure, Safe or Easy goals.
### 5. Universally Available
Syncthing should run on every common computer. We are mindful that the
latest technology is not always available to any given individual.
> Computers include desktops, laptops, servers, virtual machines, small
> general purpose computers such as Raspberry Pis and, *where possible*,
> tablets and phones. NAS appliances, toasters, cars, firearms, thermostats
> and so on may include computing capabitilies but it is not our goal for
> Syncthing to run smoothly on these devices.
### 6. For Individuals
Syncthing is primarily about empowering the individual user with safe,
secure and easy to use file synchronization.
> We acknowledge that it's also useful in an enterprise setting and include
> functionality to support that. If this is in conflict with the
> requirements of the individual, those will however take priority.
### 7. Everything Else
There are many things we care about that don't make it on to the list. It is
fine to optimize for these values as well, as long as they are not in
conflict with the stated goals above.
> For example, performance is a thing we care about. We just don't care more
> about it than safety, security, etc. Maintainability of the code base and
> providing entertainment value for the maintainers are also things that
> matter. It is understood that there are aspects of Syncthing that are
> suboptimal or even in opposition with the goals above. However, we
> continuously strive to align Syncthing more and more with these goals.

View File

@@ -1,11 +1,17 @@
Do not report security issues in this bug tracker. Instead, contact
security@syncthing.net directly - see https://syncthing.net/security.html
for more information.
### DO NOT REPORT SECURITY ISSUES IN THIS ISSUE TRACKER
If your issue is a support request ("How do I get my devices to connect?"
or similar), please use the support forum at https://forum.syncthing.net/
where a large number of helpful people hang out. This issue tracker is for
reporting bugs or feature requests directly to the developers.
Instead, contact security@syncthing.net directly - see
https://syncthing.net/security.html for more information.
### DO NOT POST SUPPORT REQUESTS OR GENERAL QUESTIONS IN THIS ISSUE TRACKER
Please use the forum at https://forum.syncthing.net/ where a large number of
helpful people hang out. This issue tracker is for reporting bugs or feature
requests directly to the developers. Worst case you might get a short
"that's a bug, please report it on GitHub" response on the forum, in which
case we thank you for your patience and following our advice. :)
### Please do post actual bug reports and feature requests.
If your issue is a bug report, replace this boilerplate with a description
of the problem, being sure to include at least:

View File

@@ -357,7 +357,7 @@ Exhibit A - Source Code Form License Notice
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
file, You can obtain one at https://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular
file, then You may include the notice in a location (such as a LICENSE

39
NICKS
View File

@@ -1,7 +1,11 @@
# This file maps email addresses used in commits to nicks used the changelog.
# It is auto generated from the AUTHORS file by script/authors.go.
0x010C <antoine.lamielle@0x010c.fr>
0x010C <gh@0x010c.fr>
acogdev <jake@acogdev.com>
adelq <aqalieh95@gmail.com>
adelq <adelq@users.noreply.github.com>
alessandro.g89 <alessandro.g89@gmail.com>
alex2108 <register-github@alex-graf.de>
andersonvom <andersonvom@gmail.com>
@@ -11,6 +15,7 @@ AudriusButkevicius <audrius.butkevicius@gmail.com>
aviau <alexandre@alexandreviau.net>
aviau <aviau@debian.org>
bencurthoys <ben@bencurthoys.com>
benshep <bjashepherd@gmail.com>
bigbear2nd <bigbear2nd@gmail.com>
brbecker <brbecker@gmail.com>
brendanlong <self@brendanlong.com>
@@ -21,6 +26,7 @@ buinsky <vix_booja@tut.by>
burkemw3 <mburke@amplify.com>
burkemw3 <burkemw3@gmail.com>
calmh <jakob@nym.se>
calmh <jakob@kastelo.net>
canton7 <antony.male@gmail.com>
Cathryne <cathryne.linenweaver@gmail.com>
Cathryne <Cathryne@users.noreply.github.com>
@@ -28,11 +34,15 @@ cdata <chris@scriptolo.gy>
cdhowie <me@chrishowie.com>
ceh <emil@hessman.se>
cqcallaw <enlightened.despot@gmail.com>
damajor <damajor@gmail.com>
dinosore <dinosore@dbrsoftware.co.uk>
dtchanpura <dtchanpura@gmail.com>
dtchanpura <dcprime314@gmail.com>
dva <denisva@gmail.com>
dzarda <dzardacz@gmail.com>
eipiminus1 <eipiminusone+github@gmail.com>
eipiminus1 <eipiminus1@users.noreply.github.com>
elopio <yo@elopio.net>
facastagnini <federico.castagnini@gmail.com>
filoozoom <philippe@schommers.be>
frioux <frew@afoolishmanifesto.com>
@@ -40,11 +50,15 @@ frioux <frioux@gmail.com>
fti7 <frank@isemann.name>
gillisig <gilli@vx.is>
hadogenes <szafar@linux.pl>
imsodin <freisim93@gmail.com>
ironmig <kma1660@gmail.com>
jarlebring <jarlebring@gmail.com>
jayachithra <s.k.jayachithra@gmail.com>
jedie <github.com@jensdiemer.de>
jedie <git@jensdiemer.de>
jgke <jgke@jgke.fi>
jmdaweb <jmdaweb@hotmail.com>
jmdaweb <jmdaweb@users.noreply.github.com>
jpjp <jamespatterson@operamail.com>
jpjp <jpjp@users.noreply.github.com>
kamadak <kamada@nanohz.org>
@@ -57,8 +71,13 @@ kozec <kozec@kozec.com>
kralo <max.schulze@online.de>
kralo <kralo@users.noreply.github.com>
krozycki <rozycki.karol@gmail.com>
Kudalufi <kurt@va1der.ca>
Kudalufi <kurt.fitzner@gmail.com>
kwhite17 <kevinwhite1710@gmail.com>
letiemble <laurent.etiemble@gmail.com>
letiemble <laurent.etiemble@monobjc.net>
liusy182 <liusy182@gmail.com>
liusy182 <liusy182@hotmail.com>
lkwg82 <lkwg82@gmx.de>
LordLandon <lordlandon@gmail.com>
majedev <majed.alhajry@gmail.com>
@@ -69,10 +88,14 @@ mateon1 <matin1111@wp.pl>
mogwa1 <devriesb@gmail.com>
moshen <moshen.colin@gmail.com>
Moter8 <moter8@gmail.com>
mpx <mark@kyne.com.au>
mvdan <mvdan@mvdan.cc>
Niller303 <nielsproest@hotmail.com>
Niller303 <seje.niels@hotmail.com>
norgeous <daniel@harte.me>
norgeous <daniel@danielharte.co.uk>
norgeous <norgeous@users.noreply.github.com>
nov1n <robert@carosi.nl>
nrm21 <natemorrison@gmail.com>
Nutomic <me@nutomic.com>
pascalj <github@pascalj.com>
@@ -82,26 +105,36 @@ philips <brandon@ifup.org>
piobpl <piotrb10@gmail.com>
plouj <ploujj@gmail.com>
pluby <phill.luby@newredo.com>
ProactiveServices <aD@simplypeachy.co.uk>
ProactiveServices <simplypeachy@users.noreply.github.com>
ProactiveServices <ProactiveServices@users.noreply.github.com>
pyfisch <pyfisch@gmail.com>
qbit <qbit@deftly.net>
ralder <ralder@yandex.ru>
rasa <ross@smithii.com>
Rewt0r <rewt0r@gmx.com>
Rewt0r <Rewt0r@users.noreply.github.com>
rumpelsepp <stefan@sevenbyte.org>
rumpelsepp <rumpelsepp@sevenbyte.org>
sacheendra <sacheendra.t@gmail.com>
scienmind <scintertech@cryptolab.net>
sciurius <jvromans@squirrel.nl>
seehuhn <voss@seehuhn.de>
simplypeachy <aD@simplypeachy.co.uk>
simplypeachy <simplypeachy@users.noreply.github.com>
Smiley73 <heiko@zuerker.org>
snnd <dw@risu.io>
snugghash <suhas.gundimeda@gmail.com>
snugghash <snugghash@gmail.com>
Stefan-Code <stefan.github@gmail.com>
Stefan-Code <Stefan.github@gmail.com>
timabell <tim@timwise.co.uk>
timhowes <timhowes@berkeley.edu>
tnn2 <tnn@nygren.pp.se>
tobiastom <t.tom@succont.de>
tojrobinson <tully@tojr.org>
tpng <benny.tpng@gmail.com>
tylerbrazier <tyler@tylerbrazier.com>
Unrud <unrud@openaliasbox.org>
Unrud <Unrud@users.noreply.github.com>
uok <ueomkail@gmail.com>
uok <uok@users.noreply.github.com>
veeti <veeti.paananen@rojekti.fi>
@@ -110,6 +143,8 @@ wkennington <william@wkennington.com>
WSGCSysadmin <e.meitner@willystreet.coop>
wweich <wweich@users.noreply.github.com>
wweich <wweich@gmx.de>
wweich <wulf@weich-kr.de>
xduugu <cedric@gmx.ca>
zaynetro <romanznet@gmail.com>
Zillode <zillode@zillode.be>
zukoo <fxgsell@gmail.com>

View File

@@ -1,23 +1,59 @@
# Syncthing
[![Syncthing][14]][15]
[![Latest Build (Official)](https://img.shields.io/jenkins/s/http/build.syncthing.net/syncthing.svg?style=flat-square&label=unix%20build)](http://build.syncthing.net/job/syncthing/lastBuild/)
[![API Documentation](https://img.shields.io/badge/api-Godoc-blue.svg?style=flat-square)](http://godoc.org/github.com/syncthing/syncthing)
---
[![Latest Downloads](https://img.shields.io/badge/latest-downloads-brightgreen.svg?style=flat-square)](https://build.syncthing.net/latest/)
[![Latest Linux & Cross Build](https://img.shields.io/teamcity/https/build.syncthing.net/s/Syncthing_BuildLinuxCross.svg?style=flat-square&label=linux+%26+cross+build)](https://build.syncthing.net/viewType.html?buildTypeId=Syncthing_BuildLinuxCross&guest=1)
[![Latest Windows Build](https://img.shields.io/teamcity/https/build.syncthing.net/s/Syncthing_BuildWindows.svg?style=flat-square&label=windows+build)](https://build.syncthing.net/viewType.html?buildTypeId=Syncthing_BuildWindows&guest=1)
[![Latest Mac Build](https://img.shields.io/teamcity/https/build.syncthing.net/s/Syncthing_BuildMac.svg?style=flat-square&label=mac+build)](https://build.syncthing.net/viewType.html?buildTypeId=Syncthing_BuildMac&guest=1)
[![MPLv2 License](https://img.shields.io/badge/license-MPLv2-blue.svg?style=flat-square)](https://www.mozilla.org/MPL/2.0/)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/88/badge)](https://bestpractices.coreinfrastructure.org/projects/88)
[![Go Report Card](https://goreportcard.com/badge/github.com/syncthing/syncthing)](https://goreportcard.com/report/github.com/syncthing/syncthing)
This is the Syncthing project which pursues the following goals:
## Goals
1. Define a protocol for synchronization of a folder between a number of
collaborating devices. This protocol should be well defined, unambiguous,
easily understood, free to use, efficient, secure and language neutral.
This is called the [Block Exchange Protocol][1].
Syncthing is a **continuous file synchronization program**. It synchronizes
files between two or more computers. We strive to fulfill the goals below.
The goals are listed in order of importance, the most important one being
the first. This is the summary version of the goal list - for more
commentary, see the full [Goals document][13].
2. Provide the reference implementation to demonstrate the usability of
said protocol. This is the `syncthing` utility. We hope that
alternative, compatible implementations of the protocol will arise.
Syncthing should be:
The two are evolving together; the protocol is not to be considered
stable until Syncthing 1.0 is released, at which point it is locked down
for incompatible changes.
1. Safe From Data Loss
Protecting the user's data is paramount. We take every reasonable
precaution to avoid corrupting the user's files.
2. Secure Against Attackers
Again, protecting the user's data is paramount. Regardless of our other
goals we must never allow the user's data to be susceptible to
eavesdropping or modification by unauthorized parties.
3. Easy to Use
Syncthing should be approachable, understandable and inclusive.
4. Automatic
User interaction should be required only when absolutely necessary.
5. Universally Available
Syncthing should run on every common computer. We are mindful that the
latest technology is not always available to any given individual.
6. For Individuals
Syncthing is primarily about empowering the individual user with safe,
secure and easy to use file synchronization.
7. Everything Else
There are many things we care about that don't make it on to the list. It
is fine to optimize for these values, as long as they are not in conflict
with the stated goals above.
## Getting Started
@@ -60,15 +96,18 @@ Please see the [Syncthing documentation site][6].
All code is licensed under the [MPLv2 License][7].
[1]: http://docs.syncthing.net/specs/bep-v1.html
[2]: http://docs.syncthing.net/intro/getting-started.html
[1]: https://docs.syncthing.net/specs/bep-v1.html
[2]: https://docs.syncthing.net/intro/getting-started.html
[3]: https://github.com/syncthing/syncthing/blob/master/etc
[4]: http://www.freenode.net/irc_servers.shtml
[5]: http://docs.syncthing.net/dev/building.html
[6]: http://docs.syncthing.net/
[4]: https://www.freenode.net/
[5]: https://docs.syncthing.net/dev/building.html
[6]: https://docs.syncthing.net/
[7]: https://github.com/syncthing/syncthing/blob/master/LICENSE
[8]: https://forum.syncthing.net/
[9]: https://kiwiirc.com/client/irc.freenode.net/#syncthing
[10]: https://github.com/syncthing/syncthing/issues
[11]: http://docs.syncthing.net/users/contrib.html#gui-wrappers
[11]: https://docs.syncthing.net/users/contrib.html#gui-wrappers
[12]: https://www.bountysource.com/teams/syncthing/issues
[13]: https://github.com/syncthing/syncthing/blob/master/GOALS.md
[14]: assets/logo-text-128.png
[15]: https://syncthing.net/

View File

Binary file not shown.

655
build.go
View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
// +build ignore
@@ -12,7 +12,10 @@ import (
"archive/tar"
"archive/zip"
"bytes"
"compress/flate"
"compress/gzip"
"crypto/sha256"
"errors"
"flag"
"fmt"
"io"
@@ -26,29 +29,36 @@ import (
"runtime"
"strconv"
"strings"
"syscall"
"text/template"
"time"
)
var (
versionRe = regexp.MustCompile(`-[0-9]{1,3}-g[0-9a-f]{5,10}`)
goarch string
goos string
noupgrade bool
version string
goVersion float64
race bool
debug = os.Getenv("BUILDDEBUG") != ""
versionRe = regexp.MustCompile(`-[0-9]{1,3}-g[0-9a-f]{5,10}`)
goarch string
goos string
noupgrade bool
version string
goVersion float64
race bool
debug = os.Getenv("BUILDDEBUG") != ""
noBuildGopath bool
extraTags string
installSuffix string
pkgdir string
)
type target struct {
name string
buildPkg string
binaryName string
archiveFiles []archiveFile
debianFiles []archiveFile
tags []string
name string
debname string
debdeps []string
debpost string
description string
buildPkg string
binaryName string
archiveFiles []archiveFile
installationFiles []archiveFile
tags []string
}
type archiveFile struct {
@@ -61,14 +71,18 @@ var targets = map[string]target{
"all": {
// Only valid for the "build" and "install" commands as it lacks all
// the archive creation stuff.
buildPkg: "./cmd/...",
buildPkg: "github.com/syncthing/syncthing/cmd/...",
tags: []string{"purego"},
},
"syncthing": {
// The default target for "build", "install", "tar", "zip", "deb", etc.
name: "syncthing",
buildPkg: "./cmd/syncthing",
binaryName: "syncthing", // .exe will be added automatically for Windows builds
name: "syncthing",
debname: "syncthing",
debdeps: []string{"libc6", "procps"},
debpost: "script/post-upgrade",
description: "Open Source Continuous File Synchronization",
buildPkg: "github.com/syncthing/syncthing/cmd/syncthing",
binaryName: "syncthing", // .exe will be added automatically for Windows builds
archiveFiles: []archiveFile{
{src: "{{binary}}", dst: "{{binary}}", perm: 0755},
{src: "README.md", dst: "README.txt", perm: 0644},
@@ -76,7 +90,7 @@ var targets = map[string]target{
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
// All files from etc/ and extra/ added automatically in init().
},
debianFiles: []archiveFile{
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "README.md", dst: "deb/usr/share/doc/syncthing/README.txt", perm: 0644},
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing/LICENSE.txt", perm: 0644},
@@ -94,58 +108,70 @@ var targets = map[string]target{
{src: "etc/linux-systemd/system/syncthing@.service", dst: "deb/lib/systemd/system/syncthing@.service", perm: 0644},
{src: "etc/linux-systemd/system/syncthing-resume.service", dst: "deb/lib/systemd/system/syncthing-resume.service", perm: 0644},
{src: "etc/linux-systemd/user/syncthing.service", dst: "deb/usr/lib/systemd/user/syncthing.service", perm: 0644},
{src: "etc/firewall-ufw/syncthing", dst: "deb/etc/ufw/applications.d/syncthing", perm: 0644},
},
},
"stdiscosrv": {
name: "stdiscosrv",
buildPkg: "./cmd/stdiscosrv",
binaryName: "stdiscosrv", // .exe will be added automatically for Windows builds
name: "stdiscosrv",
debname: "syncthing-discosrv",
debdeps: []string{"libc6"},
description: "Syncthing Discovery Server",
buildPkg: "github.com/syncthing/syncthing/cmd/stdiscosrv",
binaryName: "stdiscosrv", // .exe will be added automatically for Windows builds
archiveFiles: []archiveFile{
{src: "{{binary}}", dst: "{{binary}}", perm: 0755},
{src: "cmd/stdiscosrv/README.md", dst: "README.txt", perm: 0644},
{src: "cmd/stdiscosrv/LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
},
debianFiles: []archiveFile{
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "cmd/stdiscosrv/README.md", dst: "deb/usr/share/doc/stdiscosrv/README.txt", perm: 0644},
{src: "cmd/stdiscosrv/LICENSE", dst: "deb/usr/share/doc/stdiscosrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/stdiscosrv/AUTHORS.txt", perm: 0644},
{src: "cmd/stdiscosrv/README.md", dst: "deb/usr/share/doc/syncthing-discosrv/README.txt", perm: 0644},
{src: "cmd/stdiscosrv/LICENSE", dst: "deb/usr/share/doc/syncthing-discosrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-discosrv/AUTHORS.txt", perm: 0644},
{src: "man/stdiscosrv.1", dst: "deb/usr/share/man/man1/stdiscosrv.1", perm: 0644},
},
tags: []string{"purego"},
},
"strelaysrv": {
name: "strelaysrv",
buildPkg: "./cmd/strelaysrv",
binaryName: "strelaysrv", // .exe will be added automatically for Windows builds
name: "strelaysrv",
debname: "syncthing-relaysrv",
debdeps: []string{"libc6"},
description: "Syncthing Relay Server",
buildPkg: "github.com/syncthing/syncthing/cmd/strelaysrv",
binaryName: "strelaysrv", // .exe will be added automatically for Windows builds
archiveFiles: []archiveFile{
{src: "{{binary}}", dst: "{{binary}}", perm: 0755},
{src: "cmd/strelaysrv/README.md", dst: "README.txt", perm: 0644},
{src: "cmd/strelaysrv/LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
},
debianFiles: []archiveFile{
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "cmd/strelaysrv/README.md", dst: "deb/usr/share/doc/strelaysrv/README.txt", perm: 0644},
{src: "cmd/strelaysrv/LICENSE", dst: "deb/usr/share/doc/strelaysrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/strelaysrv/AUTHORS.txt", perm: 0644},
{src: "cmd/strelaysrv/README.md", dst: "deb/usr/share/doc/syncthing-relaysrv/README.txt", perm: 0644},
{src: "cmd/strelaysrv/LICENSE", dst: "deb/usr/share/doc/syncthing-relaysrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-relaysrv/AUTHORS.txt", perm: 0644},
{src: "man/strelaysrv.1", dst: "deb/usr/share/man/man1/strelaysrv.1", perm: 0644},
},
},
"strelaypoolsrv": {
name: "strelaypoolsrv",
buildPkg: "./cmd/strelaypoolsrv",
binaryName: "strelaypoolsrv", // .exe will be added automatically for Windows builds
name: "strelaypoolsrv",
debname: "syncthing-relaypoolsrv",
debdeps: []string{"libc6"},
description: "Syncthing Relay Pool Server",
buildPkg: "github.com/syncthing/syncthing/cmd/strelaypoolsrv",
binaryName: "strelaypoolsrv", // .exe will be added automatically for Windows builds
archiveFiles: []archiveFile{
{src: "{{binary}}", dst: "{{binary}}", perm: 0755},
{src: "cmd/strelaypoolsrv/README.md", dst: "README.txt", perm: 0644},
{src: "cmd/strelaypoolsrv/LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
},
debianFiles: []archiveFile{
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "cmd/strelaypoolsrv/README.md", dst: "deb/usr/share/doc/relaysrv/README.txt", perm: 0644},
{src: "cmd/strelaypoolsrv/LICENSE", dst: "deb/usr/share/doc/relaysrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/relaysrv/AUTHORS.txt", perm: 0644},
{src: "cmd/strelaypoolsrv/README.md", dst: "deb/usr/share/doc/syncthing-relaypoolsrv/README.txt", perm: 0644},
{src: "cmd/strelaypoolsrv/LICENSE", dst: "deb/usr/share/doc/syncthing-relaypoolsrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-relaypoolsrv/AUTHORS.txt", perm: 0644},
},
},
}
@@ -161,17 +187,16 @@ func init() {
syncthingPkg.archiveFiles = append(syncthingPkg.archiveFiles, archiveFile{src: file, dst: file, perm: 0644})
}
for _, file := range listFiles("extra") {
syncthingPkg.debianFiles = append(syncthingPkg.debianFiles, archiveFile{src: file, dst: "deb/usr/share/doc/syncthing/" + filepath.Base(file), perm: 0644})
syncthingPkg.installationFiles = append(syncthingPkg.installationFiles, archiveFile{src: file, dst: "deb/usr/share/doc/syncthing/" + filepath.Base(file), perm: 0644})
}
targets["syncthing"] = syncthingPkg
}
const minGoVersion = 1.3
func main() {
log.SetOutput(os.Stdout)
log.SetFlags(0)
parseFlags()
if debug {
t0 := time.Now()
defer func() {
@@ -179,28 +204,45 @@ func main() {
}()
}
if os.Getenv("GOPATH") == "" {
setGoPath()
if gopath := gopath(); gopath == "" {
gopath, err := temporaryBuildDir()
if err != nil {
log.Fatal(err)
}
if !noBuildGopath {
lazyRebuildAssets()
if err := buildGOPATH(gopath); err != nil {
log.Fatal(err)
}
}
os.Setenv("GOPATH", gopath)
log.Println("GOPATH is", gopath)
} else {
inside := false
wd, _ := os.Getwd()
wd, _ = filepath.EvalSymlinks(wd)
for _, p := range filepath.SplitList(gopath) {
p, _ = filepath.EvalSymlinks(p)
if filepath.Join(p, "src/github.com/syncthing/syncthing") == wd {
inside = true
break
}
}
if !inside {
fmt.Println("You seem to have GOPATH set but the Syncthing source not placed correctly within it, which may cause problems.")
}
}
// We use Go 1.5+ vendoring.
os.Setenv("GO15VENDOREXPERIMENT", "1")
// Set path to $GOPATH/bin:$PATH so that we can for sure find tools we
// might have installed during "build.go setup".
os.Setenv("PATH", fmt.Sprintf("%s%cbin%c%s", os.Getenv("GOPATH"), os.PathSeparator, os.PathListSeparator, os.Getenv("PATH")))
parseFlags()
checkArchitecture()
goVersion, _ = checkRequiredGoVersion()
// Invoking build.go with no parameters at all builds everything (incrementally),
// which is what you want for maximum error checking during development.
if flag.NArg() == 0 {
runCommand("install", targets["all"])
runCommand("vet", target{})
runCommand("lint", target{})
} else {
// with any command given but not a target, the target is
// "syncthing". So "go run build.go install" is "go run build.go install
@@ -220,7 +262,7 @@ func main() {
func checkArchitecture() {
switch goarch {
case "386", "amd64", "arm", "arm64", "ppc64", "ppc64le":
case "386", "amd64", "arm", "arm64", "ppc64", "ppc64le", "mips", "mipsle":
break
default:
log.Printf("Unknown goarch %q; proceed with caution!", goarch)
@@ -237,20 +279,23 @@ func runCommand(cmd string, target target) {
if noupgrade {
tags = []string{"noupgrade"}
}
tags = append(tags, strings.Fields(extraTags)...)
install(target, tags)
metalintShort()
case "build":
var tags []string
if noupgrade {
tags = []string{"noupgrade"}
}
tags = append(tags, strings.Fields(extraTags)...)
build(target, tags)
case "test":
test("./lib/...", "./cmd/...")
test("github.com/syncthing/syncthing/lib/...", "github.com/syncthing/syncthing/cmd/...")
case "bench":
bench("./lib/...", "./cmd/...")
bench("github.com/syncthing/syncthing/lib/...", "github.com/syncthing/syncthing/cmd/...")
case "assets":
rebuildAssets()
@@ -273,87 +318,73 @@ func runCommand(cmd string, target target) {
case "deb":
buildDeb(target)
case "snap":
buildSnap(target)
case "clean":
clean()
case "vet":
vet("build.go")
vet("cmd", "lib")
metalintShort()
case "lint":
lint(".")
lint("./cmd/...")
lint("./lib/...")
metalintShort()
case "metalint":
if isGometalinterInstalled() {
dirs := []string{".", "./cmd/...", "./lib/..."}
ok := gometalinter("deadcode", dirs, "test/util.go")
ok = gometalinter("structcheck", dirs) && ok
ok = gometalinter("varcheck", dirs) && ok
if !ok {
os.Exit(1)
}
metalint()
case "version":
fmt.Println(getVersion())
case "gopath":
gopath, err := temporaryBuildDir()
if err != nil {
log.Fatal(err)
}
fmt.Println(gopath)
default:
log.Fatalf("Unknown command %q", cmd)
}
}
// setGoPath sets GOPATH correctly with the assumption that we are
// in $GOPATH/src/github.com/syncthing/syncthing.
func setGoPath() {
cwd, err := os.Getwd()
if err != nil {
log.Fatal(err)
}
gopath := filepath.Clean(filepath.Join(cwd, "../../../../"))
log.Println("GOPATH is", gopath)
os.Setenv("GOPATH", gopath)
}
func parseFlags() {
flag.StringVar(&goarch, "goarch", runtime.GOARCH, "GOARCH")
flag.StringVar(&goos, "goos", runtime.GOOS, "GOOS")
flag.BoolVar(&noupgrade, "no-upgrade", noupgrade, "Disable upgrade functionality")
flag.StringVar(&version, "version", getVersion(), "Set compiled in version string")
flag.BoolVar(&race, "race", race, "Use race detector")
flag.BoolVar(&noBuildGopath, "no-build-gopath", noBuildGopath, "Don't build GOPATH, assume it's OK")
flag.StringVar(&extraTags, "tags", extraTags, "Extra tags, space separated")
flag.StringVar(&installSuffix, "installsuffix", installSuffix, "Install suffix, optional")
flag.StringVar(&pkgdir, "pkgdir", "", "Set -pkgdir parameter for `go build`")
flag.Parse()
}
func checkRequiredGoVersion() (float64, bool) {
re := regexp.MustCompile(`go(\d+\.\d+)`)
ver := runtime.Version()
if m := re.FindStringSubmatch(ver); len(m) == 2 {
vs := string(m[1])
// This is a standard go build. Verify that it's new enough.
f, err := strconv.ParseFloat(vs, 64)
if err != nil {
log.Printf("*** Couldn't parse Go version out of %q.\n*** This isn't known to work, proceed on your own risk.", vs)
return 0, false
}
if f < 1.5 {
log.Printf("*** Go version %.01f doesn't support the vendoring mechanism.\n*** Ensure correct dependencies in your $GOPATH.", f)
} else if f < minGoVersion {
log.Fatalf("*** Go version %.01f is less than required %.01f.\n*** This is known not to work, not proceeding.", f, minGoVersion)
}
return f, true
func setup() {
packages := []string{
"github.com/alecthomas/gometalinter",
"github.com/AlekSi/gocov-xml",
"github.com/axw/gocov/gocov",
"github.com/FiloSottile/gvt",
"github.com/golang/lint/golint",
"github.com/gordonklaus/ineffassign",
"github.com/mdempsky/unconvert",
"github.com/mitchellh/go-wordwrap",
"github.com/opennota/check/cmd/...",
"github.com/tsenart/deadcode",
"golang.org/x/net/html",
"golang.org/x/tools/cmd/cover",
"honnef.co/go/tools/cmd/gosimple",
"honnef.co/go/tools/cmd/staticcheck",
"honnef.co/go/tools/cmd/unused",
}
for _, pkg := range packages {
fmt.Println(pkg)
runPrint("go", "get", "-u", pkg)
}
log.Printf("*** Unknown Go version %q.\n*** This isn't known to work, proceed on your own risk.", ver)
return 0, false
}
func setup() {
runPrint("go", "get", "-v", "golang.org/x/tools/cmd/cover")
runPrint("go", "get", "-v", "golang.org/x/net/html")
runPrint("go", "get", "-v", "github.com/FiloSottile/gvt")
runPrint("go", "get", "-v", "github.com/axw/gocov/gocov")
runPrint("go", "get", "-v", "github.com/AlekSi/gocov-xml")
runPrint("go", "get", "-v", "bitbucket.org/tebeka/go2xunit")
runPrint("go", "get", "-v", "github.com/alecthomas/gometalinter")
runPrint("go", "get", "-v", "github.com/mitchellh/go-wordwrap")
runPrint("go", "install", "-v", "github.com/syncthing/syncthing/vendor/github.com/gogo/protobuf/protoc-gen-gogofast")
}
func test(pkgs ...string) {
@@ -389,9 +420,15 @@ func install(target target, tags []string) {
}
os.Setenv("GOBIN", filepath.Join(cwd, "bin"))
args := []string{"install", "-v", "-ldflags", ldflags()}
if pkgdir != "" {
args = append(args, "-pkgdir", pkgdir)
}
if len(tags) > 0 {
args = append(args, "-tags", strings.Join(tags, " "))
}
if installSuffix != "" {
args = append(args, "-installsuffix", installSuffix)
}
if race {
args = append(args, "-race")
}
@@ -407,11 +444,17 @@ func build(target target, tags []string) {
tags = append(target.tags, tags...)
rmr(target.binaryName)
rmr(target.BinaryName())
args := []string{"build", "-i", "-v", "-ldflags", ldflags()}
if pkgdir != "" {
args = append(args, "-pkgdir", pkgdir)
}
if len(tags) > 0 {
args = append(args, "-tags", strings.Join(tags, " "))
}
if installSuffix != "" {
args = append(args, "-installsuffix", installSuffix)
}
if race {
args = append(args, "-race")
}
@@ -435,22 +478,20 @@ func buildTar(target target) {
build(target, tags)
if goos == "darwin" {
macosCodesign(target.binaryName)
macosCodesign(target.BinaryName())
}
for i := range target.archiveFiles {
target.archiveFiles[i].src = strings.Replace(target.archiveFiles[i].src, "{{binary}}", target.binaryName, 1)
target.archiveFiles[i].dst = strings.Replace(target.archiveFiles[i].dst, "{{binary}}", target.binaryName, 1)
target.archiveFiles[i].src = strings.Replace(target.archiveFiles[i].src, "{{binary}}", target.BinaryName(), 1)
target.archiveFiles[i].dst = strings.Replace(target.archiveFiles[i].dst, "{{binary}}", target.BinaryName(), 1)
target.archiveFiles[i].dst = name + "/" + target.archiveFiles[i].dst
}
tarGz(filename, target.archiveFiles)
log.Println(filename)
fmt.Println(filename)
}
func buildZip(target target) {
target.binaryName += ".exe"
name := archiveName(target)
filename := name + ".zip"
@@ -463,21 +504,21 @@ func buildZip(target target) {
build(target, tags)
for i := range target.archiveFiles {
target.archiveFiles[i].src = strings.Replace(target.archiveFiles[i].src, "{{binary}}", target.binaryName, 1)
target.archiveFiles[i].dst = strings.Replace(target.archiveFiles[i].dst, "{{binary}}", target.binaryName, 1)
target.archiveFiles[i].src = strings.Replace(target.archiveFiles[i].src, "{{binary}}", target.BinaryName(), 1)
target.archiveFiles[i].dst = strings.Replace(target.archiveFiles[i].dst, "{{binary}}", target.BinaryName(), 1)
target.archiveFiles[i].dst = name + "/" + target.archiveFiles[i].dst
}
zipFile(filename, target.archiveFiles)
log.Println(filename)
fmt.Println(filename)
}
func buildDeb(target target) {
os.RemoveAll("deb")
// "goarch" here is set to whatever the Debian packages expect. We correct
// "it to what we actually know how to build and keep the Debian variant
// "name in "debarch".
// it to what we actually know how to build and keep the Debian variant
// name in "debarch".
debarch := goarch
switch goarch {
case "i386":
@@ -488,63 +529,116 @@ func buildDeb(target target) {
build(target, []string{"noupgrade"})
for i := range target.debianFiles {
target.debianFiles[i].src = strings.Replace(target.debianFiles[i].src, "{{binary}}", target.binaryName, 1)
target.debianFiles[i].dst = strings.Replace(target.debianFiles[i].dst, "{{binary}}", target.binaryName, 1)
for i := range target.installationFiles {
target.installationFiles[i].src = strings.Replace(target.installationFiles[i].src, "{{binary}}", target.BinaryName(), 1)
target.installationFiles[i].dst = strings.Replace(target.installationFiles[i].dst, "{{binary}}", target.BinaryName(), 1)
}
for _, af := range target.debianFiles {
for _, af := range target.installationFiles {
if err := copyFile(af.src, af.dst, af.perm); err != nil {
log.Fatal(err)
}
}
os.MkdirAll("deb/DEBIAN", 0755)
data := map[string]string{
"name": target.name,
"arch": debarch,
"version": version[1:],
"date": time.Now().Format(time.RFC1123),
maintainer := "Syncthing Release Management <release@syncthing.net>"
debver := version
if strings.HasPrefix(debver, "v") {
debver = debver[1:]
// Debian interprets dashes as separator between main version and
// Debian package version, and thus thinks 0.14.26-rc.1 is better
// than just 0.14.26. This rectifies that.
debver = strings.Replace(debver, "-", "~", -1)
}
debTemplateFiles := append(listFiles("debtpl/common"), listFiles("debtpl/"+target.name)...)
for _, file := range debTemplateFiles {
tpl, err := template.New(filepath.Base(file)).ParseFiles(file)
if err != nil {
log.Fatal(err)
}
outFile := filepath.Join("deb/DEBIAN", filepath.Base(file))
out, err := os.Create(outFile)
if err != nil {
log.Fatal(err)
}
if err := tpl.Execute(out, data); err != nil {
log.Fatal(err)
}
if err := out.Close(); err != nil {
log.Fatal(err)
}
info, _ := os.Lstat(file)
os.Chmod(outFile, info.Mode())
args := []string{
"-t", "deb",
"-s", "dir",
"-C", "deb",
"-n", target.debname,
"-v", debver,
"-a", debarch,
"-m", maintainer,
"--vendor", maintainer,
"--description", target.description,
"--url", "https://syncthing.net/",
"--license", "MPL-2",
}
for _, dep := range target.debdeps {
args = append(args, "-d", dep)
}
if target.debpost != "" {
args = append(args, "--after-upgrade", target.debpost)
}
runPrint("fpm", args...)
}
func buildSnap(target target) {
os.RemoveAll("snap")
tmpl, err := template.ParseFiles("snapcraft.yaml.template")
if err != nil {
log.Fatal(err)
}
f, err := os.Create("snapcraft.yaml")
defer f.Close()
if err != nil {
log.Fatal(err)
}
snaparch := goarch
if snaparch == "armhf" {
goarch = "arm"
}
snapver := version
if strings.HasPrefix(snapver, "v") {
snapver = snapver[1:]
}
snapgrade := "devel"
if matched, _ := regexp.MatchString(`^\d+\.\d+\.\d+(-rc.\d+)?$`, snapver); matched {
snapgrade = "stable"
}
err = tmpl.Execute(f, map[string]string{
"Version": snapver,
"Architecture": snaparch,
"Grade": snapgrade,
})
if err != nil {
log.Fatal(err)
}
runPrint("snapcraft", "clean")
build(target, []string{"noupgrade"})
runPrint("snapcraft")
}
// copyFile copies a file from src to dst, ensuring the containing directory
// exists. The permission bits are copied as well. If dst already exists and
// the contents are identical to src the modification time is not updated.
func copyFile(src, dst string, perm os.FileMode) error {
dstDir := filepath.Dir(dst)
os.MkdirAll(dstDir, 0755) // ignore error
srcFd, err := os.Open(src)
in, err := ioutil.ReadFile(src)
if err != nil {
return err
}
defer srcFd.Close()
dstFd, err := os.OpenFile(dst, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, perm)
out, err := ioutil.ReadFile(dst)
if err != nil {
// The destination probably doesn't exist, we should create
// it.
goto copy
}
if bytes.Equal(in, out) {
// The permission bits may have changed without the contents
// changing so we always mirror them.
os.Chmod(dst, perm)
return nil
}
copy:
os.MkdirAll(filepath.Dir(dst), 0777)
if err := ioutil.WriteFile(dst, in, perm); err != nil {
return err
}
defer dstFd.Close()
_, err = io.Copy(dstFd, srcFd)
return err
return nil
}
func listFiles(dir string) []string {
@@ -583,14 +677,15 @@ func shouldRebuildAssets(target, srcdir string) bool {
// so we should rebuild it.
currentBuild := info.ModTime()
assetsAreNewer := false
stop := errors.New("no need to iterate further")
filepath.Walk(srcdir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if assetsAreNewer {
return nil
if info.ModTime().After(currentBuild) {
assetsAreNewer = true
return stop
}
assetsAreNewer = info.ModTime().After(currentBuild)
return nil
})
@@ -598,7 +693,7 @@ func shouldRebuildAssets(target, srcdir string) bool {
}
func proto() {
runPrint("go", "generate", "./lib/...")
runPrint("go", "generate", "github.com/syncthing/syncthing/lib/...")
}
func translate() {
@@ -639,7 +734,9 @@ func ldflags() string {
func rmr(paths ...string) {
for _, path := range paths {
log.Println("rm -r", path)
if debug {
log.Println("rm -r", path)
}
os.RemoveAll(path)
}
}
@@ -723,8 +820,9 @@ func getBranchSuffix() string {
}
branch = parts[len(parts)-1]
if branch == "master" {
// master builds are the default.
switch branch {
case "master", "release":
// these are not special
return ""
}
@@ -755,6 +853,10 @@ func buildStamp() int64 {
}
func buildUser() string {
if v := os.Getenv("BUILD_USER"); v != "" {
return v
}
u, err := user.Current()
if err != nil {
return "unknown-user"
@@ -763,6 +865,10 @@ func buildUser() string {
}
func buildHost() string {
if v := os.Getenv("BUILD_HOST"); v != "" {
return v
}
h, err := os.Hostname()
if err != nil {
return "unknown-host"
@@ -840,7 +946,10 @@ func tarGz(out string, files []archiveFile) {
log.Fatal(err)
}
gw := gzip.NewWriter(fd)
gw, err := gzip.NewWriterLevel(fd, gzip.BestCompression)
if err != nil {
log.Fatal(err)
}
tw := tar.NewWriter(gw)
for _, f := range files {
@@ -893,6 +1002,21 @@ func zipFile(out string, files []archiveFile) {
zw := zip.NewWriter(fd)
var fw *flate.Writer
// Register the deflator.
zw.RegisterCompressor(zip.Deflate, func(out io.Writer) (io.WriteCloser, error) {
var err error
if fw == nil {
// Creating a flate compressor for every file is
// expensive, create one and reuse it.
fw, err = flate.NewWriter(out, flate.BestCompression)
} else {
fw.Reset(out)
}
return fw, err
})
for _, f := range files {
sf, err := os.Open(f.src)
if err != nil {
@@ -949,48 +1073,6 @@ func zipFile(out string, files []archiveFile) {
}
}
func vet(dirs ...string) {
params := []string{"tool", "vet", "-all"}
params = append(params, dirs...)
bs, err := runError("go", params...)
if len(bs) > 0 {
log.Printf("%s", bs)
}
if err != nil {
if exitStatus(err) == 3 {
// Exit code 3, the "vet" tool is not installed
return
}
// A genuine error exit from the vet tool.
log.Fatal(err)
}
}
func lint(pkg string) {
bs, err := runError("golint", pkg)
if err != nil {
log.Println(`- No golint, not linting. Try "go get -u github.com/golang/lint/golint".`)
return
}
analCommentPolicy := regexp.MustCompile(`exported (function|method|const|type|var) [^\s]+ should have comment`)
for _, line := range strings.Split(string(bs), "\n") {
if line == "" {
continue
}
if analCommentPolicy.MatchString(line) {
continue
}
if strings.Contains(line, ".pb.go:") {
continue
}
log.Println(line)
}
}
func macosCodesign(file string) {
if pass := os.Getenv("CODESIGN_KEYCHAIN_PASS"); pass != "" {
bs, err := runError("security", "unlock-keychain", "-p", pass)
@@ -1010,50 +1092,129 @@ func macosCodesign(file string) {
}
}
func exitStatus(err error) int {
if err, ok := err.(*exec.ExitError); ok {
if ws, ok := err.ProcessState.Sys().(syscall.WaitStatus); ok {
return ws.ExitStatus()
}
}
return -1
func metalint() {
lazyRebuildAssets()
runPrint("go", "test", "-run", "Metalint", "./meta")
}
func isGometalinterInstalled() bool {
if _, err := runError("gometalinter", "--disable-all"); err != nil {
log.Println("gometalinter is not installed")
return false
}
return true
func metalintShort() {
lazyRebuildAssets()
runPrint("go", "test", "-short", "-run", "Metalint", "./meta")
}
func gometalinter(linter string, dirs []string, excludes ...string) bool {
params := []string{"--disable-all"}
params = append(params, fmt.Sprintf("--deadline=%ds", 60))
params = append(params, "--enable="+linter)
func temporaryBuildDir() (string, error) {
// The base of our temp dir is "syncthing-xxxxxxxx" where the x:es
// are eight bytes from the sha256 of our working directory. We do
// this because we want a name in the global temp dir that doesn't
// conflict with someone else building syncthing on the same
// machine, yet is persistent between runs from the same source
// directory.
wd, err := os.Getwd()
if err != nil {
return "", err
}
hash := sha256.Sum256([]byte(wd))
base := fmt.Sprintf("syncthing-%x", hash[:4])
for _, exclude := range excludes {
params = append(params, "--exclude="+exclude)
// The temp dir is taken from $STTMPDIR if set, otherwise the system
// default (potentially infrluenced by $TMPDIR on unixes).
var tmpDir string
if t := os.Getenv("STTMPDIR"); t != "" {
tmpDir = t
} else {
tmpDir = os.TempDir()
}
return filepath.Join(tmpDir, base), nil
}
func buildGOPATH(gopath string) error {
pkg := filepath.Join(gopath, "src/github.com/syncthing/syncthing")
dirs := []string{"cmd", "lib", "meta", "script", "test", "vendor"}
if debug {
t0 := time.Now()
log.Println("build temporary GOPATH in", gopath)
defer func() {
log.Println("... in", time.Since(t0))
}()
}
// Walk the sources and copy the files into the temporary GOPATH.
// Remember which files are supposed to be present so we can clean
// out everything else in the next step. The copyFile() step will
// only actually copy the file if it doesn't exist or the contents
// differ.
exists := map[string]struct{}{}
for _, dir := range dirs {
params = append(params, dir)
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
dst := filepath.Join(pkg, path)
exists[dst] = struct{}{}
if err := copyFile(path, dst, info.Mode()); err != nil {
return err
}
return nil
})
if err != nil {
return err
}
}
bs, _ := runError("gometalinter", params...)
// Walk the temporary GOPATH and remove any files that we wouldn't
// have copied there in the previous step.
nerr := 0
for _, line := range strings.Split(string(bs), "\n") {
if line == "" {
continue
filepath.Walk(pkg, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if strings.Contains(line, ".pb.go:") {
continue
if info.IsDir() {
return nil
}
log.Println(line)
nerr++
}
if _, ok := exists[path]; !ok {
os.Remove(path)
}
return nil
})
return nerr == 0
return nil
}
func gopath() string {
if gopath := os.Getenv("GOPATH"); gopath != "" {
// The env var is set, use that.
return gopath
}
// Ask Go what it thinks.
bs, err := runError("go", "env", "GOPATH")
if err != nil {
return ""
}
// We got something. Check if we are in fact available in that location.
gopath := string(bs)
if _, err := os.Stat(filepath.Join(gopath, "src/github.com/syncthing/syncthing/build.go")); err == nil {
// That seems to be the gopath.
return gopath
}
// The gopath is not valid.
return ""
}
func (t target) BinaryName() string {
if goos == "windows" {
return t.binaryName + ".exe"
}
return t.binaryName
}

View File

@@ -61,9 +61,9 @@ case "${1:-default}" in
prerelease)
go run script/authors.go
build transifex
git add -A gui/default/assets/ lib/auto/
pushd man ; ./refresh.sh ; popd
git add -A man
git add -A gui man
git commit -m 'gui, man: Update docs & translations'
;;
noupgrade)
@@ -93,35 +93,6 @@ case "${1:-default}" in
done
;;
test-cov)
ulimit -t 600 &>/dev/null || true
ulimit -d 512000 &>/dev/null || true
ulimit -m 512000 &>/dev/null || true
echo "mode: set" > coverage.out
fail=0
# For every package in the repo
for dir in $(go list ./lib/... ./cmd/...) ; do
# run the tests
GOPATH="$(pwd)/Godeps/_workspace:$GOPATH" go test -race -coverprofile=profile.out $dir
if [ -f profile.out ] ; then
# and if there was test output, append it to coverage.out
grep -v "mode: " profile.out >> coverage.out
rm profile.out
fi
done
gocov convert coverage.out | gocov-xml > coverage.xml
# This is usually run from within Jenkins. If it is, we need to
# tweak the paths in coverage.xml so cobertura finds the
# source.
if [[ "${WORKSPACE:-default}" != "default" ]] ; then
sed "s#$WORKSPACE##g" < coverage.xml > coverage.xml.new && mv coverage.xml.new coverage.xml
fi
;;
test-xunit)
ulimit -t 600 &>/dev/null || true
ulimit -d 512000 &>/dev/null || true
@@ -131,55 +102,6 @@ case "${1:-default}" in
go2xunit -output tests.xml -fail < tests.out
;;
docker-all)
img=${DOCKERIMG:-syncthing/build:latest}
docker run --rm -h syncthing-builder -u $(id -u) -t \
-v $(pwd):/go/src/github.com/syncthing/syncthing \
-w /go/src/github.com/syncthing/syncthing \
-e "STTRACE=$STTRACE" \
"$img" \
sh -c './build.sh clean \
&& ./build.sh test-cov \
&& ./build.sh bench \
&& ./build.sh all'
;;
docker-test)
img=${DOCKERIMG:-syncthing/build:latest}
docker run --rm -h syncthing-builder -u $(id -u) -t \
-v $(pwd):/go/src/github.com/syncthing/syncthing \
-w /go/src/github.com/syncthing/syncthing \
-e "STTRACE=$STTRACE" \
"$img" \
sh -euxc './build.sh clean \
&& go run build.go -race \
&& export GOPATH=$(pwd)/Godeps/_workspace:$GOPATH \
&& cd test \
&& go test -tags integration -v -timeout 90m -short \
&& git clean -fxd .'
;;
docker-lint)
img=${DOCKERIMG:-syncthing/build:latest}
docker run --rm -h syncthing-builder -u $(id -u) -t \
-v $(pwd):/go/src/github.com/syncthing/syncthing \
-w /go/src/github.com/syncthing/syncthing \
-e "STTRACE=$STTRACE" \
"$img" \
sh -euxc 'go run build.go lint'
;;
docker-vet)
img=${DOCKERIMG:-syncthing/build:latest}
docker run --rm -h syncthing-builder -u $(id -u) -t \
-v $(pwd):/go/src/github.com/syncthing/syncthing \
-w /go/src/github.com/syncthing/syncthing \
-e "STTRACE=$STTRACE" \
"$img" \
sh -euxc 'go run build.go vet'
;;
*)
echo "Unknown build command $1"
;;

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
// This doesn't build on Windows due to the Rusage stuff.

View File

@@ -1,4 +1,4 @@
Copyright (C) 2014 Jakob Borg
Copyright (C) 2014 Audrius Butkevičius
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in

115
cmd/stcli/client.go Normal file
View File

@@ -0,0 +1,115 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"bytes"
"crypto/tls"
"net/http"
"strings"
"github.com/AudriusButkevicius/cli"
)
type APIClient struct {
httpClient http.Client
endpoint string
apikey string
username string
password string
id string
csrf string
}
var instance *APIClient
func getClient(c *cli.Context) *APIClient {
if instance != nil {
return instance
}
endpoint := c.GlobalString("endpoint")
if !strings.HasPrefix(endpoint, "http") {
endpoint = "http://" + endpoint
}
httpClient := http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
InsecureSkipVerify: c.GlobalBool("insecure"),
},
},
}
client := APIClient{
httpClient: httpClient,
endpoint: endpoint,
apikey: c.GlobalString("apikey"),
username: c.GlobalString("username"),
password: c.GlobalString("password"),
}
if client.apikey == "" {
request, err := http.NewRequest("GET", client.endpoint, nil)
die(err)
response := client.handleRequest(request)
client.id = response.Header.Get("X-Syncthing-ID")
if client.id == "" {
die("Failed to get device ID")
}
for _, item := range response.Cookies() {
if item.Name == "CSRF-Token-"+client.id[:5] {
client.csrf = item.Value
goto csrffound
}
}
die("Failed to get CSRF token")
csrffound:
}
instance = &client
return &client
}
func (client *APIClient) handleRequest(request *http.Request) *http.Response {
if client.apikey != "" {
request.Header.Set("X-API-Key", client.apikey)
}
if client.username != "" || client.password != "" {
request.SetBasicAuth(client.username, client.password)
}
if client.csrf != "" {
request.Header.Set("X-CSRF-Token-"+client.id[:5], client.csrf)
}
response, err := client.httpClient.Do(request)
die(err)
if response.StatusCode == 404 {
die("Invalid endpoint or API call")
} else if response.StatusCode == 401 {
die("Invalid username or password")
} else if response.StatusCode == 403 {
if client.apikey == "" {
die("Invalid CSRF token")
}
die("Invalid API key")
} else if response.StatusCode != 200 {
body := strings.TrimSpace(string(responseToBArray(response)))
if body != "" {
die(body)
}
die("Unknown HTTP status returned: " + response.Status)
}
return response
}
func httpGet(c *cli.Context, url string) *http.Response {
client := getClient(c)
request, err := http.NewRequest("GET", client.endpoint+"/rest/"+url, nil)
die(err)
return client.handleRequest(request)
}
func httpPost(c *cli.Context, url string, body string) *http.Response {
client := getClient(c)
request, err := http.NewRequest("POST", client.endpoint+"/rest/"+url, bytes.NewBufferString(body))
die(err)
return client.handleRequest(request)
}

188
cmd/stcli/cmd_devices.go Normal file
View File

@@ -0,0 +1,188 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"fmt"
"strings"
"github.com/AudriusButkevicius/cli"
"github.com/syncthing/syncthing/lib/config"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "devices",
HideHelp: true,
Usage: "Device command group",
Subcommands: []cli.Command{
{
Name: "list",
Usage: "List registered devices",
Requires: &cli.Requires{},
Action: devicesList,
},
{
Name: "add",
Usage: "Add a new device",
Requires: &cli.Requires{"device id", "device name?"},
Action: devicesAdd,
},
{
Name: "remove",
Usage: "Remove an existing device",
Requires: &cli.Requires{"device id"},
Action: devicesRemove,
},
{
Name: "get",
Usage: "Get a property of a device",
Requires: &cli.Requires{"device id", "property"},
Action: devicesGet,
},
{
Name: "set",
Usage: "Set a property of a device",
Requires: &cli.Requires{"device id", "property", "value..."},
Action: devicesSet,
},
},
})
}
func devicesList(c *cli.Context) {
cfg := getConfig(c)
first := true
writer := newTableWriter()
for _, device := range cfg.Devices {
if !first {
fmt.Fprintln(writer)
}
fmt.Fprintln(writer, "ID:\t", device.DeviceID, "\t")
fmt.Fprintln(writer, "Name:\t", device.Name, "\t(name)")
fmt.Fprintln(writer, "Address:\t", strings.Join(device.Addresses, " "), "\t(address)")
fmt.Fprintln(writer, "Compression:\t", device.Compression, "\t(compression)")
fmt.Fprintln(writer, "Certificate name:\t", device.CertName, "\t(certname)")
fmt.Fprintln(writer, "Introducer:\t", device.Introducer, "\t(introducer)")
first = false
}
writer.Flush()
}
func devicesAdd(c *cli.Context) {
nid := c.Args()[0]
id := parseDeviceID(nid)
newDevice := config.DeviceConfiguration{
DeviceID: id,
Name: nid,
Addresses: []string{"dynamic"},
}
if len(c.Args()) > 1 {
newDevice.Name = c.Args()[1]
}
if len(c.Args()) > 2 {
addresses := c.Args()[2:]
for _, item := range addresses {
if item == "dynamic" {
continue
}
validAddress(item)
}
newDevice.Addresses = addresses
}
cfg := getConfig(c)
for _, device := range cfg.Devices {
if device.DeviceID == id {
die("Device " + nid + " already exists")
}
}
cfg.Devices = append(cfg.Devices, newDevice)
setConfig(c, cfg)
}
func devicesRemove(c *cli.Context) {
nid := c.Args()[0]
id := parseDeviceID(nid)
if nid == getMyID(c) {
die("Cannot remove yourself")
}
cfg := getConfig(c)
for i, device := range cfg.Devices {
if device.DeviceID == id {
last := len(cfg.Devices) - 1
cfg.Devices[i] = cfg.Devices[last]
cfg.Devices = cfg.Devices[:last]
setConfig(c, cfg)
return
}
}
die("Device " + nid + " not found")
}
func devicesGet(c *cli.Context) {
nid := c.Args()[0]
id := parseDeviceID(nid)
arg := c.Args()[1]
cfg := getConfig(c)
for _, device := range cfg.Devices {
if device.DeviceID != id {
continue
}
switch strings.ToLower(arg) {
case "name":
fmt.Println(device.Name)
case "address":
fmt.Println(strings.Join(device.Addresses, "\n"))
case "compression":
fmt.Println(device.Compression.String())
case "certname":
fmt.Println(device.CertName)
case "introducer":
fmt.Println(device.Introducer)
default:
die("Invalid property: " + arg + "\nAvailable properties: name, address, compression, certname, introducer")
}
return
}
die("Device " + nid + " not found")
}
func devicesSet(c *cli.Context) {
nid := c.Args()[0]
id := parseDeviceID(nid)
arg := c.Args()[1]
config := getConfig(c)
for i, device := range config.Devices {
if device.DeviceID != id {
continue
}
switch strings.ToLower(arg) {
case "name":
config.Devices[i].Name = strings.Join(c.Args()[2:], " ")
case "address":
for _, item := range c.Args()[2:] {
if item == "dynamic" {
continue
}
validAddress(item)
}
config.Devices[i].Addresses = c.Args()[2:]
case "compression":
err := config.Devices[i].Compression.UnmarshalText([]byte(c.Args()[2]))
die(err)
case "certname":
config.Devices[i].CertName = strings.Join(c.Args()[2:], " ")
case "introducer":
config.Devices[i].Introducer = parseBool(c.Args()[2])
default:
die("Invalid property: " + arg + "\nAvailable properties: name, address, compression, certname, introducer")
}
setConfig(c, config)
return
}
die("Device " + nid + " not found")
}

67
cmd/stcli/cmd_errors.go Normal file
View File

@@ -0,0 +1,67 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"encoding/json"
"fmt"
"strings"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "errors",
HideHelp: true,
Usage: "Error command group",
Subcommands: []cli.Command{
{
Name: "show",
Usage: "Show pending errors",
Requires: &cli.Requires{},
Action: errorsShow,
},
{
Name: "push",
Usage: "Push an error to active clients",
Requires: &cli.Requires{"error message..."},
Action: errorsPush,
},
{
Name: "clear",
Usage: "Clear pending errors",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/error/clear"),
},
},
})
}
func errorsShow(c *cli.Context) {
response := httpGet(c, "system/error")
var data map[string][]map[string]interface{}
json.Unmarshal(responseToBArray(response), &data)
writer := newTableWriter()
for _, item := range data["errors"] {
time := item["time"].(string)[:19]
time = strings.Replace(time, "T", " ", 1)
err := item["error"].(string)
err = strings.TrimSpace(err)
fmt.Fprintln(writer, time+":\t"+err)
}
writer.Flush()
}
func errorsPush(c *cli.Context) {
err := strings.Join(c.Args(), " ")
response := httpPost(c, "system/error", strings.TrimSpace(err))
if response.StatusCode != 200 {
err = fmt.Sprint("Failed to push error\nStatus code: ", response.StatusCode)
body := string(responseToBArray(response))
if body != "" {
err += "\nBody: " + body
}
die(err)
}
}

361
cmd/stcli/cmd_folders.go Normal file
View File

@@ -0,0 +1,361 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"fmt"
"path/filepath"
"strings"
"github.com/AudriusButkevicius/cli"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/fs"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "folders",
HideHelp: true,
Usage: "Folder command group",
Subcommands: []cli.Command{
{
Name: "list",
Usage: "List available folders",
Requires: &cli.Requires{},
Action: foldersList,
},
{
Name: "add",
Usage: "Add a new folder",
Requires: &cli.Requires{"folder id", "directory"},
Action: foldersAdd,
},
{
Name: "remove",
Usage: "Remove an existing folder",
Requires: &cli.Requires{"folder id"},
Action: foldersRemove,
},
{
Name: "override",
Usage: "Override changes from other nodes for a master folder",
Requires: &cli.Requires{"folder id"},
Action: foldersOverride,
},
{
Name: "get",
Usage: "Get a property of a folder",
Requires: &cli.Requires{"folder id", "property"},
Action: foldersGet,
},
{
Name: "set",
Usage: "Set a property of a folder",
Requires: &cli.Requires{"folder id", "property", "value..."},
Action: foldersSet,
},
{
Name: "unset",
Usage: "Unset a property of a folder",
Requires: &cli.Requires{"folder id", "property"},
Action: foldersUnset,
},
{
Name: "devices",
Usage: "Folder devices command group",
HideHelp: true,
Subcommands: []cli.Command{
{
Name: "list",
Usage: "List of devices which the folder is shared with",
Requires: &cli.Requires{"folder id"},
Action: foldersDevicesList,
},
{
Name: "add",
Usage: "Share a folder with a device",
Requires: &cli.Requires{"folder id", "device id"},
Action: foldersDevicesAdd,
},
{
Name: "remove",
Usage: "Unshare a folder with a device",
Requires: &cli.Requires{"folder id", "device id"},
Action: foldersDevicesRemove,
},
{
Name: "clear",
Usage: "Unshare a folder with all devices",
Requires: &cli.Requires{"folder id"},
Action: foldersDevicesClear,
},
},
},
},
})
}
func foldersList(c *cli.Context) {
cfg := getConfig(c)
first := true
writer := newTableWriter()
for _, folder := range cfg.Folders {
if !first {
fmt.Fprintln(writer)
}
fs := folder.Filesystem()
fmt.Fprintln(writer, "ID:\t", folder.ID, "\t")
fmt.Fprintln(writer, "Path:\t", fs.URI(), "\t(directory)")
fmt.Fprintln(writer, "Path type:\t", fs.Type(), "\t(directory-type)")
fmt.Fprintln(writer, "Folder type:\t", folder.Type, "\t(type)")
fmt.Fprintln(writer, "Ignore permissions:\t", folder.IgnorePerms, "\t(permissions)")
fmt.Fprintln(writer, "Rescan interval in seconds:\t", folder.RescanIntervalS, "\t(rescan)")
if folder.Versioning.Type != "" {
fmt.Fprintln(writer, "Versioning:\t", folder.Versioning.Type, "\t(versioning)")
for key, value := range folder.Versioning.Params {
fmt.Fprintf(writer, "Versioning %s:\t %s \t(versioning-%s)\n", key, value, key)
}
}
first = false
}
writer.Flush()
}
func foldersAdd(c *cli.Context) {
cfg := getConfig(c)
abs, err := filepath.Abs(c.Args()[1])
die(err)
folder := config.FolderConfiguration{
ID: c.Args()[0],
Path: filepath.Clean(abs),
FilesystemType: fs.FilesystemTypeBasic,
}
cfg.Folders = append(cfg.Folders, folder)
setConfig(c, cfg)
}
func foldersRemove(c *cli.Context) {
cfg := getConfig(c)
rid := c.Args()[0]
for i, folder := range cfg.Folders {
if folder.ID == rid {
last := len(cfg.Folders) - 1
cfg.Folders[i] = cfg.Folders[last]
cfg.Folders = cfg.Folders[:last]
setConfig(c, cfg)
return
}
}
die("Folder " + rid + " not found")
}
func foldersOverride(c *cli.Context) {
cfg := getConfig(c)
rid := c.Args()[0]
for _, folder := range cfg.Folders {
if folder.ID == rid && folder.Type == config.FolderTypeSendOnly {
response := httpPost(c, "db/override", "")
if response.StatusCode != 200 {
err := fmt.Sprint("Failed to override changes\nStatus code: ", response.StatusCode)
body := string(responseToBArray(response))
if body != "" {
err += "\nBody: " + body
}
die(err)
}
return
}
}
die("Folder " + rid + " not found or folder not master")
}
func foldersGet(c *cli.Context) {
cfg := getConfig(c)
rid := c.Args()[0]
arg := strings.ToLower(c.Args()[1])
for _, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
if strings.HasPrefix(arg, "versioning-") {
arg = arg[11:]
value, ok := folder.Versioning.Params[arg]
if ok {
fmt.Println(value)
return
}
die("Versioning property " + c.Args()[1][11:] + " not found")
}
switch arg {
case "directory":
fmt.Println(folder.Filesystem().URI())
case "directory-type":
fmt.Println(folder.Filesystem().Type())
case "type":
fmt.Println(folder.Type)
case "permissions":
fmt.Println(folder.IgnorePerms)
case "rescan":
fmt.Println(folder.RescanIntervalS)
case "versioning":
if folder.Versioning.Type != "" {
fmt.Println(folder.Versioning.Type)
}
default:
die("Invalid property: " + c.Args()[1] + "\nAvailable properties: directory, directory-type, type, permissions, versioning, versioning-<key>")
}
return
}
die("Folder " + rid + " not found")
}
func foldersSet(c *cli.Context) {
rid := c.Args()[0]
arg := strings.ToLower(c.Args()[1])
val := strings.Join(c.Args()[2:], " ")
cfg := getConfig(c)
for i, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
if strings.HasPrefix(arg, "versioning-") {
cfg.Folders[i].Versioning.Params[arg[11:]] = val
setConfig(c, cfg)
return
}
switch arg {
case "directory":
cfg.Folders[i].Path = val
case "directory-type":
var fsType fs.FilesystemType
fsType.UnmarshalText([]byte(val))
cfg.Folders[i].FilesystemType = fsType
case "type":
var t config.FolderType
if err := t.UnmarshalText([]byte(val)); err != nil {
die("Invalid folder type: " + err.Error())
}
cfg.Folders[i].Type = t
case "permissions":
cfg.Folders[i].IgnorePerms = parseBool(val)
case "rescan":
cfg.Folders[i].RescanIntervalS = parseInt(val)
case "versioning":
cfg.Folders[i].Versioning.Type = val
default:
die("Invalid property: " + c.Args()[1] + "\nAvailable properties: directory, master, permissions, versioning, versioning-<key>")
}
setConfig(c, cfg)
return
}
die("Folder " + rid + " not found")
}
func foldersUnset(c *cli.Context) {
rid := c.Args()[0]
arg := strings.ToLower(c.Args()[1])
cfg := getConfig(c)
for i, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
if strings.HasPrefix(arg, "versioning-") {
arg = arg[11:]
if _, ok := folder.Versioning.Params[arg]; ok {
delete(cfg.Folders[i].Versioning.Params, arg)
setConfig(c, cfg)
return
}
die("Versioning property " + c.Args()[1][11:] + " not found")
}
switch arg {
case "versioning":
cfg.Folders[i].Versioning.Type = ""
cfg.Folders[i].Versioning.Params = make(map[string]string)
default:
die("Invalid property: " + c.Args()[1] + "\nAvailable properties: versioning, versioning-<key>")
}
setConfig(c, cfg)
return
}
die("Folder " + rid + " not found")
}
func foldersDevicesList(c *cli.Context) {
rid := c.Args()[0]
cfg := getConfig(c)
for _, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
for _, device := range folder.Devices {
fmt.Println(device.DeviceID)
}
return
}
die("Folder " + rid + " not found")
}
func foldersDevicesAdd(c *cli.Context) {
rid := c.Args()[0]
nid := parseDeviceID(c.Args()[1])
cfg := getConfig(c)
for i, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
for _, device := range folder.Devices {
if device.DeviceID == nid {
die("Device " + c.Args()[1] + " is already part of this folder")
}
}
for _, device := range cfg.Devices {
if device.DeviceID == nid {
cfg.Folders[i].Devices = append(folder.Devices, config.FolderDeviceConfiguration{
DeviceID: device.DeviceID,
})
setConfig(c, cfg)
return
}
}
die("Device " + c.Args()[1] + " not found in device list")
}
die("Folder " + rid + " not found")
}
func foldersDevicesRemove(c *cli.Context) {
rid := c.Args()[0]
nid := parseDeviceID(c.Args()[1])
cfg := getConfig(c)
for ri, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
for ni, device := range folder.Devices {
if device.DeviceID == nid {
last := len(folder.Devices) - 1
cfg.Folders[ri].Devices[ni] = folder.Devices[last]
cfg.Folders[ri].Devices = cfg.Folders[ri].Devices[:last]
setConfig(c, cfg)
return
}
}
die("Device " + c.Args()[1] + " not found")
}
die("Folder " + rid + " not found")
}
func foldersDevicesClear(c *cli.Context) {
rid := c.Args()[0]
cfg := getConfig(c)
for i, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
cfg.Folders[i].Devices = []config.FolderDeviceConfiguration{}
setConfig(c, cfg)
return
}
die("Folder " + rid + " not found")
}

78
cmd/stcli/cmd_general.go Normal file
View File

@@ -0,0 +1,78 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"encoding/json"
"fmt"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, []cli.Command{
{
Name: "id",
Usage: "Get ID of the Syncthing client",
Requires: &cli.Requires{},
Action: generalID,
},
{
Name: "status",
Usage: "Configuration status, whether or not a restart is required for changes to take effect",
Requires: &cli.Requires{},
Action: generalStatus,
},
{
Name: "restart",
Usage: "Restart syncthing",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/restart"),
},
{
Name: "shutdown",
Usage: "Shutdown syncthing",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/shutdown"),
},
{
Name: "reset",
Usage: "Reset syncthing deleting all folders and devices",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/reset"),
},
{
Name: "upgrade",
Usage: "Upgrade syncthing (if a newer version is available)",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/upgrade"),
},
{
Name: "version",
Usage: "Syncthing client version",
Requires: &cli.Requires{},
Action: generalVersion,
},
}...)
}
func generalID(c *cli.Context) {
fmt.Println(getMyID(c))
}
func generalStatus(c *cli.Context) {
response := httpGet(c, "system/config/insync")
var status struct{ ConfigInSync bool }
json.Unmarshal(responseToBArray(response), &status)
if !status.ConfigInSync {
die("Config out of sync")
}
fmt.Println("Config in sync")
}
func generalVersion(c *cli.Context) {
response := httpGet(c, "system/version")
version := make(map[string]interface{})
json.Unmarshal(responseToBArray(response), &version)
prettyPrintJSON(version)
}

127
cmd/stcli/cmd_gui.go Normal file
View File

@@ -0,0 +1,127 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"fmt"
"strings"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "gui",
HideHelp: true,
Usage: "GUI command group",
Subcommands: []cli.Command{
{
Name: "dump",
Usage: "Show all GUI configuration settings",
Requires: &cli.Requires{},
Action: guiDump,
},
{
Name: "get",
Usage: "Get a GUI configuration setting",
Requires: &cli.Requires{"setting"},
Action: guiGet,
},
{
Name: "set",
Usage: "Set a GUI configuration setting",
Requires: &cli.Requires{"setting", "value"},
Action: guiSet,
},
{
Name: "unset",
Usage: "Unset a GUI configuration setting",
Requires: &cli.Requires{"setting"},
Action: guiUnset,
},
},
})
}
func guiDump(c *cli.Context) {
cfg := getConfig(c).GUI
writer := newTableWriter()
fmt.Fprintln(writer, "Enabled:\t", cfg.Enabled, "\t(enabled)")
fmt.Fprintln(writer, "Use HTTPS:\t", cfg.UseTLS(), "\t(tls)")
fmt.Fprintln(writer, "Listen Addresses:\t", cfg.Address(), "\t(address)")
if cfg.User != "" {
fmt.Fprintln(writer, "Authentication User:\t", cfg.User, "\t(username)")
fmt.Fprintln(writer, "Authentication Password:\t", cfg.Password, "\t(password)")
}
if cfg.APIKey != "" {
fmt.Fprintln(writer, "API Key:\t", cfg.APIKey, "\t(apikey)")
}
writer.Flush()
}
func guiGet(c *cli.Context) {
cfg := getConfig(c).GUI
arg := c.Args()[0]
switch strings.ToLower(arg) {
case "enabled":
fmt.Println(cfg.Enabled)
case "tls":
fmt.Println(cfg.UseTLS())
case "address":
fmt.Println(cfg.Address())
case "user":
if cfg.User != "" {
fmt.Println(cfg.User)
}
case "password":
if cfg.User != "" {
fmt.Println(cfg.Password)
}
case "apikey":
if cfg.APIKey != "" {
fmt.Println(cfg.APIKey)
}
default:
die("Invalid setting: " + arg + "\nAvailable settings: enabled, tls, address, user, password, apikey")
}
}
func guiSet(c *cli.Context) {
cfg := getConfig(c)
arg := c.Args()[0]
val := c.Args()[1]
switch strings.ToLower(arg) {
case "enabled":
cfg.GUI.Enabled = parseBool(val)
case "tls":
cfg.GUI.RawUseTLS = parseBool(val)
case "address":
validAddress(val)
cfg.GUI.RawAddress = val
case "user":
cfg.GUI.User = val
case "password":
cfg.GUI.Password = val
case "apikey":
cfg.GUI.APIKey = val
default:
die("Invalid setting: " + arg + "\nAvailable settings: enabled, tls, address, user, password, apikey")
}
setConfig(c, cfg)
}
func guiUnset(c *cli.Context) {
cfg := getConfig(c)
arg := c.Args()[0]
switch strings.ToLower(arg) {
case "user":
cfg.GUI.User = ""
case "password":
cfg.GUI.Password = ""
case "apikey":
cfg.GUI.APIKey = ""
default:
die("Invalid setting: " + arg + "\nAvailable settings: user, password, apikey")
}
setConfig(c, cfg)
}

173
cmd/stcli/cmd_options.go Normal file
View File

@@ -0,0 +1,173 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"fmt"
"strings"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "options",
HideHelp: true,
Usage: "Options command group",
Subcommands: []cli.Command{
{
Name: "dump",
Usage: "Show all Syncthing option settings",
Requires: &cli.Requires{},
Action: optionsDump,
},
{
Name: "get",
Usage: "Get a Syncthing option setting",
Requires: &cli.Requires{"setting"},
Action: optionsGet,
},
{
Name: "set",
Usage: "Set a Syncthing option setting",
Requires: &cli.Requires{"setting", "value..."},
Action: optionsSet,
},
},
})
}
func optionsDump(c *cli.Context) {
cfg := getConfig(c).Options
writer := newTableWriter()
fmt.Fprintln(writer, "Sync protocol listen addresses:\t", strings.Join(cfg.ListenAddresses, " "), "\t(addresses)")
fmt.Fprintln(writer, "Global discovery enabled:\t", cfg.GlobalAnnEnabled, "\t(globalannenabled)")
fmt.Fprintln(writer, "Global discovery servers:\t", strings.Join(cfg.GlobalAnnServers, " "), "\t(globalannserver)")
fmt.Fprintln(writer, "Local discovery enabled:\t", cfg.LocalAnnEnabled, "\t(localannenabled)")
fmt.Fprintln(writer, "Local discovery port:\t", cfg.LocalAnnPort, "\t(localannport)")
fmt.Fprintln(writer, "Outgoing rate limit in KiB/s:\t", cfg.MaxSendKbps, "\t(maxsend)")
fmt.Fprintln(writer, "Incoming rate limit in KiB/s:\t", cfg.MaxRecvKbps, "\t(maxrecv)")
fmt.Fprintln(writer, "Reconnect interval in seconds:\t", cfg.ReconnectIntervalS, "\t(reconnect)")
fmt.Fprintln(writer, "Start browser:\t", cfg.StartBrowser, "\t(browser)")
fmt.Fprintln(writer, "Enable UPnP:\t", cfg.NATEnabled, "\t(nat)")
fmt.Fprintln(writer, "UPnP Lease in minutes:\t", cfg.NATLeaseM, "\t(natlease)")
fmt.Fprintln(writer, "UPnP Renewal period in minutes:\t", cfg.NATRenewalM, "\t(natrenew)")
fmt.Fprintln(writer, "Restart on Wake Up:\t", cfg.RestartOnWakeup, "\t(wake)")
reporting := "unrecognized value"
switch cfg.URAccepted {
case -1:
reporting = "false"
case 0:
reporting = "undecided/false"
case 1:
reporting = "true"
}
fmt.Fprintln(writer, "Anonymous usage reporting:\t", reporting, "\t(reporting)")
writer.Flush()
}
func optionsGet(c *cli.Context) {
cfg := getConfig(c).Options
arg := c.Args()[0]
switch strings.ToLower(arg) {
case "address":
fmt.Println(strings.Join(cfg.ListenAddresses, "\n"))
case "globalannenabled":
fmt.Println(cfg.GlobalAnnEnabled)
case "globalannservers":
fmt.Println(strings.Join(cfg.GlobalAnnServers, "\n"))
case "localannenabled":
fmt.Println(cfg.LocalAnnEnabled)
case "localannport":
fmt.Println(cfg.LocalAnnPort)
case "maxsend":
fmt.Println(cfg.MaxSendKbps)
case "maxrecv":
fmt.Println(cfg.MaxRecvKbps)
case "reconnect":
fmt.Println(cfg.ReconnectIntervalS)
case "browser":
fmt.Println(cfg.StartBrowser)
case "nat":
fmt.Println(cfg.NATEnabled)
case "natlease":
fmt.Println(cfg.NATLeaseM)
case "natrenew":
fmt.Println(cfg.NATRenewalM)
case "reporting":
switch cfg.URAccepted {
case -1:
fmt.Println("false")
case 0:
fmt.Println("undecided/false")
case 1:
fmt.Println("true")
default:
fmt.Println("unknown")
}
case "wake":
fmt.Println(cfg.RestartOnWakeup)
default:
die("Invalid setting: " + arg + "\nAvailable settings: address, globalannenabled, globalannserver, localannenabled, localannport, maxsend, maxrecv, reconnect, browser, upnp, upnplease, upnprenew, reporting, wake")
}
}
func optionsSet(c *cli.Context) {
config := getConfig(c)
arg := c.Args()[0]
val := c.Args()[1]
switch strings.ToLower(arg) {
case "address":
for _, item := range c.Args().Tail() {
validAddress(item)
}
config.Options.ListenAddresses = c.Args().Tail()
case "globalannenabled":
config.Options.GlobalAnnEnabled = parseBool(val)
case "globalannserver":
for _, item := range c.Args().Tail() {
validAddress(item)
}
config.Options.GlobalAnnServers = c.Args().Tail()
case "localannenabled":
config.Options.LocalAnnEnabled = parseBool(val)
case "localannport":
config.Options.LocalAnnPort = parsePort(val)
case "maxsend":
config.Options.MaxSendKbps = parseUint(val)
case "maxrecv":
config.Options.MaxRecvKbps = parseUint(val)
case "reconnect":
config.Options.ReconnectIntervalS = parseUint(val)
case "browser":
config.Options.StartBrowser = parseBool(val)
case "nat":
config.Options.NATEnabled = parseBool(val)
case "natlease":
config.Options.NATLeaseM = parseUint(val)
case "natrenew":
config.Options.NATRenewalM = parseUint(val)
case "reporting":
switch strings.ToLower(val) {
case "u", "undecided", "unset":
config.Options.URAccepted = 0
default:
boolvalue := parseBool(val)
if boolvalue {
config.Options.URAccepted = 1
} else {
config.Options.URAccepted = -1
}
}
case "wake":
config.Options.RestartOnWakeup = parseBool(val)
default:
die("Invalid setting: " + arg + "\nAvailable settings: address, globalannenabled, globalannserver, localannenabled, localannport, maxsend, maxrecv, reconnect, browser, upnp, upnplease, upnprenew, reporting, wake")
}
setConfig(c, config)
}

72
cmd/stcli/cmd_report.go Normal file
View File

@@ -0,0 +1,72 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"encoding/json"
"fmt"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "report",
HideHelp: true,
Usage: "Reporting command group",
Subcommands: []cli.Command{
{
Name: "system",
Usage: "Report system state",
Requires: &cli.Requires{},
Action: reportSystem,
},
{
Name: "connections",
Usage: "Report about connections to other devices",
Requires: &cli.Requires{},
Action: reportConnections,
},
{
Name: "usage",
Usage: "Usage report",
Requires: &cli.Requires{},
Action: reportUsage,
},
},
})
}
func reportSystem(c *cli.Context) {
response := httpGet(c, "system/status")
data := make(map[string]interface{})
json.Unmarshal(responseToBArray(response), &data)
prettyPrintJSON(data)
}
func reportConnections(c *cli.Context) {
response := httpGet(c, "system/connections")
data := make(map[string]map[string]interface{})
json.Unmarshal(responseToBArray(response), &data)
var overall map[string]interface{}
for key, value := range data {
if key == "total" {
overall = value
continue
}
value["Device ID"] = key
prettyPrintJSON(value)
fmt.Println()
}
if overall != nil {
fmt.Println("=== Overall statistics ===")
prettyPrintJSON(overall)
}
}
func reportUsage(c *cli.Context) {
response := httpGet(c, "svc/report")
report := make(map[string]interface{})
json.Unmarshal(responseToBArray(response), &report)
prettyPrintJSON(report)
}

31
cmd/stcli/labels.go Normal file
View File

@@ -0,0 +1,31 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
var jsonAttributeLabels = map[string]string{
"folderMaxMiB": "Largest folder size in MiB",
"folderMaxFiles": "Largest folder file count",
"longVersion": "Long version",
"totMiB": "Total size in MiB",
"totFiles": "Total files",
"uniqueID": "Unique ID",
"numFolders": "Folder count",
"numDevices": "Device count",
"memoryUsageMiB": "Memory usage in MiB",
"memorySize": "Total memory in MiB",
"sha256Perf": "SHA256 Benchmark",
"At": "Last contacted",
"Completion": "Percent complete",
"InBytesTotal": "Total bytes received",
"OutBytesTotal": "Total bytes sent",
"ClientVersion": "Client version",
"alloc": "Memory allocated in bytes",
"sys": "Memory using in bytes",
"cpuPercent": "CPU load in percent",
"extAnnounceOK": "External announcments working",
"goroutines": "Number of Go routines",
"myID": "Client ID",
"tilde": "Tilde expands to",
"arch": "Architecture",
"os": "OS",
}

63
cmd/stcli/main.go Normal file
View File

@@ -0,0 +1,63 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"sort"
"github.com/AudriusButkevicius/cli"
)
type ByAlphabet []cli.Command
func (a ByAlphabet) Len() int { return len(a) }
func (a ByAlphabet) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (a ByAlphabet) Less(i, j int) bool { return a[i].Name < a[j].Name }
var cliCommands []cli.Command
func main() {
app := cli.NewApp()
app.Name = "syncthing-cli"
app.Author = "Audrius Butkevičius"
app.Email = "audrius.butkevicius@gmail.com"
app.Usage = "Syncthing command line interface"
app.Version = "0.1"
app.HideHelp = true
app.Flags = []cli.Flag{
cli.StringFlag{
Name: "endpoint, e",
Value: "http://127.0.0.1:8384",
Usage: "End point to connect to",
EnvVar: "STENDPOINT",
},
cli.StringFlag{
Name: "apikey, k",
Value: "",
Usage: "API Key",
EnvVar: "STAPIKEY",
},
cli.StringFlag{
Name: "username, u",
Value: "",
Usage: "Username",
EnvVar: "STUSERNAME",
},
cli.StringFlag{
Name: "password, p",
Value: "",
Usage: "Password",
EnvVar: "STPASSWORD",
},
cli.BoolFlag{
Name: "insecure, i",
Usage: "Do not verify SSL certificate",
EnvVar: "STINSECURE",
},
}
sort.Sort(ByAlphabet(cliCommands))
app.Commands = cliCommands
app.RunAndExitOnError()
}

165
cmd/stcli/utils.go Normal file
View File

@@ -0,0 +1,165 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"os"
"regexp"
"sort"
"strconv"
"strings"
"text/tabwriter"
"unicode"
"github.com/AudriusButkevicius/cli"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/protocol"
)
func responseToBArray(response *http.Response) []byte {
defer response.Body.Close()
bytes, err := ioutil.ReadAll(response.Body)
if err != nil {
die(err)
}
return bytes
}
func die(vals ...interface{}) {
if len(vals) > 1 || vals[0] != nil {
os.Stderr.WriteString(fmt.Sprintln(vals...))
os.Exit(1)
}
}
func wrappedHTTPPost(url string) func(c *cli.Context) {
return func(c *cli.Context) {
httpPost(c, url, "")
}
}
func prettyPrintJSON(json map[string]interface{}) {
writer := newTableWriter()
remap := make(map[string]interface{})
for k, v := range json {
key, ok := jsonAttributeLabels[k]
if !ok {
key = firstUpper(k)
}
remap[key] = v
}
jsonKeys := make([]string, 0, len(remap))
for key := range remap {
jsonKeys = append(jsonKeys, key)
}
sort.Strings(jsonKeys)
for _, k := range jsonKeys {
var value string
rvalue := remap[k]
switch rvalue.(type) {
case int, int16, int32, int64, uint, uint16, uint32, uint64, float32, float64:
value = fmt.Sprintf("%.0f", rvalue)
default:
value = fmt.Sprint(rvalue)
}
if value == "" {
continue
}
fmt.Fprintln(writer, k+":\t"+value)
}
writer.Flush()
}
func firstUpper(str string) string {
for i, v := range str {
return string(unicode.ToUpper(v)) + str[i+1:]
}
return ""
}
func newTableWriter() *tabwriter.Writer {
writer := new(tabwriter.Writer)
writer.Init(os.Stdout, 0, 8, 0, '\t', 0)
return writer
}
func getMyID(c *cli.Context) string {
response := httpGet(c, "system/status")
data := make(map[string]interface{})
json.Unmarshal(responseToBArray(response), &data)
return data["myID"].(string)
}
func getConfig(c *cli.Context) config.Configuration {
response := httpGet(c, "system/config")
config := config.Configuration{}
json.Unmarshal(responseToBArray(response), &config)
return config
}
func setConfig(c *cli.Context, cfg config.Configuration) {
body, err := json.Marshal(cfg)
die(err)
response := httpPost(c, "system/config", string(body))
if response.StatusCode != 200 {
die("Unexpected status code", response.StatusCode)
}
}
func parseBool(input string) bool {
val, err := strconv.ParseBool(input)
if err != nil {
die(input + " is not a valid value for a boolean")
}
return val
}
func parseInt(input string) int {
val, err := strconv.ParseInt(input, 0, 64)
if err != nil {
die(input + " is not a valid value for an integer")
}
return int(val)
}
func parseUint(input string) int {
val, err := strconv.ParseUint(input, 0, 64)
if err != nil {
die(input + " is not a valid value for an unsigned integer")
}
return int(val)
}
func parsePort(input string) int {
port := parseUint(input)
if port < 1 || port > 65535 {
die(input + " is not a valid port\nExpected value between 1 and 65535")
}
return port
}
func validAddress(input string) {
tokens := strings.Split(input, ":")
if len(tokens) != 2 {
die(input + " is not a valid value for an address\nExpected format <ip or hostname>:<port>")
}
matched, err := regexp.MatchString("^[a-zA-Z0-9]+([-a-zA-Z0-9.]+[-a-zA-Z0-9]+)?$", tokens[0])
die(err)
if !matched {
die(input + " is not a valid value for an address\nExpected format <ip or hostname>:<port>")
}
parsePort(tokens[1])
}
func parseDeviceID(input string) protocol.DeviceID {
device, err := protocol.DeviceIDFromString(input)
if err != nil {
die(input + " is not a valid device id")
}
return device
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -15,8 +15,6 @@ import (
"log"
"os"
"path/filepath"
"github.com/syncthing/syncthing/lib/symlinks"
)
func main() {
@@ -90,10 +88,10 @@ func startWalker(dir string, res chan<- fileInfo, abort <-chan struct{}) chan er
}
rn, _ := filepath.Rel(dir, path)
if rn == "." || rn == ".stfolder" {
if rn == "." {
return nil
}
if rn == ".stversions" {
if rn == ".stversions" || rn == ".stfolder" {
return filepath.SkipDir
}
@@ -104,7 +102,7 @@ func startWalker(dir string, res chan<- fileInfo, abort <-chan struct{}) chan er
mode: os.ModeSymlink,
}
tgt, _, err := symlinks.Read(path)
tgt, err := os.Readlink(path)
if err != nil {
return err
}

View File

@@ -2,12 +2,11 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"crypto/rand"
"encoding/binary"
"flag"
@@ -45,7 +44,7 @@ func main() {
flag.Parse()
if fake {
log.Println("My ID:", protocol.DeviceIDFromBytes(myID))
log.Println("My ID:", myID)
}
runbeacon(beacon.NewMulticast(mc), fake)
@@ -75,17 +74,17 @@ func recv(bc beacon.Interface) {
var ann discover.Announce
ann.Unmarshal(data[4:])
if bytes.Equal(ann.ID, myID) {
if ann.ID == myID {
// This is one of our own fake packets, don't print it.
continue
}
// Print announcement details for the first packet from a given
// device ID and source address, or if -all was given.
key := string(ann.ID) + src.String()
key := ann.ID.String() + src.String()
if all || !seen[key] {
log.Printf("Announcement from %v\n", src)
log.Printf(" %v at %s\n", protocol.DeviceIDFromBytes(ann.ID), strings.Join(ann.Addresses, ", "))
log.Printf(" %v at %s\n", ann.ID, strings.Join(ann.Addresses, ", "))
seen[key] = true
}
}
@@ -106,9 +105,9 @@ func send(bc beacon.Interface) {
}
// returns a random but recognizable device ID
func randomDeviceID() []byte {
var id [32]byte
func randomDeviceID() protocol.DeviceID {
var id protocol.DeviceID
copy(id[:], randomPrefix)
rand.Read(id[len(randomPrefix):])
return id[:]
return id
}

View File

@@ -1,14 +1,8 @@
stdiscosrv
==========
[![Latest Build](http://img.shields.io/jenkins/s/http/build.syncthing.net/stdiscosrv.svg?style=flat-square)](http://build.syncthing.net/job/stdiscosrv/lastBuild/)
This is the global discovery server for the `syncthing` project.
To get it, run `go get github.com/syncthing/stdiscosrv` or download the
[latest build](http://build.syncthing.net/job/stdiscosrv/lastSuccessfulBuild/artifact/)
from the build server.
Usage
-----
@@ -38,3 +32,7 @@ In all cases, the appropriate tables and indexes will be created at first
startup. If it doesn't exit with an error, you're fine.
See `stdiscosrv -help` for other options.
##### Third-party attribution
[cznic/lldb](https://github.com/cznic/lldb), Copyright (C) 2014 The lldb Authors.

View File

@@ -14,6 +14,7 @@ import (
"time"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/tlsutil"
"github.com/thejerf/suture"
)
@@ -86,7 +87,11 @@ func main() {
if !useHTTP {
cert, err = tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
log.Fatalln("Failed to load X509 key pair:", err)
log.Println("Failed to load keypair. Generating one, this might take a while...")
cert, err = tlsutil.NewCertificate(certFile, keyFile, "stdiscosrv", 3072)
if err != nil {
log.Fatalln("Failed to generate X509 key pair:", err)
}
}
devID := protocol.NewDeviceID(cert.Certificate[0])

View File

@@ -27,8 +27,9 @@ func postgresSetup(db *sql.DB) error {
return err
}
var tmp string
row := db.QueryRow(`SELECT 'DevicesDeviceIDIndex'::regclass`)
if err = row.Scan(nil); err != nil {
if err = row.Scan(&tmp); err != nil {
_, err = db.Exec(`CREATE INDEX DevicesDeviceIDIndex ON Devices (DeviceID)`)
}
if err != nil {
@@ -36,7 +37,7 @@ func postgresSetup(db *sql.DB) error {
}
row = db.QueryRow(`SELECT 'DevicesSeenIndex'::regclass`)
if err = row.Scan(nil); err != nil {
if err = row.Scan(&tmp); err != nil {
_, err = db.Exec(`CREATE INDEX DevicesSeenIndex ON Devices (Seen)`)
}
if err != nil {
@@ -46,14 +47,14 @@ func postgresSetup(db *sql.DB) error {
_, err = db.Exec(`CREATE TABLE IF NOT EXISTS Addresses (
DeviceID CHAR(63) NOT NULL,
Seen TIMESTAMP NOT NULL,
Address VARCHAR(256) NOT NULL
Address VARCHAR(2048) NOT NULL
)`)
if err != nil {
return err
}
row = db.QueryRow(`SELECT 'AddressesDeviceIDSeenIndex'::regclass`)
if err = row.Scan(nil); err != nil {
if err = row.Scan(&tmp); err != nil {
_, err = db.Exec(`CREATE INDEX AddressesDeviceIDSeenIndex ON Addresses (DeviceID, Seen)`)
}
if err != nil {
@@ -61,7 +62,7 @@ func postgresSetup(db *sql.DB) error {
}
row = db.QueryRow(`SELECT 'AddressesDeviceIDAddressIndex'::regclass`)
if err = row.Scan(nil); err != nil {
if err = row.Scan(&tmp); err != nil {
_, err = db.Exec(`CREATE INDEX AddressesDeviceIDAddressIndex ON Addresses (DeviceID, Address)`)
}
if err != nil {

View File

@@ -19,9 +19,9 @@ import (
"time"
"github.com/golang/groupcache/lru"
"github.com/juju/ratelimit"
"github.com/syncthing/syncthing/lib/protocol"
"golang.org/x/net/context"
"golang.org/x/time/rate"
)
type querysrv struct {
@@ -62,6 +62,10 @@ func (i requestID) String() string {
return fmt.Sprintf("%016x", int64(i))
}
type contextKey int
const idKey contextKey = iota
func negCacheFor(lastSeen time.Time) int {
since := time.Since(lastSeen).Seconds()
if since >= maxDeviceAge {
@@ -132,7 +136,7 @@ var topCtx = context.Background()
func (s *querysrv) handler(w http.ResponseWriter, req *http.Request) {
reqID := requestID(rand.Int63())
ctx := context.WithValue(topCtx, "id", reqID)
ctx := context.WithValue(topCtx, idKey, reqID)
if debug {
log.Println(reqID, req.Method, req.URL)
@@ -186,7 +190,7 @@ func (s *querysrv) handler(w http.ResponseWriter, req *http.Request) {
}
func (s *querysrv) handleGET(ctx context.Context, w http.ResponseWriter, req *http.Request) {
reqID := ctx.Value("id").(requestID)
reqID := ctx.Value(idKey).(requestID)
deviceID, err := protocol.DeviceIDFromString(req.URL.Query().Get("device"))
if err != nil {
@@ -238,7 +242,7 @@ func (s *querysrv) handleGET(ctx context.Context, w http.ResponseWriter, req *ht
}
func (s *querysrv) handlePOST(ctx context.Context, remoteIP net.IP, w http.ResponseWriter, req *http.Request) {
reqID := ctx.Value("id").(requestID)
reqID := ctx.Value(idKey).(requestID)
rawCert := certificateBytes(req)
if rawCert == nil {
@@ -299,7 +303,7 @@ func (s *querysrv) Stop() {
}
func (s *querysrv) handleAnnounce(ctx context.Context, remote net.IP, deviceID protocol.DeviceID, addresses []string) (userErr, internalErr error) {
reqID := ctx.Value("id").(requestID)
reqID := ctx.Value(idKey).(requestID)
tx, err := s.db.Begin()
if err != nil {
@@ -369,21 +373,21 @@ func (s *querysrv) limit(remote net.IP) bool {
bkt, ok := s.limiter.Get(key)
if ok {
bkt := bkt.(*ratelimit.Bucket)
if bkt.TakeAvailable(1) != 1 {
bkt := bkt.(*rate.Limiter)
if !bkt.Allow() {
// Rate limit exceeded; ignore packet
return true
}
} else {
// One packet per ten seconds average rate, burst ten packets
s.limiter.Add(key, ratelimit.NewBucket(10*time.Second/time.Duration(limitAvg), int64(limitBurst)))
// limitAvg is in packets per ten seconds.
s.limiter.Add(key, rate.NewLimiter(rate.Limit(limitAvg)/10, limitBurst))
}
return false
}
func (s *querysrv) updateDevice(ctx context.Context, tx *sql.Tx, device protocol.DeviceID) error {
reqID := ctx.Value("id").(requestID)
reqID := ctx.Value(idKey).(requestID)
t0 := time.Now()
res, err := tx.Stmt(s.prep["updateDevice"]).Exec(device.String())
if err != nil {
@@ -451,7 +455,7 @@ func (s *querysrv) getDeviceSeen(device protocol.DeviceID) (time.Time, error) {
if err := row.Scan(&seen); err != nil {
return time.Time{}, err
}
return seen, nil
return seen.In(time.UTC), nil
}
func handlePing(w http.ResponseWriter, r *http.Request) {

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,11 +2,12 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"context"
"flag"
"log"
"os"
@@ -40,7 +41,8 @@ func main() {
log.Println("Lstat:")
log.Printf(" Size: %d bytes", fi.Size())
log.Printf(" Mode: 0%o", fi.Mode())
log.Printf(" Time: %v (%d)", fi.ModTime(), fi.ModTime().Unix())
log.Printf(" Time: %v", fi.ModTime())
log.Printf(" %d.%09d", fi.ModTime().Unix(), fi.ModTime().Nanosecond())
log.Println()
if !fi.Mode().IsDir() && !fi.Mode().IsRegular() {
@@ -52,7 +54,8 @@ func main() {
log.Println("Stat:")
log.Printf(" Size: %d bytes", fi.Size())
log.Printf(" Mode: 0%o", fi.Mode())
log.Printf(" Time: %v (%d)", fi.ModTime(), fi.ModTime().Unix())
log.Printf(" Time: %v", fi.ModTime())
log.Printf(" %d.%09d", fi.ModTime().Unix(), fi.ModTime().Nanosecond())
log.Println()
}
@@ -68,7 +71,7 @@ func main() {
if *standardBlocks || blockSize < protocol.BlockSize {
blockSize = protocol.BlockSize
}
bs, err := scanner.Blocks(fd, blockSize, fi.Size(), nil)
bs, err := scanner.Blocks(context.TODO(), fd, blockSize, fi.Size(), nil, true)
if err != nil {
log.Fatal(err)
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -66,7 +66,7 @@ func checkServers(deviceID protocol.DeviceID, servers ...string) {
}()
}
for _ = range servers {
for range servers {
res := <-resc
u, _ := url.Parse(res.server)

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -66,7 +66,7 @@ func generateFiles(dir string, files, maxexp int, srcname string) error {
}
func generateOneFile(fd io.ReadSeeker, p1 string, s int64) error {
src := io.LimitReader(&inifiteReader{fd}, int64(s))
src := io.LimitReader(&inifiteReader{fd}, s)
dst, err := os.Create(p1)
if err != nil {
return err
@@ -85,12 +85,7 @@ func generateOneFile(fd io.ReadSeeker, p1 string, s int64) error {
_ = os.Chmod(p1, os.FileMode(rand.Intn(0777)|0400))
t := time.Now().Add(-time.Duration(rand.Intn(30*86400)) * time.Second)
err = os.Chtimes(p1, t, t)
if err != nil {
return err
}
return nil
return os.Chtimes(p1, t, t)
}
func randomName() string {

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -10,6 +10,7 @@ import (
"encoding/binary"
"fmt"
"log"
"time"
"github.com/syncthing/syncthing/lib/db"
"github.com/syncthing/syncthing/lib/protocol"
@@ -53,7 +54,13 @@ func dump(ldb *db.Instance) {
fmt.Printf("[fstat] K:%x V:%x\n", it.Key(), it.Value())
case db.KeyTypeVirtualMtime:
fmt.Printf("[mtime] K:%x V:%x\n", it.Key(), it.Value())
folder := binary.BigEndian.Uint32(key[1:])
name := nulString(key[1+4:])
val := it.Value()
var real, virt time.Time
real.UnmarshalBinary(val[:len(val)/2])
virt.UnmarshalBinary(val[len(val)/2:])
fmt.Printf("[mtime] F:%d N:%q R:%v V:%v\n", folder, name, real, virt)
case db.KeyTypeFolderIdx:
key := binary.BigEndian.Uint32(it.Key()[1:])

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -12,7 +12,7 @@ import (
"path/filepath"
"runtime"
"github.com/syncthing/syncthing/lib/osutil"
"github.com/syncthing/syncthing/lib/fs"
)
func nulString(bs []byte) string {
@@ -33,7 +33,7 @@ func defaultConfigDir() string {
return filepath.Join(os.Getenv("AppData"), "Syncthing")
case "darwin":
dir, err := osutil.ExpandTilde("~/Library/Application Support/Syncthing")
dir, err := fs.ExpandTilde("~/Library/Application Support/Syncthing")
if err != nil {
log.Fatal(err)
}
@@ -43,7 +43,7 @@ func defaultConfigDir() string {
if xdgCfg := os.Getenv("XDG_CONFIG_HOME"); xdgCfg != "" {
return filepath.Join(xdgCfg, "syncthing")
}
dir, err := osutil.ExpandTilde("~/.config/syncthing")
dir, err := fs.ExpandTilde("~/.config/syncthing")
if err != nil {
log.Fatal(err)
}

View File

@@ -1,15 +1,24 @@
# relaypoolsrv
[![Latest Build](http://img.shields.io/jenkins/s/http/build.syncthing.net/relaypoolsrv.svg?style=flat-square)](http://build.syncthing.net/job/relaypoolsrv/lastBuild/)
This is the relay pool server for the `syncthing` project, which allows
community hosted [relaysrv](https://github.com/syncthing/relaysrv)'s to join
the public pool.
This is the relay pool server for the `syncthing` project, which allows community hosted [relaysrv](https://github.com/syncthing/relaysrv)'s to join the public pool.
Servers that join the pool are then advertised to users of `syncthing` as
potential connection points for those who are unable to connect directly due
to NAT or firewall issues.
Servers that join the pool are then advertised to users of `syncthing` as potential connection points for those who are unable to connect directly due to NAT or firewall issues.
There is very little reason why you'd want to run this yourself, as `relaypoolsrv` is just used for announcement and lookup of public relay servers. If you are looking to setup a private or a public relay, please check the documentation for [relaysrv](https://github.com/syncthing/relaysrv), which also explains how to join the default public pool.
If you still want to run it, you can run `go get github.com/syncthing/relaypoolsrv` download it or download the
[latest build](http://build.syncthing.net/job/relaypoolsrv/lastSuccessfulBuild/artifact/)
from the build server.
There is very little reason why you'd want to run this yourself, as
`relaypoolsrv` is just used for announcement and lookup of public relay
servers. If you are looking to setup a private or a public relay, please
check the documentation for
[relaysrv](https://github.com/syncthing/relaysrv), which also explains how
to join the default public pool.
See `relaypoolsrv -help` for configuration options.
##### Third-party attributions
[oschwald/geoip2-golang](https://github.com/oschwald/geoip2-golang), [oschwald/maxminddb-golang](https://github.com/oschwald/maxminddb-golang), Copyright (C) 2015 [Gregory J. Oschwald](mailto:oschwald@gmail.com).
[lib/pq](https://github.com/lib/pq)</a>, Copyright (C) 2011-2013 'pq' Contributors Portions Copyright (C) 2011 Blake Mizerany.

View File

@@ -2,15 +2,15 @@
<html lang="en" ng-app="syncthing" ng-controller="relayDataController">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="">
<meta name="author" content="">
<meta charset="utf-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=edge"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<meta name="description" content=""/>
<meta name="author" content=""/>
<title>Relay stats</title>
<link href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" rel="stylesheet">
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css">
<link href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" rel="stylesheet"/>
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css"/>
<style>
#map {
@@ -36,9 +36,9 @@
<body class="ng-cloak">
<div class="container">
<h1>Relay Pool Data</h2>
<h1>Relay Pool Data</h1>
<div ng-if="relays === undefined" class="text-center">
<img src="//cdnjs.cloudflare.com/ajax/libs/galleriffic/2.0.1/css/loader.gif"/>
<img src="//cdnjs.cloudflare.com/ajax/libs/galleriffic/2.0.1/css/loader.gif" alt=""/>
<p>Please wait while we gather data</p>
</div>
<div>
@@ -48,7 +48,7 @@
</p>
</div>
<div id="map"></div> <!-- Can't hide the map, otherwise it freaks out -->
<p>The circle size represents how much bytes the relay transfered relative to other relays</p>
<p>The circle size represents how much bytes the relay transferred relative to other relays</p>
</div>
<div>
<table class="table table-striped table-condensed table">
@@ -56,32 +56,32 @@
<tr>
<th rowspan="2">Address</td>
<th rowspan="2">
<a ng-click="sortType = 'status.numActiveSessions || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.numActiveSessions'; sortReverse = !sortReverse">
Sessions
<span ng-show="sortType == 'status.numActiveSessions || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.numActiveSessions || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.numActiveSessions' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.numActiveSessions' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
<th rowspan="2">
<a ng-click="sortType = 'status.numConnections || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.numConnections'; sortReverse = !sortReverse">
Connections
<span ng-show="sortType == 'status.numConnections || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.numConnections || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.numConnections' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.numConnections' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
<th rowspan="2">
<a ng-click="sortType = 'status.bytesProxied || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.bytesProxied'; sortReverse = !sortReverse">
Data relayed
<span ng-show="sortType == 'status.bytesProxied || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.bytesProxied || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.bytesProxied' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.bytesProxied' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
<th colspan="6" class="text-center">Transfer rate in the last period</th>
<th rowspan="2">
<a ng-click="sortType = 'status.uptimeSeconds || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.uptimeSeconds'; sortReverse = !sortReverse">
Uptime hours
<span ng-show="sortType == 'status.uptimeSeconds || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.uptimeSeconds || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.uptimeSeconds' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.uptimeSeconds' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
<th rowspan="2">
@@ -94,51 +94,51 @@
</tr>
<tr>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[0] || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[0]'; sortReverse = !sortReverse">
10s
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[0] || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[0] || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[0]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[0]' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[1] || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[1]'; sortReverse = !sortReverse">
1m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[1] || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[1] || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[1]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[1]' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[2] || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[2]'; sortReverse = !sortReverse">
5m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[2] || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[2] || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[2]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[2]' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[3] || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[3]'; sortReverse = !sortReverse">
15m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[3] || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[3] || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[3]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[3]' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[4] || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[4]'; sortReverse = !sortReverse">
30m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[4] || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[4] || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[4]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[4]' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[5] || -1'; sortReverse = !sortReverse">
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[5]'; sortReverse = !sortReverse">
60m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[5] || -1' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[5] || -1' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[5]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[5]' && sortReverse" class="fa fa-caret-up"></span>
</a>
</th>
</tr>
</thead>
<tbody>
<tr ng-repeat="relay in relays | orderBy:sortType:sortReverse ">
<tr ng-repeat="relay in relays | orderBy:sortType:sortReverse:sortCompare" ng-mouseover="relay.showMarker()" ng-mouseleave="relay.hideMarker()">
<td>{{ relay.address }}</td>
<td ng-if="relay.status === undefined" colspan="11" class="text-center">Looking up...</td>
<td ng-if-start="relay.status !== undefined">{{ relay.status.numActiveSessions }}</td>
@@ -184,10 +184,10 @@
</div>
<script src="//code.jquery.com/jquery-2.1.4.min.js"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/angular.js/1.4.7/angular.min.js"></script>
<script src="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
<script src="//maps.googleapis.com/maps/api/js"></script>
<script type="text/javascript" src="//code.jquery.com/jquery-2.1.4.min.js"></script>
<script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/angular.js/1.5.8/angular.min.js"></script>
<script type="text/javascript" src="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
<script type="text/javascript" src="//maps.googleapis.com/maps/api/js"></script>
</body>
<script>
@@ -235,8 +235,20 @@
$scope.mapBounds = new google.maps.LatLngBounds();
$scope.tooltipTemplate = $('#infoTemplate').html();
$scope.usedLocations = {};
$scope.sortType = 'status.numActiveSessions || -1';
$scope.sortType = 'status.numActiveSessions';
$scope.sortReverse = true;
$scope.sortCompare = function(a, b) {
if (a.value == b.value) {
return 0;
}
if (a.type == "undefined") {
return -1;
}
if (b.type == "undefined") {
return 1;
}
return a.value > b.value ? 1 : -1;
}
$http.get("/endpoint").then(function(response) {
$scope.relays = response.data.relays;
@@ -295,13 +307,16 @@
content: $compile($scope.tooltipTemplate)(scope)[0],
});
relay.marker.addListener('mouseover', function() {
relay.showMarker = function() {
relay.marker.info.open($scope.map, relay.marker);
});
}
relay.marker.addListener('mouseout', function() {
relay.hideMarker = function() {
relay.marker.info.close();
});
}
relay.marker.addListener('mouseover', relay.showMarker);
relay.marker.addListener('mouseout', relay.hideMarker);
$scope.mapBounds.extend(relay.marker.position);
}
@@ -325,7 +340,8 @@
var timeoutRequest = $q.defer();
var resolveStatus = $q.defer();
$http.get("http://" + relay.uri.hostname + (relay.uri.args.statusAddr || ":22070") + "/status", { timeout: timeoutRequest.promise }).then(function (response) {
$http.get("http://" + relay.uri.hostname + ':' + ((relay.uri.args.statusAddr && relay.uri.args.statusAddr.split(':')[1]) || "22070") + "/status", { timeout: timeoutRequest.promise }).then(function (response) {
relay.status = response.data;
resolveStatus.resolve();
angular.forEach($scope.totals, function(value, key) {
@@ -379,12 +395,12 @@
<span ng-if="relay.status.options['global-rate'] != undefined">
<span ng-if="relay.status.options['global-rate'] > 0">Global rate limit: {{ relay.status.options['global-rate'] | bytes }}/s</span>
<span ng-if="relay.status.options['global-rate'] == 0">Global rate limit: unlimited</span>
</br>
<br/>
</span>
<span ng-if="relay.status.options['per-session-rate'] != undefined">
<span ng-if="relay.status.options['per-session-rate'] > 0">Session rate limit: {{ relay.status.options['per-session-rate'] | bytes }}/s</span>
<span ng-if="relay.status.options['per-session-rate'] == 0">Session rate limit: unlimited</span>
</br>
<br/>
</span>
</div>
<div ng-if="!relay.status">

View File

@@ -1,6 +1,6 @@
// Copyright (C) 2015 Audrius Butkevicius and Contributors (see the CONTRIBUTORS file).
//go:generate go run genassets.go gui auto/gui.go
//go:generate go run ../../script/genassets.go gui >auto/gui.go
package main
@@ -23,14 +23,12 @@ import (
"time"
"github.com/golang/groupcache/lru"
"github.com/juju/ratelimit"
"github.com/oschwald/geoip2-golang"
"github.com/syncthing/syncthing/cmd/strelaypoolsrv/auto"
"github.com/syncthing/syncthing/lib/relay/client"
"github.com/syncthing/syncthing/lib/sync"
"github.com/syncthing/syncthing/lib/tlsutil"
"golang.org/x/time/rate"
)
type location struct {
@@ -65,17 +63,18 @@ var (
dir string
evictionTime = time.Hour
debug bool
getLRUSize = 10 << 10
getLimitBurst int64 = 10
getLimitAvg = 1
postLRUSize = 1 << 10
postLimitBurst int64 = 2
postLimitAvg = 1
getLRUSize = 10 << 10
getLimitBurst = 10
getLimitAvg = 1
postLRUSize = 1 << 10
postLimitBurst = 2
postLimitAvg = 1
getLimit time.Duration
postLimit time.Duration
permRelaysFile string
ipHeader string
geoipPath string
proto string
getMut = sync.NewRWMutex()
getLRUCache *lru.Cache
@@ -98,13 +97,14 @@ func main() {
flag.DurationVar(&evictionTime, "eviction", evictionTime, "After how long the relay is evicted")
flag.IntVar(&getLRUSize, "get-limit-cache", getLRUSize, "Get request limiter cache size")
flag.IntVar(&getLimitAvg, "get-limit-avg", 2, "Allowed average get request rate, per 10 s")
flag.Int64Var(&getLimitBurst, "get-limit-burst", getLimitBurst, "Allowed burst get requests")
flag.IntVar(&getLimitBurst, "get-limit-burst", getLimitBurst, "Allowed burst get requests")
flag.IntVar(&postLRUSize, "post-limit-cache", postLRUSize, "Post request limiter cache size")
flag.IntVar(&postLimitAvg, "post-limit-avg", 2, "Allowed average post request rate, per minute")
flag.Int64Var(&postLimitBurst, "post-limit-burst", postLimitBurst, "Allowed burst post requests")
flag.IntVar(&postLimitBurst, "post-limit-burst", postLimitBurst, "Allowed burst post requests")
flag.StringVar(&permRelaysFile, "perm-relays", "", "Path to list of permanent relays")
flag.StringVar(&ipHeader, "ip-header", "", "Name of header which holds clients ip:port. Only meaningful when running behind a reverse proxy.")
flag.StringVar(&geoipPath, "geoip", "GeoLite2-City.mmdb", "Path to GeoLite2-City database")
flag.StringVar(&proto, "protocol", "tcp", "Protocol used for listening. 'tcp' for IPv4 and IPv6, 'tcp4' for IPv4, 'tcp6' for IPv6")
flag.Parse()
@@ -154,12 +154,12 @@ func main() {
},
}
listener, err = tls.Listen("tcp", listen, tlsCfg)
listener, err = tls.Listen(proto, listen, tlsCfg)
} else {
if debug {
log.Println("Starting plain listener on", listen)
}
listener, err = net.Listen("tcp", listen)
listener, err = net.Listen(proto, listen)
}
if err != nil {
@@ -247,13 +247,13 @@ func handleRequest(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*")
switch r.Method {
case "GET":
if limit(r.RemoteAddr, getLRUCache, getMut, getLimit, int64(getLimitBurst)) {
if limit(r.RemoteAddr, getLRUCache, getMut, getLimit, getLimitBurst) {
w.WriteHeader(429)
return
}
handleGetRequest(w, r)
case "POST":
if limit(r.RemoteAddr, postLRUCache, postMut, postLimit, int64(postLimitBurst)) {
if limit(r.RemoteAddr, postLRUCache, postMut, postLimit, postLimitBurst) {
w.WriteHeader(429)
return
}
@@ -324,8 +324,9 @@ func handlePostRequest(w http.ResponseWriter, r *http.Request) {
return
}
ip := net.ParseIP(host)
// The client did not provide an IP address, use the IP address of the client.
if host == "" {
if ip == nil || ip.IsUnspecified() {
uri.Host = net.JoinHostPort(rhost, port)
newRelay.URL = uri.String()
} else if host != rhost {
@@ -443,7 +444,7 @@ func evict(relay relay) func() {
}
}
func limit(addr string, cache *lru.Cache, lock sync.RWMutex, rate time.Duration, burst int64) bool {
func limit(addr string, cache *lru.Cache, lock sync.RWMutex, intv time.Duration, burst int) bool {
host, _, err := net.SplitHostPort(addr)
if err != nil {
return false
@@ -453,14 +454,14 @@ func limit(addr string, cache *lru.Cache, lock sync.RWMutex, rate time.Duration,
bkt, ok := cache.Get(host)
lock.RUnlock()
if ok {
bkt := bkt.(*ratelimit.Bucket)
if bkt.TakeAvailable(1) != 1 {
bkt := bkt.(*rate.Limiter)
if !bkt.Allow() {
// Rate limit
return true
}
} else {
lock.Lock()
cache.Add(host, ratelimit.NewBucket(rate, burst))
cache.Add(host, rate.NewLimiter(rate.Every(intv), burst))
lock.Unlock()
}
return false

View File

@@ -1,28 +1,20 @@
strelaysrv
==========
[![Latest Build](http://img.shields.io/jenkins/s/http/build.syncthing.net/strelaysrv.svg?style=flat-square)](http://build.syncthing.net/job/strelaysrv/lastBuild/)
This is the relay server for the `syncthing` project.
To get it, run `go get github.com/syncthing/strelaysrv` or download the
[latest build](http://build.syncthing.net/job/strelaysrv/lastSuccessfulBuild/artifact/)
from the build server.
:exclamation:Warnings:exclamation: - Read or regret
-----
By default, all relay servers will join the default public relay pool, which means that the relay server will be availble for public use, and **will consume your bandwidth** helping others to connect.
By default, all relay servers will join to the default public relay pool, which means that the relay server will be available for public use, and **will consume your bandwidth** helping others to connect.
If you wish to disable this behaviour, please specify `-pools=""` argument.
If you wish to disable this behaviour, please specify the `-pools=""` argument.
Please note that `strelaysrv` is only usable by `syncthing` **version v0.12 and onwards**.
To run `strelaysrv` you need to have port 22067 available to the internet, which means you might need to allow it through your firewall if you **have a public IP, or setup a port-forwarding** (22067 to 22067) if you are behind a router.
To run `strelaysrv` you need to have port 22067 available to the internet, which means you might need to port forward it and/or allow it through your firewall.
Furthermore, **by default strelaysrv will also expose a /status HTTP endpoint on port 22070**, which is used by the pool servers to peek at metrics of the strelaysrv, such as what are the current transfer rates, how many clients are connected, etc, etc. If you wish this information to be available, similarlly you might want to allow it through your firewall, or port-forward it (22070 to 22070) on your NAT device.
This is **not mandatory** for the strelaysrv to function, and is used only to gather metrics and present them in the overview page of the pool server, displaying stats about the specific relay.
Furthermore, by default `strelaysrv` will also expose a /status HTTP endpoint on port 22070, which is used by the pool servers to read metrics of the `strelaysrv`, such as the current transfer rates, how many clients are connected, etc. If you wish this information to be available you may need to port forward and allow it through your firewall. This is not mandatory for the `strelaysrv` to function, and is used only to gather metrics and present them in the overview page of the pool server.
At the point of writing the endpoint output looks as follows:
@@ -66,7 +58,7 @@ If you wish to disable the /status endpoint, provide `-status-srv=""` as one of
Running for public use
----
Make sure you have a public IP with port 22067 open, or make sure you have port-forwarding (22067 to 22067) if you are behind a router.
Make sure you have a public IP with port 22067 open, or have forwarded port 22067 if you are behind a NAT.
Run the `strelaysrv` with no arguments (or `-debug` if you want more output), and that should be enough for the server to join the public relay pool.
You should see a message saying:
@@ -79,37 +71,37 @@ See `strelaysrv -help` for other options, such as rate limits, timeout intervals
Running for private use
-----
Once you've started the `strelaysrv`, it will generate a key pair and print an URI:
Once you've started the `strelaysrv`, it will generate a key pair and print a URI:
```bash
relay://:22067/?id=EZQOIDM-6DDD4ZI-DJ65NSM-4OQWRAT-EIKSMJO-OZ552BO-WQZEGYY-STS5RQM&pingInterval=1m0s&networkTimeout=2m0s&sessionLimitBps=0&globalLimitBps=0&statusAddr=:22070
```
This URI contains partial address of the relay server, as well as it's options which in the future may be taken into account when choosing the best suitable relay out of multiple available.
This URI contains a partial address of the relay server, as well as its options which in the future may be taken into account when choosing the most suitable relay.
Because `-listen` option was not used, the `strelaysrv` does not know it's external IP, therefore you should replace the host part of the URI with your public IP address on which the `strelaysrv` will be available:
Because the `-listen` option was not used `strelaysrv` does not know its external IP, therefore you should replace the host part of the URI with your public IP address on which the `strelaysrv` will be available:
```bash
relay://123.123.123.123:22067/?id=EZQOIDM-6DDD4ZI-DJ65NSM-4OQWRAT-EIKSMJO-OZ552BO-WQZEGYY-STS5RQM&pingInterval=1m0s&networkTimeout=2m0s&sessionLimitBps=0&globalLimitBps=0&statusAddr=:22070
relay://192.0.2.1:22067/?id=EZQOIDM-6DDD4ZI-DJ65NSM-4OQWRAT-EIKSMJO-OZ552BO-WQZEGYY-STS5RQM&pingInterval=1m0s&networkTimeout=2m0s&sessionLimitBps=0&globalLimitBps=0&statusAddr=:22070
```
If you do not care about certificate pinning (improved security) or do not care about passing verbose settings to the clients, you can shorten the URL to just the host part:
```bash
relay://123.123.123.123:22067
relay://192.0.2.1:22067
```
This URI can then be used in `syncthing` as one of the relay servers.
This URI can then be used in `syncthing` clients as one of the relay servers by adding the URI to the "Sync Protocol Listen Address" field, under Actions and Settings.
See `strelaysrv -help` for other options, such as rate limits, timeout intervals, etc.
Other items available in this repo
----
##### testutil
A test utility which can be used to test connectivity of a relay server.
You need to generate two x509 key pairs (key.pem and cert.pem), one for the client, another one for the server, in separate directories.
A test utility which can be used to test the connectivity of a relay server.
You need to generate two x509 key pairs (key.pem and cert.pem), one for the client and one for the server, in separate directories.
Afterwards, start the client:
```bash
./testutil -relay="relay://uri.of.relay" -keys=certs/client/ -join
./testutil -relay="relay://192.0.2.1:22067" -keys=certs/client/ -join
```
This prints out the client ID:
@@ -120,7 +112,7 @@ This prints out the client ID:
In the other terminal run the following:
```bash
./testutil -relay="relay://uri.of.relay" -keys=certs/server/ -connect=BG2C5ZA-W7XPFDO-LH222Z6-65F3HJX-ADFTGRT-3SBFIGM-KV26O2Q-E5RMRQ2
./testutil -relay="relay://192.0.2.1:22067" -keys=certs/server/ -connect=BG2C5ZA-W7XPFDO-LH222Z6-65F3HJX-ADFTGRT-3SBFIGM-KV26O2Q-E5RMRQ2
```
Which should then give you an interactive prompt, where you can type things in one terminal, and they get relayed to the other terminal.
@@ -137,5 +129,3 @@ Relay related libraries used by this repo
Only used by the testutil.
[Available here](https://github.com/syncthing/syncthing/tree/master/lib/relay/client)

View File

@@ -23,7 +23,7 @@ var (
numConnections int64
)
func listener(addr string, config *tls.Config) {
func listener(proto, addr string, config *tls.Config) {
tcpListener, err := net.Listen("tcp", addr)
if err != nil {
log.Fatalln(err)
@@ -59,6 +59,13 @@ func listener(addr string, config *tls.Config) {
func protocolConnectionHandler(tcpConn net.Conn, config *tls.Config) {
conn := tls.Server(tcpConn, config)
if err := conn.SetDeadline(time.Now().Add(messageTimeout)); err != nil {
if debug {
log.Println("Weird error setting deadline:", err, "on", conn.RemoteAddr())
}
conn.Close()
return
}
err := conn.Handshake()
if err != nil {
if debug {
@@ -81,6 +88,7 @@ func protocolConnectionHandler(tcpConn net.Conn, config *tls.Config) {
conn.Close()
return
}
conn.SetDeadline(time.Time{})
id := syncthingprotocol.NewDeviceID(certs[0].Raw)
@@ -96,7 +104,9 @@ func protocolConnectionHandler(tcpConn net.Conn, config *tls.Config) {
go messageReader(conn, messages, errors)
pingTicker := time.NewTicker(pingInterval)
defer pingTicker.Stop()
timeoutTicker := time.NewTimer(networkTimeout)
defer timeoutTicker.Stop()
joined := false
for {
@@ -167,10 +177,16 @@ func protocolConnectionHandler(tcpConn net.Conn, config *tls.Config) {
continue
}
peerOutbox <- serverInvitation
select {
case peerOutbox <- serverInvitation:
if debug {
log.Println("Sent invitation from", id, "to", requestedPeer)
}
case <-time.After(time.Second):
if debug {
log.Println("Could not send invitation from", id, "to", requestedPeer, "as peer disconnected")
}
if debug {
log.Println("Sent invitation from", id, "to", requestedPeer)
}
conn.Close()
@@ -198,7 +214,6 @@ func protocolConnectionHandler(tcpConn net.Conn, config *tls.Config) {
if debug {
log.Printf("Closing connection %s: %s", id, err)
}
close(outbox)
// Potentially closing a second time.
conn.Close()
@@ -254,10 +269,6 @@ func protocolConnectionHandler(tcpConn net.Conn, config *tls.Config) {
conn.Close()
case msg := <-outbox:
if msg == nil {
conn.Close()
return
}
if debug {
log.Printf("Sending message %T to %s", msg, id)
}
@@ -276,6 +287,7 @@ func sessionConnectionHandler(conn net.Conn) {
if debug {
log.Println("Weird error setting deadline:", err, "on", conn.RemoteAddr())
}
conn.Close()
return
}

View File

@@ -8,6 +8,7 @@ import (
"fmt"
"log"
"net"
"net/http"
"net/url"
"os"
"os/signal"
@@ -19,10 +20,15 @@ import (
"syscall"
"time"
"github.com/juju/ratelimit"
"github.com/syncthing/syncthing/lib/osutil"
"github.com/syncthing/syncthing/lib/relay/protocol"
"github.com/syncthing/syncthing/lib/tlsutil"
"golang.org/x/time/rate"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/nat"
_ "github.com/syncthing/syncthing/lib/pmp"
_ "github.com/syncthing/syncthing/lib/upnp"
syncthingprotocol "github.com/syncthing/syncthing/lib/protocol"
)
@@ -58,24 +64,32 @@ var (
limitCheckTimer *time.Timer
sessionLimitBps int
globalLimitBps int
overLimit int32
descriptorLimit int64
sessionLimiter *ratelimit.Bucket
globalLimiter *ratelimit.Bucket
sessionLimitBps int
globalLimitBps int
overLimit int32
descriptorLimit int64
sessionLimiter *rate.Limiter
globalLimiter *rate.Limiter
networkBufferSize int
statusAddr string
poolAddrs string
pools []string
providedBy string
defaultPoolAddrs = "https://relays.syncthing.net/endpoint"
natEnabled bool
natLease int
natRenewal int
natTimeout int
pprofEnabled bool
)
func main() {
log.SetFlags(log.Lshortfile | log.LstdFlags)
var dir, extAddress string
var dir, extAddress, proto string
flag.StringVar(&listen, "listen", ":22067", "Protocol listen address")
flag.StringVar(&dir, "keys", ".", "Directory where cert.pem and key.pem is stored")
@@ -89,18 +103,43 @@ func main() {
flag.StringVar(&poolAddrs, "pools", defaultPoolAddrs, "Comma separated list of relay pool addresses to join")
flag.StringVar(&providedBy, "provided-by", "", "An optional description about who provides the relay")
flag.StringVar(&extAddress, "ext-address", "", "An optional address to advertise as being available on.\n\tAllows listening on an unprivileged port with port forwarding from e.g. 443, and be connected to on port 443.")
flag.StringVar(&proto, "protocol", "tcp", "Protocol used for listening. 'tcp' for IPv4 and IPv6, 'tcp4' for IPv4, 'tcp6' for IPv6")
flag.BoolVar(&natEnabled, "nat", false, "Use UPnP/NAT-PMP to acquire external port mapping")
flag.IntVar(&natLease, "nat-lease", 60, "NAT lease length in minutes")
flag.IntVar(&natRenewal, "nat-renewal", 30, "NAT renewal frequency in minutes")
flag.IntVar(&natTimeout, "nat-timeout", 10, "NAT discovery timeout in seconds")
flag.BoolVar(&pprofEnabled, "pprof", false, "Enable the built in profiling on the status server")
flag.IntVar(&networkBufferSize, "network-buffer", 2048, "Network buffer size (two of these per proxied connection)")
flag.Parse()
if extAddress == "" {
extAddress = listen
}
addr, err := net.ResolveTCPAddr("tcp", extAddress)
if len(providedBy) > 30 {
log.Fatal("Provided-by cannot be longer than 30 characters")
}
addr, err := net.ResolveTCPAddr(proto, extAddress)
if err != nil {
log.Fatal(err)
}
laddr, err := net.ResolveTCPAddr(proto, listen)
if err != nil {
log.Fatal(err)
}
if laddr.IP != nil && !laddr.IP.IsUnspecified() {
laddr.Port = 0
transport, ok := http.DefaultTransport.(*http.Transport)
if ok {
transport.Dial = (&net.Dialer{
Timeout: 30 * time.Second,
LocalAddr: laddr,
}).Dial
}
}
log.Println(LongVersion)
maxDescriptors, err := osutil.MaximizeOpenFileLimit()
@@ -149,18 +188,49 @@ func main() {
log.Println("ID:", id)
}
wrapper := config.Wrap("config", config.New(id))
wrapper.SetOptions(config.OptionsConfiguration{
NATLeaseM: natLease,
NATRenewalM: natRenewal,
NATTimeoutS: natTimeout,
})
natSvc := nat.NewService(id, wrapper)
mapping := mapping{natSvc.NewMapping(nat.TCP, addr.IP, addr.Port)}
if natEnabled {
go natSvc.Serve()
found := make(chan struct{})
mapping.OnChanged(func(_ *nat.Mapping, _, _ []nat.Address) {
select {
case found <- struct{}{}:
default:
}
})
// Need to wait a few extra seconds, since NAT library waits exactly natTimeout seconds on all interfaces.
timeout := time.Duration(natTimeout+2) * time.Second
log.Printf("Waiting %s to acquire NAT mapping", timeout)
select {
case <-found:
log.Printf("Found NAT mapping: %s", mapping.ExternalAddresses())
case <-time.After(timeout):
log.Println("Timeout out waiting for NAT mapping.")
}
}
if sessionLimitBps > 0 {
sessionLimiter = ratelimit.NewBucketWithRate(float64(sessionLimitBps), int64(2*sessionLimitBps))
sessionLimiter = rate.NewLimiter(rate.Limit(sessionLimitBps), 2*sessionLimitBps)
}
if globalLimitBps > 0 {
globalLimiter = ratelimit.NewBucketWithRate(float64(globalLimitBps), int64(2*globalLimitBps))
globalLimiter = rate.NewLimiter(rate.Limit(globalLimitBps), 2*globalLimitBps)
}
if statusAddr != "" {
go statusService(statusAddr)
}
uri, err := url.Parse(fmt.Sprintf("relay://%s/?id=%s&pingInterval=%s&networkTimeout=%s&sessionLimitBps=%d&globalLimitBps=%d&statusAddr=%s&providedBy=%s", extAddress, id, pingInterval, networkTimeout, sessionLimitBps, globalLimitBps, statusAddr, providedBy))
uri, err := url.Parse(fmt.Sprintf("relay://%s/?id=%s&pingInterval=%s&networkTimeout=%s&sessionLimitBps=%d&globalLimitBps=%d&statusAddr=%s&providedBy=%s", mapping.Address(), id, pingInterval, networkTimeout, sessionLimitBps, globalLimitBps, statusAddr, providedBy))
if err != nil {
log.Fatalln("Failed to construct URI", err)
}
@@ -178,11 +248,11 @@ func main() {
for _, pool := range pools {
pool = strings.TrimSpace(pool)
if len(pool) > 0 {
go poolHandler(pool, uri)
go poolHandler(pool, uri, mapping)
}
}
go listener(listen, tlsCfg)
go listener(proto, listen, tlsCfg)
sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
@@ -212,7 +282,7 @@ func main() {
func monitorLimits() {
limitCheckTimer = time.NewTimer(time.Minute)
for _ = range limitCheckTimer.C {
for range limitCheckTimer.C {
if atomic.LoadInt64(&numConnections)+atomic.LoadInt64(&numProxies) > descriptorLimit {
atomic.StoreInt32(&overLimit, 1)
log.Println("Gone past our connection limits. Starting to refuse new/drop idle connections.")
@@ -222,3 +292,15 @@ func monitorLimits() {
limitCheckTimer.Reset(time.Minute)
}
}
type mapping struct {
*nat.Mapping
}
func (m *mapping) Address() nat.Address {
ext := m.ExternalAddresses()
if len(ext) > 0 {
return ext[0]
}
return m.Mapping.Address()
}

View File

@@ -12,16 +12,19 @@ import (
"time"
)
func poolHandler(pool string, uri *url.URL) {
func poolHandler(pool string, uri *url.URL, mapping mapping) {
if debug {
log.Println("Joining", pool)
}
for {
uriCopy := *uri
uriCopy.Host = mapping.Address().String()
var b bytes.Buffer
json.NewEncoder(&b).Encode(struct {
URL string `json:"url"`
}{
uri.String(),
uriCopy.String(),
})
resp, err := http.Post(pool, "application/json", &b)
@@ -39,7 +42,7 @@ func poolHandler(pool string, uri *url.URL) {
log.Println(pool, "under load, will retry in a minute")
time.Sleep(time.Minute)
continue
} else if resp.StatusCode == 403 {
} else if resp.StatusCode == 401 {
log.Println(pool, "failed to join due to IP address not matching external address. Aborting")
return
} else if resp.StatusCode == 200 {

View File

@@ -7,15 +7,16 @@ import (
"encoding/hex"
"fmt"
"log"
"math"
"net"
"sync"
"sync/atomic"
"time"
"github.com/juju/ratelimit"
"github.com/syncthing/syncthing/lib/relay/protocol"
"golang.org/x/time/rate"
syncthingprotocol "github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/relay/protocol"
)
var (
@@ -26,7 +27,7 @@ var (
bytesProxied int64
)
func newSession(serverid, clientid syncthingprotocol.DeviceID, sessionRateLimit, globalRateLimit *ratelimit.Bucket) *session {
func newSession(serverid, clientid syncthingprotocol.DeviceID, sessionRateLimit, globalRateLimit *rate.Limiter) *session {
serverkey := make([]byte, 32)
_, err := rand.Read(serverkey)
if err != nil {
@@ -108,7 +109,7 @@ type session struct {
clientkey []byte
clientid syncthingprotocol.DeviceID
rateLimit func(bytes int64)
rateLimit func(bytes int)
connsChan chan net.Conn
conns []net.Conn
@@ -216,7 +217,7 @@ done:
func (s *session) GetClientInvitationMessage() protocol.SessionInvitation {
return protocol.SessionInvitation{
From: s.serverid[:],
Key: []byte(s.clientkey),
Key: s.clientkey,
Address: sessionAddress,
Port: sessionPort,
ServerSocket: false,
@@ -226,7 +227,7 @@ func (s *session) GetClientInvitationMessage() protocol.SessionInvitation {
func (s *session) GetServerInvitationMessage() protocol.SessionInvitation {
return protocol.SessionInvitation{
From: s.clientid[:],
Key: []byte(s.serverkey),
Key: s.serverkey,
Address: sessionAddress,
Port: sessionPort,
ServerSocket: true,
@@ -253,7 +254,7 @@ func (s *session) proxy(c1, c2 net.Conn) error {
atomic.AddInt64(&numProxies, 1)
defer atomic.AddInt64(&numProxies, -1)
buf := make([]byte, 65536)
buf := make([]byte, networkBufferSize)
for {
c1.SetReadDeadline(time.Now().Add(networkTimeout))
n, err := c1.Read(buf)
@@ -268,7 +269,7 @@ func (s *session) proxy(c1, c2 net.Conn) error {
}
if s.rateLimit != nil {
s.rateLimit(int64(n))
s.rateLimit(n)
}
c2.SetWriteDeadline(time.Now().Add(networkTimeout))
@@ -283,7 +284,7 @@ func (s *session) String() string {
return fmt.Sprintf("<%s/%s>", hex.EncodeToString(s.clientkey)[:5], hex.EncodeToString(s.serverkey)[:5])
}
func makeRateLimitFunc(sessionRateLimit, globalRateLimit *ratelimit.Bucket) func(int64) {
func makeRateLimitFunc(sessionRateLimit, globalRateLimit *rate.Limiter) func(int) {
// This may be a case of super duper premature optimization... We build an
// optimized function to do the rate limiting here based on what we need
// to do and then use it in the loop.
@@ -298,29 +299,55 @@ func makeRateLimitFunc(sessionRateLimit, globalRateLimit *ratelimit.Bucket) func
if sessionRateLimit == nil {
// We only have a global limiter
return func(bytes int64) {
globalRateLimit.Wait(bytes)
return func(bytes int) {
take(bytes, globalRateLimit)
}
}
if globalRateLimit == nil {
// We only have a session limiter
return func(bytes int64) {
sessionRateLimit.Wait(bytes)
return func(bytes int) {
take(bytes, sessionRateLimit)
}
}
// We have both. Queue the bytes on both the global and session specific
// rate limiters. Wait for both in parallell, so that the actual send
// happens when both conditions are satisfied. In practice this just means
// wait the longer of the two times.
return func(bytes int64) {
t0 := sessionRateLimit.Take(bytes)
t1 := globalRateLimit.Take(bytes)
if t0 > t1 {
time.Sleep(t0)
} else {
time.Sleep(t1)
}
// rate limiters.
return func(bytes int) {
take(bytes, sessionRateLimit, globalRateLimit)
}
}
// take is a utility function to consume tokens from a set of rate.Limiters.
// Tokens are consumed in parallel on all limiters, respecting their
// individual burst sizes.
func take(tokens int, ls ...*rate.Limiter) {
// minBurst is the smallest burst size supported by all limiters.
minBurst := int(math.MaxInt32)
for _, l := range ls {
if burst := l.Burst(); burst < minBurst {
minBurst = burst
}
}
for tokens > 0 {
// chunk is how many tokens we can consume at a time
chunk := tokens
if chunk > minBurst {
chunk = minBurst
}
// maxDelay is the longest delay mandated by any of the limiters for
// the chosen chunk size.
var maxDelay time.Duration
for _, l := range ls {
res := l.ReserveN(time.Now(), chunk)
if del := res.Delay(); del > maxDelay {
maxDelay = del
}
}
time.Sleep(maxDelay)
tokens -= chunk
}
}

View File

@@ -6,6 +6,7 @@ import (
"encoding/json"
"log"
"net/http"
"net/http/pprof"
"runtime"
"sync/atomic"
"time"
@@ -16,8 +17,19 @@ var rc *rateCalculator
func statusService(addr string) {
rc = newRateCalculator(360, 10*time.Second, &bytesProxied)
http.HandleFunc("/status", getStatus)
if err := http.ListenAndServe(addr, nil); err != nil {
handler := http.NewServeMux()
handler.HandleFunc("/status", getStatus)
if pprofEnabled {
handler.HandleFunc("/debug/pprof/", pprof.Index)
}
srv := http.Server{
Addr: addr,
Handler: handler,
ReadTimeout: 15 * time.Second,
}
srv.SetKeepAlivesEnabled(false)
if err := srv.ListenAndServe(); err != nil {
log.Fatal(err)
}
}
@@ -42,7 +54,7 @@ func getStatus(w http.ResponseWriter, r *http.Request) {
status["goMaxProcs"] = runtime.GOMAXPROCS(-1)
status["goNumRoutine"] = runtime.NumGoroutine()
status["kbps10s1m5m15m30m60m"] = []int64{
rc.rate(10/10) * 8 / 1000,
rc.rate(1) * 8 / 1000, // each interval is 10s
rc.rate(60/10) * 8 / 1000,
rc.rate(5*60/10) * 8 / 1000,
rc.rate(15*60/10) * 8 / 1000,

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -130,7 +130,7 @@ func printProgress(prefix string, count *int64) {
expectedIterations := float64(int(1) << uint(wantBits))
fmt.Printf("Want %d bits for prefix %q, about %.2g certs to test (statistically speaking)\n", wantBits, prefix, expectedIterations)
for _ = range time.NewTicker(15 * time.Second).C {
for range time.NewTicker(15 * time.Second).C {
tried := atomic.LoadInt64(count)
elapsed := time.Since(started)
rate := float64(tried) / elapsed.Seconds()

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -20,13 +20,13 @@ func TestAuditService(t *testing.T) {
service := newAuditService(buf)
// Event sent before start, will not be logged
events.Default.Log(events.Ping, "the first event")
events.Default.Log(events.ConfigSaved, "the first event")
go service.Serve()
service.WaitForStart()
// Event that should end up in the audit log
events.Default.Log(events.Ping, "the second event")
events.Default.Log(events.ConfigSaved, "the second event")
// We need to give the events time to arrive, since the channels are buffered etc.
time.Sleep(10 * time.Millisecond)
@@ -35,7 +35,7 @@ func TestAuditService(t *testing.T) {
service.WaitForStop()
// This event should not be logged, since we have stopped.
events.Default.Log(events.Ping, "the third event")
events.Default.Log(events.ConfigSaved, "the third event")
result := string(buf.Bytes())
t.Log(result)

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -0,0 +1,13 @@
// Copyright (C) 2017 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
//+build noupgrade
package main
func init() {
BuildTags = append(BuildTags, "noupgrade")
}

View File

@@ -0,0 +1,13 @@
// Copyright (C) 2017 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
//+build race
package main
func init() {
BuildTags = append(BuildTags, "race")
}

59
cmd/syncthing/cpuusage.go Normal file
View File

@@ -0,0 +1,59 @@
// Copyright (C) 2017 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"math"
"time"
metrics "github.com/rcrowley/go-metrics"
)
const cpuTickRate = 5 * time.Second
type cpuService struct {
avg metrics.EWMA
stop chan struct{}
}
func newCPUService() *cpuService {
return &cpuService{
// 10 second average. Magic alpha value comes from looking at EWMA package
// definitions of EWMA1, EWMA5. The tick rate *must* be five seconds (hard
// coded in the EWMA package).
avg: metrics.NewEWMA(1 - math.Exp(-float64(cpuTickRate)/float64(time.Second)/10.0)),
stop: make(chan struct{}),
}
}
func (s *cpuService) Serve() {
// Initialize prevUsage to an actual value returned by cpuUsage
// instead of zero, because at least Windows returns a huge negative
// number here that then slowly increments...
prevUsage := cpuUsage()
ticker := time.NewTicker(cpuTickRate)
defer ticker.Stop()
for {
select {
case <-ticker.C:
curUsage := cpuUsage()
s.avg.Update(int64((curUsage - prevUsage) / time.Millisecond))
prevUsage = curUsage
s.avg.Tick()
case <-s.stop:
return
}
}
}
func (s *cpuService) Stop() {
close(s.stop)
}
func (s *cpuService) Rate() float64 {
return s.avg.Rate()
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
//+build solaris
@@ -59,40 +59,20 @@ type prusage_t struct {
}
func solarisPrusage(pid int, rusage *prusage_t) error {
fd, err := os.Open(fmt.Sprintf("/proc/%d/usage", pid))
var procFile = fmt.Sprintf("/proc/%d/usage", os.Getpid())
func cpuUsage() time.Duration {
fd, err := os.Open(procFile)
if err != nil {
return err
return 0
}
var rusage prusage_t
err = binary.Read(fd, binary.LittleEndian, rusage)
fd.Close()
return err
}
func init() {
go trackCPUUsage()
}
func trackCPUUsage() {
var prevUsage int64
var prevTime = time.Now().UnixNano()
var rusage prusage_t
var pid = os.Getpid()
for _ = range time.NewTicker(time.Second).C {
err := solarisPrusage(pid, &rusage)
if err != nil {
l.Warnln("getting prusage:", err)
continue
}
curTime := time.Now().UnixNano()
timeDiff := curTime - prevTime
curUsage := rusage.Pr_utime.Nano() + rusage.Pr_stime.Nano()
usageDiff := curUsage - prevUsage
cpuUsageLock.Lock()
copy(cpuUsagePercent[1:], cpuUsagePercent[0:])
cpuUsagePercent[0] = 100 * float64(usageDiff) / float64(timeDiff)
cpuUsageLock.Unlock()
prevTime = curTime
prevUsage = curUsage
if err != nil {
return 0
}
return time.Duration(rusage.Pr_utime.Nano() + rusage.Pr_stime.Nano())
}

View File

@@ -0,0 +1,18 @@
// Copyright (C) 2014 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
//+build !windows,!solaris
package main
import "syscall"
import "time"
func cpuUsage() time.Duration {
var rusage syscall.Rusage
syscall.Getrusage(syscall.RUSAGE_SELF, &rusage)
return time.Duration(rusage.Utime.Nano() + rusage.Stime.Nano())
}

View File

@@ -0,0 +1,27 @@
// Copyright (C) 2014 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
//+build windows
package main
import "syscall"
import "time"
func cpuUsage() time.Duration {
handle, err := syscall.GetCurrentProcess()
if err != nil {
return 0
}
defer syscall.CloseHandle(handle)
var ctime, etime, ktime, utime syscall.Filetime
if err := syscall.GetProcessTimes(handle, &ctime, &etime, &ktime, &utime); err != nil {
return 0
}
return time.Duration(ktime.Nanoseconds() + utime.Nanoseconds())
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -17,6 +17,7 @@ import (
"path/filepath"
"reflect"
"runtime"
"runtime/pprof"
"sort"
"strconv"
"strings"
@@ -27,9 +28,9 @@ import (
"github.com/syncthing/syncthing/lib/db"
"github.com/syncthing/syncthing/lib/discover"
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/fs"
"github.com/syncthing/syncthing/lib/logger"
"github.com/syncthing/syncthing/lib/model"
"github.com/syncthing/syncthing/lib/osutil"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/rand"
"github.com/syncthing/syncthing/lib/stats"
@@ -44,6 +45,12 @@ var (
startTime = time.Now()
)
const (
defaultEventMask = events.AllEvents &^ events.LocalChangeDetected &^ events.RemoteChangeDetected
diskEventMask = events.LocalChangeDetected | events.RemoteChangeDetected
eventSubBufferSize = 1000
)
type apiService struct {
id protocol.DeviceID
cfg configIntf
@@ -51,7 +58,8 @@ type apiService struct {
httpsKeyFile string
statics *staticsServer
model modelIntf
eventSub events.BufferedSubscription
eventSubs map[events.EventType]events.BufferedSubscription
eventSubsMut sync.Mutex
discoverer discover.CachingMux
connectionsService connectionsIntf
fss *folderSummaryService
@@ -59,7 +67,8 @@ type apiService struct {
stop chan struct{} // signals intentional stop
configChanged chan struct{} // signals intentional listener close due to config change
started chan string // signals startup complete by sending the listener address, for testing only
startedOnce bool // the service has started successfully at least once
startedOnce chan struct{} // the service has started successfully at least once
cpu rater
guiErrors logger.Recorder
systemLog logger.Recorder
@@ -67,10 +76,10 @@ type apiService struct {
type modelIntf interface {
GlobalDirectoryTree(folder, prefix string, levels int, dirsonly bool) map[string]interface{}
Completion(device protocol.DeviceID, folder string) float64
Completion(device protocol.DeviceID, folder string) model.FolderCompletion
Override(folder string)
NeedFolderFiles(folder string, page, perpage int) ([]db.FileInfoTruncated, []db.FileInfoTruncated, []db.FileInfoTruncated, int)
NeedSize(folder string) (nfiles int, bytes int64)
NeedSize(folder string) db.Counts
ConnectionStats() map[string]interface{}
DeviceStatistics() map[string]stats.DeviceStatistics
FolderStatistics() map[string]stats.FolderStatistics
@@ -80,29 +89,30 @@ type modelIntf interface {
Availability(folder, file string, version protocol.Vector, block protocol.BlockInfo) []model.Availability
GetIgnores(folder string) ([]string, []string, error)
SetIgnores(folder string, content []string) error
PauseDevice(device protocol.DeviceID)
ResumeDevice(device protocol.DeviceID)
DelayScan(folder string, next time.Duration)
ScanFolder(folder string) error
ScanFolders() map[string]error
ScanFolderSubdirs(folder string, subs []string) error
BringToFront(folder, file string)
ConnectedTo(deviceID protocol.DeviceID) bool
GlobalSize(folder string) (nfiles, deleted int, bytes int64)
LocalSize(folder string) (nfiles, deleted int, bytes int64)
CurrentLocalVersion(folder string) (int64, bool)
RemoteLocalVersion(folder string) (int64, bool)
GlobalSize(folder string) db.Counts
LocalSize(folder string) db.Counts
CurrentSequence(folder string) (int64, bool)
RemoteSequence(folder string) (int64, bool)
State(folder string) (string, time.Time, error)
UsageReportingStats(version int) map[string]interface{}
}
type configIntf interface {
GUI() config.GUIConfiguration
Raw() config.Configuration
RawCopy() config.Configuration
Options() config.OptionsConfiguration
Replace(cfg config.Configuration) error
Subscribe(c config.Committer)
Folders() map[string]config.FolderConfiguration
Devices() map[protocol.DeviceID]config.DeviceConfiguration
SetDevice(config.DeviceConfiguration) error
SetDevices([]config.DeviceConfiguration) error
Save() error
ListenAddresses() []string
RequiresRestart() bool
@@ -110,24 +120,35 @@ type configIntf interface {
type connectionsIntf interface {
Status() map[string]interface{}
NATType() string
}
func newAPIService(id protocol.DeviceID, cfg configIntf, httpsCertFile, httpsKeyFile, assetDir string, m modelIntf, eventSub events.BufferedSubscription, discoverer discover.CachingMux, connectionsService connectionsIntf, errors, systemLog logger.Recorder) *apiService {
type rater interface {
Rate() float64
}
func newAPIService(id protocol.DeviceID, cfg configIntf, httpsCertFile, httpsKeyFile, assetDir string, m modelIntf, defaultSub, diskSub events.BufferedSubscription, discoverer discover.CachingMux, connectionsService connectionsIntf, errors, systemLog logger.Recorder, cpu rater) *apiService {
service := &apiService{
id: id,
cfg: cfg,
httpsCertFile: httpsCertFile,
httpsKeyFile: httpsKeyFile,
statics: newStaticsServer(cfg.GUI().Theme, assetDir),
model: m,
eventSub: eventSub,
id: id,
cfg: cfg,
httpsCertFile: httpsCertFile,
httpsKeyFile: httpsKeyFile,
statics: newStaticsServer(cfg.GUI().Theme, assetDir),
model: m,
eventSubs: map[events.EventType]events.BufferedSubscription{
defaultEventMask: defaultSub,
diskEventMask: diskSub,
},
eventSubsMut: sync.NewMutex(),
discoverer: discoverer,
connectionsService: connectionsService,
systemConfigMut: sync.NewMutex(),
stop: make(chan struct{}),
configChanged: make(chan struct{}),
startedOnce: make(chan struct{}),
guiErrors: errors,
systemLog: systemLog,
cpu: cpu,
}
return service
@@ -199,20 +220,20 @@ func sendJSON(w http.ResponseWriter, jsonObject interface{}) {
func (s *apiService) Serve() {
listener, err := s.getListener(s.cfg.GUI())
if err != nil {
if !s.startedOnce {
select {
case <-s.startedOnce:
// We let this be a loud user-visible warning as it may be the only
// indication they get that the GUI won't be available.
l.Warnln("Starting API/GUI:", err)
return
default:
// This is during initialization. A failure here should be fatal
// as there will be no way for the user to communicate with us
// otherwise anyway.
l.Fatalln("Starting API/GUI:", err)
}
// We let this be a loud user-visible warning as it may be the only
// indication they get that the GUI won't be available on startup.
l.Warnln("Starting API/GUI:", err)
return
}
s.startedOnce = true
defer listener.Close()
if listener == nil {
// Not much we can do here other than exit quickly. The supervisor
@@ -220,6 +241,8 @@ func (s *apiService) Serve() {
return
}
defer listener.Close()
// The GET handlers
getRestMux := http.NewServeMux()
getRestMux.HandleFunc("/rest/db/completion", s.getDBCompletion) // device folder
@@ -228,7 +251,8 @@ func (s *apiService) Serve() {
getRestMux.HandleFunc("/rest/db/need", s.getDBNeed) // folder [perpage] [page]
getRestMux.HandleFunc("/rest/db/status", s.getDBStatus) // folder
getRestMux.HandleFunc("/rest/db/browse", s.getDBBrowse) // folder [prefix] [dirsonly] [levels]
getRestMux.HandleFunc("/rest/events", s.getEvents) // since [limit]
getRestMux.HandleFunc("/rest/events", s.getIndexEvents) // [since] [limit] [timeout] [events]
getRestMux.HandleFunc("/rest/events/disk", s.getDiskEvents) // [since] [limit] [timeout]
getRestMux.HandleFunc("/rest/stats/device", s.getDeviceStats) // -
getRestMux.HandleFunc("/rest/stats/folder", s.getFolderStats) // -
getRestMux.HandleFunc("/rest/svc/deviceid", s.getDeviceID) // id
@@ -251,25 +275,29 @@ func (s *apiService) Serve() {
// The POST handlers
postRestMux := http.NewServeMux()
postRestMux.HandleFunc("/rest/db/prio", s.postDBPrio) // folder file [perpage] [page]
postRestMux.HandleFunc("/rest/db/ignores", s.postDBIgnores) // folder
postRestMux.HandleFunc("/rest/db/override", s.postDBOverride) // folder
postRestMux.HandleFunc("/rest/db/scan", s.postDBScan) // folder [sub...] [delay]
postRestMux.HandleFunc("/rest/system/config", s.postSystemConfig) // <body>
postRestMux.HandleFunc("/rest/system/error", s.postSystemError) // <body>
postRestMux.HandleFunc("/rest/system/error/clear", s.postSystemErrorClear) // -
postRestMux.HandleFunc("/rest/system/ping", s.restPing) // -
postRestMux.HandleFunc("/rest/system/reset", s.postSystemReset) // [folder]
postRestMux.HandleFunc("/rest/system/restart", s.postSystemRestart) // -
postRestMux.HandleFunc("/rest/system/shutdown", s.postSystemShutdown) // -
postRestMux.HandleFunc("/rest/system/upgrade", s.postSystemUpgrade) // -
postRestMux.HandleFunc("/rest/system/pause", s.postSystemPause) // device
postRestMux.HandleFunc("/rest/system/resume", s.postSystemResume) // device
postRestMux.HandleFunc("/rest/system/debug", s.postSystemDebug) // [enable] [disable]
postRestMux.HandleFunc("/rest/db/prio", s.postDBPrio) // folder file [perpage] [page]
postRestMux.HandleFunc("/rest/db/ignores", s.postDBIgnores) // folder
postRestMux.HandleFunc("/rest/db/override", s.postDBOverride) // folder
postRestMux.HandleFunc("/rest/db/scan", s.postDBScan) // folder [sub...] [delay]
postRestMux.HandleFunc("/rest/system/config", s.postSystemConfig) // <body>
postRestMux.HandleFunc("/rest/system/error", s.postSystemError) // <body>
postRestMux.HandleFunc("/rest/system/error/clear", s.postSystemErrorClear) // -
postRestMux.HandleFunc("/rest/system/ping", s.restPing) // -
postRestMux.HandleFunc("/rest/system/reset", s.postSystemReset) // [folder]
postRestMux.HandleFunc("/rest/system/restart", s.postSystemRestart) // -
postRestMux.HandleFunc("/rest/system/shutdown", s.postSystemShutdown) // -
postRestMux.HandleFunc("/rest/system/upgrade", s.postSystemUpgrade) // -
postRestMux.HandleFunc("/rest/system/pause", s.makeDevicePauseHandler(true)) // [device]
postRestMux.HandleFunc("/rest/system/resume", s.makeDevicePauseHandler(false)) // [device]
postRestMux.HandleFunc("/rest/system/debug", s.postSystemDebug) // [enable] [disable]
// Debug endpoints, not for general use
getRestMux.HandleFunc("/rest/debug/peerCompletion", s.getPeerCompletion)
getRestMux.HandleFunc("/rest/debug/httpmetrics", s.getSystemHTTPMetrics)
debugMux := http.NewServeMux()
debugMux.HandleFunc("/rest/debug/peerCompletion", s.getPeerCompletion)
debugMux.HandleFunc("/rest/debug/httpmetrics", s.getSystemHTTPMetrics)
debugMux.HandleFunc("/rest/debug/cpuprof", s.getCPUProf) // duration
debugMux.HandleFunc("/rest/debug/heapprof", s.getHeapProf)
getRestMux.Handle("/rest/debug/", s.whenDebugging(debugMux))
// A handler that splits requests between the two above and disables
// caching
@@ -306,13 +334,20 @@ func (s *apiService) Serve() {
}
// Add the CORS handling
handler = corsMiddleware(handler)
handler = corsMiddleware(handler, guiCfg.InsecureAllowFrameLoading)
if addressIsLocalhost(guiCfg.Address()) && !guiCfg.InsecureSkipHostCheck {
// Verify source host
handler = localhostMiddleware(handler)
}
handler = debugMiddleware(handler)
srv := http.Server{
Handler: handler,
ReadTimeout: 10 * time.Second,
Handler: handler,
// ReadTimeout must be longer than SyncthingController $scope.refresh
// interval to avoid HTTP keepalive/GUI refresh race.
ReadTimeout: 15 * time.Second,
}
s.fss = newFolderSummaryService(s.cfg, s.model)
@@ -326,6 +361,14 @@ func (s *apiService) Serve() {
s.started <- listener.Addr().String()
}
// Indicate successful initial startup, to ourselves and to interested
// listeners (i.e. the thing that starts the browser).
select {
case <-s.startedOnce:
default:
close(s.startedOnce)
}
// Serve in the background
serveError := make(chan error, 1)
@@ -357,13 +400,14 @@ func (s *apiService) String() string {
}
func (s *apiService) VerifyConfiguration(from, to config.Configuration) error {
if _, err := net.ResolveTCPAddr("tcp", to.GUI.Address()); err != nil {
return err
}
return nil
_, err := net.ResolveTCPAddr("tcp", to.GUI.Address())
return err
}
func (s *apiService) CommitConfiguration(from, to config.Configuration) bool {
// No action required when this changes, so mask the fact that it changed at all.
from.GUI.Debugging = to.GUI.Debugging
if to.GUI == from.GUI {
return true
}
@@ -417,7 +461,7 @@ func debugMiddleware(h http.Handler) http.Handler {
})
}
func corsMiddleware(next http.Handler) http.Handler {
func corsMiddleware(next http.Handler, allowFrameLoading bool) http.Handler {
// Handle CORS headers and CORS OPTIONS request.
// CORS OPTIONS request are typically sent by browser during AJAX preflight
// when the browser initiate a POST request.
@@ -429,10 +473,12 @@ func corsMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Process OPTIONS requests
if r.Method == "OPTIONS" {
// Add a generous access-control-allow-origin header for CORS requests
w.Header().Add("Access-Control-Allow-Origin", "*")
// Only GET/POST Methods are supported
w.Header().Set("Access-Control-Allow-Methods", "GET, POST")
// Only this custom header can be set
w.Header().Set("Access-Control-Allow-Headers", "X-API-Key")
// Only these headers can be set
w.Header().Set("Access-Control-Allow-Headers", "Content-Type, X-API-Key")
// The request is meant to be cached 10 minutes
w.Header().Set("Access-Control-Max-Age", "600")
@@ -442,6 +488,27 @@ func corsMiddleware(next http.Handler) http.Handler {
return
}
// Other security related headers that should be present.
// https://www.owasp.org/index.php/Security_Headers
if !allowFrameLoading {
// We don't want to be rendered in an <iframe>,
// <frame> or <object>. (Unless we do it ourselves.
// This is also an escape hatch for people who serve
// Syncthing GUI as part of their own website
// through a proxy, so they don't need to set the
// allowFrameLoading bool.)
w.Header().Set("X-Frame-Options", "SAMEORIGIN")
}
// If the browser senses an XSS attack it's allowed to take
// action. (How this would not always be the default I
// don't fully understand.)
w.Header().Set("X-XSS-Protection", "1; mode=block")
// Our content type headers are correct. Don't guess.
w.Header().Set("X-Content-Type-Options", "nosniff")
// For everything else, pass to the next handler
next.ServeHTTP(w, r)
return
@@ -487,6 +554,28 @@ func withDetailsMiddleware(id protocol.DeviceID, h http.Handler) http.Handler {
})
}
func localhostMiddleware(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if addressIsLocalhost(r.Host) {
h.ServeHTTP(w, r)
return
}
http.Error(w, "Host check error", http.StatusForbidden)
})
}
func (s *apiService) whenDebugging(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if s.cfg.GUI().Debugging {
h.ServeHTTP(w, r)
return
}
http.Error(w, "Debugging disabled", http.StatusBadRequest)
})
}
func (s *apiService) restPing(w http.ResponseWriter, r *http.Request) {
sendJSON(w, map[string]string{"ping": "pong"})
}
@@ -563,8 +652,12 @@ func (s *apiService) getDBCompletion(w http.ResponseWriter, r *http.Request) {
return
}
sendJSON(w, map[string]float64{
"completion": s.model.Completion(device, folder),
comp := s.model.Completion(device, folder)
sendJSON(w, map[string]interface{}{
"completion": comp.CompletionPct,
"needBytes": comp.NeedBytes,
"globalBytes": comp.GlobalBytes,
"needDeletes": comp.NeedDeletes,
})
}
@@ -579,16 +672,16 @@ func folderSummary(cfg configIntf, m modelIntf, folder string) map[string]interf
res["invalid"] = "" // Deprecated, retains external API for now
globalFiles, globalDeleted, globalBytes := m.GlobalSize(folder)
res["globalFiles"], res["globalDeleted"], res["globalBytes"] = globalFiles, globalDeleted, globalBytes
global := m.GlobalSize(folder)
res["globalFiles"], res["globalDirectories"], res["globalSymlinks"], res["globalDeleted"], res["globalBytes"] = global.Files, global.Directories, global.Symlinks, global.Deleted, global.Bytes
localFiles, localDeleted, localBytes := m.LocalSize(folder)
res["localFiles"], res["localDeleted"], res["localBytes"] = localFiles, localDeleted, localBytes
local := m.LocalSize(folder)
res["localFiles"], res["localDirectories"], res["localSymlinks"], res["localDeleted"], res["localBytes"] = local.Files, local.Directories, local.Symlinks, local.Deleted, local.Bytes
needFiles, needBytes := m.NeedSize(folder)
res["needFiles"], res["needBytes"] = needFiles, needBytes
need := m.NeedSize(folder)
res["needFiles"], res["needDirectories"], res["needSymlinks"], res["needDeletes"], res["needBytes"] = need.Files, need.Directories, need.Symlinks, need.Deleted, need.Bytes
res["inSyncFiles"], res["inSyncBytes"] = globalFiles-needFiles, globalBytes-needBytes
res["inSyncFiles"], res["inSyncBytes"] = global.Files-need.Files, global.Bytes-need.Bytes
var err error
res["state"], res["stateChanged"], err = m.State(folder)
@@ -596,10 +689,11 @@ func folderSummary(cfg configIntf, m modelIntf, folder string) map[string]interf
res["error"] = err.Error()
}
lv, _ := m.CurrentLocalVersion(folder)
rv, _ := m.RemoteLocalVersion(folder)
ourSeq, _ := m.CurrentSequence(folder)
remoteSeq, _ := m.RemoteSequence(folder)
res["version"] = lv + rv
res["version"] = ourSeq + remoteSeq // legacy
res["sequence"] = ourSeq + remoteSeq // new name
ignorePatterns, _, _ := m.GetIgnores(folder)
res["ignorePatterns"] = false
@@ -680,7 +774,7 @@ func (s *apiService) getDBFile(w http.ResponseWriter, r *http.Request) {
}
func (s *apiService) getSystemConfig(w http.ResponseWriter, r *http.Request) {
sendJSON(w, s.cfg.Raw())
sendJSON(w, s.cfg.RawCopy())
}
func (s *apiService) postSystemConfig(w http.ResponseWriter, r *http.Request) {
@@ -708,26 +802,16 @@ func (s *apiService) postSystemConfig(w http.ResponseWriter, r *http.Request) {
}
}
// Fixup usage reporting settings
if curAcc := s.cfg.Options().URAccepted; to.Options.URAccepted > curAcc {
// UR was enabled
to.Options.URAccepted = usageReportVersion
to.Options.URUniqueID = rand.String(8)
} else if to.Options.URAccepted < curAcc {
// UR was disabled
to.Options.URAccepted = -1
to.Options.URUniqueID = ""
}
// Activate and save
if err := s.cfg.Replace(to); err != nil {
l.Warnln("Replacing config:", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if err := s.cfg.Save(); err != nil {
l.Warnln("Saving config:", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
@@ -779,14 +863,11 @@ func (s *apiService) flushResponse(resp string, w http.ResponseWriter) {
f.Flush()
}
var cpuUsagePercent [10]float64 // The last ten seconds
var cpuUsageLock = sync.NewRWMutex()
func (s *apiService) getSystemStatus(w http.ResponseWriter, r *http.Request) {
var m runtime.MemStats
runtime.ReadMemStats(&m)
tilde, _ := osutil.ExpandTilde("~")
tilde, _ := fs.ExpandTilde("~")
res := make(map[string]interface{})
res["myID"] = myID.String()
res["goroutines"] = runtime.NumGoroutine()
@@ -808,15 +889,11 @@ func (s *apiService) getSystemStatus(w http.ResponseWriter, r *http.Request) {
}
res["connectionServiceStatus"] = s.connectionsService.Status()
cpuUsageLock.RLock()
var cpusum float64
for _, p := range cpuUsagePercent {
cpusum += p
}
cpuUsageLock.RUnlock()
res["cpuPercent"] = cpusum / float64(len(cpuUsagePercent)) / float64(runtime.NumCPU())
// cpuUsage.Rate() is in milliseconds per second, so dividing by ten
// gives us percent
res["cpuPercent"] = s.cpu.Rate() / 10 / float64(runtime.NumCPU())
res["pathSeparator"] = string(filepath.Separator)
res["urVersionMax"] = usageReportVersion
res["uptime"] = int(time.Since(startTime).Seconds())
res["startTime"] = startTime
@@ -895,7 +972,11 @@ func (s *apiService) getSystemDiscovery(w http.ResponseWriter, r *http.Request)
}
func (s *apiService) getReport(w http.ResponseWriter, r *http.Request) {
sendJSON(w, reportData(s.cfg, s.model))
version := usageReportVersion
if val, _ := strconv.Atoi(r.URL.Query().Get("version")); val > 0 {
version = val
}
sendJSON(w, reportData(s.cfg, s.model, s.connectionsService, version))
}
func (s *apiService) getRandomString(w http.ResponseWriter, r *http.Request) {
@@ -911,7 +992,9 @@ func (s *apiService) getRandomString(w http.ResponseWriter, r *http.Request) {
func (s *apiService) getDBIgnores(w http.ResponseWriter, r *http.Request) {
qs := r.URL.Query()
ignores, patterns, err := s.model.GetIgnores(qs.Get("folder"))
folder := qs.Get("folder")
ignores, patterns, err := s.model.GetIgnores(folder)
if err != nil {
http.Error(w, err.Error(), 500)
return
@@ -949,14 +1032,30 @@ func (s *apiService) postDBIgnores(w http.ResponseWriter, r *http.Request) {
s.getDBIgnores(w, r)
}
func (s *apiService) getEvents(w http.ResponseWriter, r *http.Request) {
func (s *apiService) getIndexEvents(w http.ResponseWriter, r *http.Request) {
s.fss.gotEventRequest()
mask := s.getEventMask(r.URL.Query().Get("events"))
sub := s.getEventSub(mask)
s.getEvents(w, r, sub)
}
func (s *apiService) getDiskEvents(w http.ResponseWriter, r *http.Request) {
sub := s.getEventSub(diskEventMask)
s.getEvents(w, r, sub)
}
func (s *apiService) getEvents(w http.ResponseWriter, r *http.Request, eventSub events.BufferedSubscription) {
qs := r.URL.Query()
sinceStr := qs.Get("since")
limitStr := qs.Get("limit")
timeoutStr := qs.Get("timeout")
since, _ := strconv.Atoi(sinceStr)
limit, _ := strconv.Atoi(limitStr)
s.fss.gotEventRequest()
timeout := defaultEventTimeout
if timeoutSec, timeoutErr := strconv.Atoi(timeoutStr); timeoutErr == nil && timeoutSec >= 0 { // 0 is a valid timeout
timeout = time.Duration(timeoutSec) * time.Second
}
// Flush before blocking, to indicate that we've received the request and
// that it should not be retried. Must set Content-Type header before
@@ -965,7 +1064,8 @@ func (s *apiService) getEvents(w http.ResponseWriter, r *http.Request) {
f := w.(http.Flusher)
f.Flush()
evs := s.eventSub.Since(since, nil)
// If there are no events available return an empty slice, as this gets serialized as `[]`
evs := eventSub.Since(since, []events.Event{}, timeout)
if 0 < limit && limit < len(evs) {
evs = evs[len(evs)-limit:]
}
@@ -973,12 +1073,38 @@ func (s *apiService) getEvents(w http.ResponseWriter, r *http.Request) {
sendJSON(w, evs)
}
func (s *apiService) getEventMask(evs string) events.EventType {
eventMask := defaultEventMask
if evs != "" {
eventList := strings.Split(evs, ",")
eventMask = 0
for _, ev := range eventList {
eventMask |= events.UnmarshalEventType(strings.TrimSpace(ev))
}
}
return eventMask
}
func (s *apiService) getEventSub(mask events.EventType) events.BufferedSubscription {
s.eventSubsMut.Lock()
bufsub, ok := s.eventSubs[mask]
if !ok {
evsub := events.Default.Subscribe(mask)
bufsub = events.NewBufferedSubscription(evsub, eventSubBufferSize)
s.eventSubs[mask] = bufsub
}
s.eventSubsMut.Unlock()
return bufsub
}
func (s *apiService) getSystemUpgrade(w http.ResponseWriter, r *http.Request) {
if noUpgrade {
if noUpgradeFromEnv {
http.Error(w, upgrade.ErrUpgradeUnsupported.Error(), 500)
return
}
rel, err := upgrade.LatestRelease(s.cfg.Options().ReleasesURL, Version)
opts := s.cfg.Options()
rel, err := upgrade.LatestRelease(opts.ReleasesURL, Version, opts.UpgradeToPreReleases)
if err != nil {
http.Error(w, err.Error(), 500)
return
@@ -1019,7 +1145,8 @@ func (s *apiService) getLang(w http.ResponseWriter, r *http.Request) {
}
func (s *apiService) postSystemUpgrade(w http.ResponseWriter, r *http.Request) {
rel, err := upgrade.LatestRelease(s.cfg.Options().ReleasesURL, Version)
opts := s.cfg.Options()
rel, err := upgrade.LatestRelease(opts.ReleasesURL, Version, opts.UpgradeToPreReleases)
if err != nil {
l.Warnln("getting latest release:", err)
http.Error(w, err.Error(), 500)
@@ -1040,48 +1167,56 @@ func (s *apiService) postSystemUpgrade(w http.ResponseWriter, r *http.Request) {
}
}
func (s *apiService) postSystemPause(w http.ResponseWriter, r *http.Request) {
var qs = r.URL.Query()
var deviceStr = qs.Get("device")
func (s *apiService) makeDevicePauseHandler(paused bool) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var qs = r.URL.Query()
var deviceStr = qs.Get("device")
device, err := protocol.DeviceIDFromString(deviceStr)
if err != nil {
http.Error(w, err.Error(), 500)
return
var cfgs []config.DeviceConfiguration
if deviceStr == "" {
for _, cfg := range s.cfg.Devices() {
cfg.Paused = paused
cfgs = append(cfgs, cfg)
}
} else {
device, err := protocol.DeviceIDFromString(deviceStr)
if err != nil {
http.Error(w, err.Error(), 500)
return
}
cfg, ok := s.cfg.Devices()[device]
if !ok {
http.Error(w, "not found", http.StatusNotFound)
return
}
cfg.Paused = paused
cfgs = append(cfgs, cfg)
}
if err := s.cfg.SetDevices(cfgs); err != nil {
http.Error(w, err.Error(), 500)
}
}
s.model.PauseDevice(device)
}
func (s *apiService) postSystemResume(w http.ResponseWriter, r *http.Request) {
var qs = r.URL.Query()
var deviceStr = qs.Get("device")
device, err := protocol.DeviceIDFromString(deviceStr)
if err != nil {
http.Error(w, err.Error(), 500)
return
}
s.model.ResumeDevice(device)
}
func (s *apiService) postDBScan(w http.ResponseWriter, r *http.Request) {
qs := r.URL.Query()
folder := qs.Get("folder")
if folder != "" {
subs := qs["sub"]
err := s.model.ScanFolderSubdirs(folder, subs)
if err != nil {
http.Error(w, err.Error(), 500)
return
}
nextStr := qs.Get("next")
next, err := strconv.Atoi(nextStr)
if err == nil {
s.model.DelayScan(folder, time.Duration(next)*time.Second)
}
subs := qs["sub"]
err = s.model.ScanFolderSubdirs(folder, subs)
if err != nil {
http.Error(w, err.Error(), 500)
return
}
} else {
errors := s.model.ScanFolders()
if len(errors) > 0 {
@@ -1121,7 +1256,7 @@ func (s *apiService) getPeerCompletion(w http.ResponseWriter, r *http.Request) {
for _, device := range folder.DeviceIDs() {
deviceStr := device.String()
if s.model.ConnectedTo(device) {
tot[deviceStr] += s.model.Completion(device, folder.ID)
tot[deviceStr] += s.model.Completion(device, folder.ID).CompletionPct
} else {
tot[deviceStr] = 0
}
@@ -1140,23 +1275,35 @@ func (s *apiService) getPeerCompletion(w http.ResponseWriter, r *http.Request) {
func (s *apiService) getSystemBrowse(w http.ResponseWriter, r *http.Request) {
qs := r.URL.Query()
current := qs.Get("current")
// Default value or in case of error unmarshalling ends up being basic fs.
var fsType fs.FilesystemType
fsType.UnmarshalText([]byte(qs.Get("filesystem")))
if current == "" {
if roots, err := osutil.GetFilesystemRoots(); err == nil {
filesystem := fs.NewFilesystem(fsType, "")
if roots, err := filesystem.Roots(); err == nil {
sendJSON(w, roots)
} else {
http.Error(w, err.Error(), 500)
}
return
}
search, _ := osutil.ExpandTilde(current)
pathSeparator := string(os.PathSeparator)
search, _ := fs.ExpandTilde(current)
pathSeparator := string(fs.PathSeparator)
if strings.HasSuffix(current, pathSeparator) && !strings.HasSuffix(search, pathSeparator) {
search = search + pathSeparator
}
subdirectories, _ := osutil.Glob(search + "*")
searchDir := filepath.Dir(search)
searchFile := filepath.Base(search)
fs := fs.NewFilesystem(fsType, searchDir)
subdirectories, _ := fs.Glob(searchFile + "*")
ret := make([]string, 0, len(subdirectories))
for _, subdirectory := range subdirectories {
info, err := os.Stat(subdirectory)
info, err := fs.Stat(subdirectory)
if err == nil && info.IsDir() {
ret = append(ret, subdirectory+pathSeparator)
}
@@ -1165,6 +1312,32 @@ func (s *apiService) getSystemBrowse(w http.ResponseWriter, r *http.Request) {
sendJSON(w, ret)
}
func (s *apiService) getCPUProf(w http.ResponseWriter, r *http.Request) {
duration, err := time.ParseDuration(r.FormValue("duration"))
if err != nil {
duration = 30 * time.Second
}
filename := fmt.Sprintf("syncthing-cpu-%s-%s-%s-%s.pprof", runtime.GOOS, runtime.GOARCH, Version, time.Now().Format("150405")) // hhmmss
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Disposition", "attachment; filename="+filename)
pprof.StartCPUProfile(w)
time.Sleep(duration)
pprof.StopCPUProfile()
}
func (s *apiService) getHeapProf(w http.ResponseWriter, r *http.Request) {
filename := fmt.Sprintf("syncthing-heap-%s-%s-%s-%s.pprof", runtime.GOOS, runtime.GOARCH, Version, time.Now().Format("150405")) // hhmmss
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Disposition", "attachment; filename="+filename)
runtime.GC()
pprof.WriteHeapProfile(w)
}
func (s *apiService) toNeedSlice(fs []db.FileInfoTruncated) []jsonDBFileInfo {
res := make([]jsonDBFileInfo, len(fs))
for i, f := range fs {
@@ -1186,8 +1359,8 @@ func (f jsonFileInfo) MarshalJSON() ([]byte, error) {
"deleted": f.Deleted,
"invalid": f.Invalid,
"noPermissions": f.NoPermissions,
"modified": time.Unix(f.Modified, 0),
"localVersion": f.LocalVersion,
"modified": protocol.FileInfo(f).ModTime(),
"sequence": f.Sequence,
"numBlocks": len(f.Blocks),
"version": jsonVersionVector(f.Version),
})
@@ -1204,8 +1377,8 @@ func (f jsonDBFileInfo) MarshalJSON() ([]byte, error) {
"deleted": f.Deleted,
"invalid": f.Invalid,
"noPermissions": f.NoPermissions,
"modified": time.Unix(f.Modified, 0),
"localVersion": f.LocalVersion,
"modified": db.FileInfoTruncated(f).ModTime(),
"sequence": f.Sequence,
})
}
@@ -1241,3 +1414,17 @@ func dirNames(dir string) []string {
sort.Strings(dirs)
return dirs
}
func addressIsLocalhost(addr string) bool {
host, _, err := net.SplitHostPort(addr)
if err != nil {
// There was no port, so we assume the address was just a hostname
host = addr
}
switch strings.ToLower(host) {
case "127.0.0.1", "::1", "localhost":
return true
default:
return false
}
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -37,6 +37,16 @@ func csrfMiddleware(unique string, prefix string, cfg config.GUIConfiguration, n
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Allow requests carrying a valid API key
if cfg.IsValidAPIKey(r.Header.Get("X-API-Key")) {
// Set the access-control-allow-origin header for CORS requests
// since a valid API key has been provided
w.Header().Add("Access-Control-Allow-Origin", "*")
next.ServeHTTP(w, r)
return
}
if strings.HasPrefix(r.URL.Path, "/rest/debug") {
// Debugging functions are only available when explicitly
// enabled, and can be accessed without a CSRF token
next.ServeHTTP(w, r)
return
}
@@ -106,7 +116,7 @@ func saveCsrfTokens() {
// nothing relevant we can do about them anyway...
name := locations[locCsrfTokens]
f, err := osutil.CreateAtomic(name, 0600)
f, err := osutil.CreateAtomic(name)
if err != nil {
return
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -141,17 +141,17 @@ func (s *staticsServer) serveThemes(w http.ResponseWriter, r *http.Request) {
func (s *staticsServer) mimeTypeForFile(file string) string {
// We use a built in table of the common types since the system
// TypeByExtension might be unreliable. But if we don't know, we delegate
// to the system.
// to the system. All our files are UTF-8.
ext := filepath.Ext(file)
switch ext {
case ".htm", ".html":
return "text/html"
return "text/html; charset=utf-8"
case ".css":
return "text/css"
return "text/css; charset=utf-8"
case ".js":
return "application/javascript"
return "application/javascript; charset=utf-8"
case ".json":
return "application/json"
return "application/json; charset=utf-8"
case ".png":
return "image/png"
case ".ttf":
@@ -159,7 +159,7 @@ func (s *staticsServer) mimeTypeForFile(file string) string {
case ".woff":
return "application/x-font-woff"
case ".svg":
return "image/svg+xml"
return "image/svg+xml; charset=utf-8"
default:
return mime.TypeByExtension(ext)
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -16,12 +16,14 @@ import (
"net"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"time"
"github.com/d4l3k/messagediff"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/sync"
"github.com/thejerf/suture"
@@ -69,7 +71,7 @@ func TestStopAfterBrokenConfig(t *testing.T) {
}
w := config.Wrap("/dev/null", cfg)
srv := newAPIService(protocol.LocalDeviceID, w, "../../test/h1/https-cert.pem", "../../test/h1/https-key.pem", "", nil, nil, nil, nil, nil, nil)
srv := newAPIService(protocol.LocalDeviceID, w, "../../test/h1/https-cert.pem", "../../test/h1/https-key.pem", "", nil, nil, nil, nil, nil, nil, nil, nil)
srv.started = make(chan string)
sup := suture.NewSimple("test")
@@ -460,7 +462,6 @@ func TestHTTPLogin(t *testing.T) {
if resp.StatusCode != http.StatusOK {
t.Errorf("Unexpected non-200 return code %d for authed request (ISO-8859-1)", resp.StatusCode)
}
}
func startHTTP(cfg *mockedConfig) (string, error) {
@@ -469,15 +470,17 @@ func startHTTP(cfg *mockedConfig) (string, error) {
httpsKeyFile := "../../test/h1/https-key.pem"
assetDir := "../../gui"
eventSub := new(mockedEventSub)
diskEventSub := new(mockedEventSub)
discoverer := new(mockedCachingMux)
connections := new(mockedConnections)
errorLog := new(mockedLoggerRecorder)
systemLog := new(mockedLoggerRecorder)
cpu := new(mockedCPUService)
addrChan := make(chan string)
// Instantiate the API service
svc := newAPIService(protocol.LocalDeviceID, cfg, httpsCertFile, httpsKeyFile, assetDir, model,
eventSub, discoverer, connections, errorLog, systemLog)
eventSub, diskEventSub, discoverer, connections, errorLog, systemLog, cpu)
svc.started = addrChan
// Actually start the API service
@@ -491,7 +494,12 @@ func startHTTP(cfg *mockedConfig) (string, error) {
if err != nil {
return "", fmt.Errorf("Weird address from API service: %v", err)
}
baseURL := fmt.Sprintf("http://127.0.0.1:%d", tcpAddr.Port)
host, _, _ := net.SplitHostPort(cfg.gui.RawAddress)
if host == "" || host == "0.0.0.0" {
host = "127.0.0.1"
}
baseURL := fmt.Sprintf("http://%s", net.JoinHostPort(host, strconv.Itoa(tcpAddr.Port)))
return baseURL, nil
}
@@ -501,6 +509,9 @@ func TestCSRFRequired(t *testing.T) {
cfg := new(mockedConfig)
cfg.gui.APIKey = testAPIKey
baseURL, err := startHTTP(cfg)
if err != nil {
t.Fatal("Unexpected error from getting base URL:", err)
}
cli := &http.Client{
Timeout: time.Second,
@@ -666,3 +677,283 @@ func testConfigPost(data io.Reader) (*http.Response, error) {
req.Header.Set("X-API-Key", testAPIKey)
return cli.Do(req)
}
func TestHostCheck(t *testing.T) {
// An API service bound to localhost should reject non-localhost host Headers
cfg := new(mockedConfig)
cfg.gui.RawAddress = "127.0.0.1:0"
baseURL, err := startHTTP(cfg)
if err != nil {
t.Fatal(err)
}
// A normal HTTP get to the localhost-bound service should succeed
resp, err := http.Get(baseURL)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Error("Regular HTTP get: expected 200 OK, not", resp.Status)
}
// A request with a suspicious Host header should fail
req, _ := http.NewRequest("GET", baseURL, nil)
req.Host = "example.com"
resp, err = http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Error("Suspicious Host header: expected 403 Forbidden, not", resp.Status)
}
// A request with an explicit "localhost:8384" Host header should pass
req, _ = http.NewRequest("GET", baseURL, nil)
req.Host = "localhost:8384"
resp, err = http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Error("Explicit localhost:8384: expected 200 OK, not", resp.Status)
}
// A request with an explicit "localhost" Host header (no port) should pass
req, _ = http.NewRequest("GET", baseURL, nil)
req.Host = "localhost"
resp, err = http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Error("Explicit localhost: expected 200 OK, not", resp.Status)
}
// A server with InsecureSkipHostCheck set behaves differently
cfg = new(mockedConfig)
cfg.gui.RawAddress = "127.0.0.1:0"
cfg.gui.InsecureSkipHostCheck = true
baseURL, err = startHTTP(cfg)
if err != nil {
t.Fatal(err)
}
// A request with a suspicious Host header should be allowed
req, _ = http.NewRequest("GET", baseURL, nil)
req.Host = "example.com"
resp, err = http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Error("Incorrect host header, check disabled: expected 200 OK, not", resp.Status)
}
// A server bound to a wildcard address also doesn't do the check
cfg = new(mockedConfig)
cfg.gui.RawAddress = "0.0.0.0:0"
cfg.gui.InsecureSkipHostCheck = true
baseURL, err = startHTTP(cfg)
if err != nil {
t.Fatal(err)
}
// A request with a suspicious Host header should be allowed
req, _ = http.NewRequest("GET", baseURL, nil)
req.Host = "example.com"
resp, err = http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Error("Incorrect host header, wildcard bound: expected 200 OK, not", resp.Status)
}
// This should all work over IPv6 as well
cfg = new(mockedConfig)
cfg.gui.RawAddress = "[::1]:0"
baseURL, err = startHTTP(cfg)
if err != nil {
t.Fatal(err)
}
// A normal HTTP get to the localhost-bound service should succeed
resp, err = http.Get(baseURL)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Error("Regular HTTP get (IPv6): expected 200 OK, not", resp.Status)
}
// A request with a suspicious Host header should fail
req, _ = http.NewRequest("GET", baseURL, nil)
req.Host = "example.com"
resp, err = http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Error("Suspicious Host header (IPv6): expected 403 Forbidden, not", resp.Status)
}
// A request with an explicit "localhost:8384" Host header should pass
req, _ = http.NewRequest("GET", baseURL, nil)
req.Host = "localhost:8384"
resp, err = http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Error("Explicit localhost:8384 (IPv6): expected 200 OK, not", resp.Status)
}
}
func TestAddressIsLocalhost(t *testing.T) {
testcases := []struct {
address string
result bool
}{
// These are all valid localhost addresses
{"localhost", true},
{"LOCALHOST", true},
{"::1", true},
{"127.0.0.1", true},
{"localhost:8080", true},
{"LOCALHOST:8000", true},
{"[::1]:8080", true},
{"127.0.0.1:8080", true},
// These are all non-localhost addresses
{"example.com", false},
{"example.com:8080", false},
{"192.0.2.10", false},
{"192.0.2.10:8080", false},
{"0.0.0.0", false},
{"0.0.0.0:8080", false},
{"::", false},
{"[::]:8080", false},
{":8080", false},
}
for _, tc := range testcases {
result := addressIsLocalhost(tc.address)
if result != tc.result {
t.Errorf("addressIsLocalhost(%q)=%v, expected %v", tc.address, result, tc.result)
}
}
}
func TestAccessControlAllowOriginHeader(t *testing.T) {
const testAPIKey = "foobarbaz"
cfg := new(mockedConfig)
cfg.gui.APIKey = testAPIKey
baseURL, err := startHTTP(cfg)
if err != nil {
t.Fatal(err)
}
cli := &http.Client{
Timeout: time.Second,
}
req, _ := http.NewRequest("GET", baseURL+"/rest/system/status", nil)
req.Header.Set("X-API-Key", testAPIKey)
resp, err := cli.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatal("GET on /rest/system/status should succeed, not", resp.Status)
}
if resp.Header.Get("Access-Control-Allow-Origin") != "*" {
t.Fatal("GET on /rest/system/status should return a 'Access-Control-Allow-Origin: *' header")
}
}
func TestOptionsRequest(t *testing.T) {
const testAPIKey = "foobarbaz"
cfg := new(mockedConfig)
cfg.gui.APIKey = testAPIKey
baseURL, err := startHTTP(cfg)
if err != nil {
t.Fatal(err)
}
cli := &http.Client{
Timeout: time.Second,
}
req, _ := http.NewRequest("OPTIONS", baseURL+"/rest/system/status", nil)
resp, err := cli.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatal("OPTIONS on /rest/system/status should succeed, not", resp.Status)
}
if resp.Header.Get("Access-Control-Allow-Origin") != "*" {
t.Fatal("OPTIONS on /rest/system/status should return a 'Access-Control-Allow-Origin: *' header")
}
if resp.Header.Get("Access-Control-Allow-Methods") != "GET, POST" {
t.Fatal("OPTIONS on /rest/system/status should return a 'Access-Control-Allow-Methods: GET, POST' header")
}
if resp.Header.Get("Access-Control-Allow-Headers") != "Content-Type, X-API-Key" {
t.Fatal("OPTIONS on /rest/system/status should return a 'Access-Control-Allow-Headers: Content-Type, X-API-KEY' header")
}
}
func TestEventMasks(t *testing.T) {
cfg := new(mockedConfig)
defSub := new(mockedEventSub)
diskSub := new(mockedEventSub)
svc := newAPIService(protocol.LocalDeviceID, cfg, "", "", "", nil, defSub, diskSub, nil, nil, nil, nil, nil)
if mask := svc.getEventMask(""); mask != defaultEventMask {
t.Errorf("incorrect default mask %x != %x", int64(mask), int64(defaultEventMask))
}
expected := events.FolderSummary | events.LocalChangeDetected
if mask := svc.getEventMask("FolderSummary,LocalChangeDetected"); mask != expected {
t.Errorf("incorrect parsed mask %x != %x", int64(mask), int64(expected))
}
expected = 0
if mask := svc.getEventMask("WeirdEvent,something else that doesn't exist"); mask != expected {
t.Errorf("incorrect parsed mask %x != %x", int64(mask), int64(expected))
}
if res := svc.getEventSub(defaultEventMask); res != defSub {
t.Errorf("should have returned the given default event sub")
}
if res := svc.getEventSub(diskEventMask); res != diskSub {
t.Errorf("should have returned the given disk event sub")
}
if res := svc.getEventSub(events.LocalIndexUpdated); res == nil || res == defSub || res == diskSub {
t.Errorf("should have returned a valid, non-default event sub")
}
}

View File

@@ -1,37 +0,0 @@
// Copyright (C) 2014 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
//+build !windows,!solaris
package main
import (
"syscall"
"time"
)
func init() {
go trackCPUUsage()
}
func trackCPUUsage() {
var prevUsage int64
var prevTime = time.Now().UnixNano()
var rusage syscall.Rusage
for _ = range time.NewTicker(time.Second).C {
syscall.Getrusage(syscall.RUSAGE_SELF, &rusage)
curTime := time.Now().UnixNano()
timeDiff := curTime - prevTime
curUsage := rusage.Utime.Nano() + rusage.Stime.Nano()
usageDiff := curUsage - prevUsage
cpuUsageLock.Lock()
copy(cpuUsagePercent[1:], cpuUsagePercent[0:])
cpuUsagePercent[0] = 100 * float64(usageDiff) / float64(timeDiff)
cpuUsageLock.Unlock()
prevTime = curTime
prevUsage = curUsage
}
}

View File

@@ -1,58 +0,0 @@
// Copyright (C) 2014 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
//+build windows
package main
import (
"syscall"
"time"
)
func init() {
go trackCPUUsage()
}
func trackCPUUsage() {
handle, err := syscall.GetCurrentProcess()
if err != nil {
l.Warnln("Cannot track CPU usage:", err)
return
}
var ctime, etime, ktime, utime syscall.Filetime
err = syscall.GetProcessTimes(handle, &ctime, &etime, &ktime, &utime)
if err != nil {
l.Warnln("Cannot track CPU usage:", err)
return
}
prevTime := ctime.Nanoseconds()
prevUsage := ktime.Nanoseconds() + utime.Nanoseconds() // Always overflows
for _ = range time.NewTicker(time.Second).C {
err := syscall.GetProcessTimes(handle, &ctime, &etime, &ktime, &utime)
if err != nil {
continue
}
curTime := time.Now().UnixNano()
timeDiff := curTime - prevTime
// This is sometimes 0, no clue why.
if timeDiff == 0 {
continue
}
curUsage := ktime.Nanoseconds() + utime.Nanoseconds()
usageDiff := curUsage - prevUsage
cpuUsageLock.Lock()
copy(cpuUsagePercent[1:], cpuUsagePercent[0:])
cpuUsagePercent[0] = 100 * float64(usageDiff) / float64(timeDiff)
cpuUsageLock.Unlock()
prevTime = curTime
prevUsage = curUsage
}
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -13,7 +13,7 @@ import (
"strings"
"time"
"github.com/syncthing/syncthing/lib/osutil"
"github.com/syncthing/syncthing/lib/fs"
)
type locationEnum string
@@ -65,7 +65,7 @@ func expandLocations() error {
dir = strings.Replace(dir, "${"+varName+"}", value, -1)
}
var err error
dir, err = osutil.ExpandTilde(dir)
dir, err = fs.ExpandTilde(dir)
if err != nil {
return err
}
@@ -86,7 +86,7 @@ func defaultConfigDir() string {
return filepath.Join(os.Getenv("AppData"), "Syncthing")
case "darwin":
dir, err := osutil.ExpandTilde("~/Library/Application Support/Syncthing")
dir, err := fs.ExpandTilde("~/Library/Application Support/Syncthing")
if err != nil {
l.Fatalln(err)
}
@@ -96,7 +96,7 @@ func defaultConfigDir() string {
if xdgCfg := os.Getenv("XDG_CONFIG_HOME"); xdgCfg != "" {
return filepath.Join(xdgCfg, "syncthing")
}
dir, err := osutil.ExpandTilde("~/.config/syncthing")
dir, err := fs.ExpandTilde("~/.config/syncthing")
if err != nil {
l.Fatalln(err)
}
@@ -106,7 +106,7 @@ func defaultConfigDir() string {
// homeDir returns the user's home directory, or dies trying.
func homeDir() string {
home, err := osutil.ExpandTilde("~")
home, err := fs.ExpandTilde("~")
if err != nil {
l.Fatalln(err)
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -12,11 +12,11 @@ import (
"errors"
"flag"
"fmt"
"io"
"io/ioutil"
"log"
"net"
"net/http"
_ "net/http/pprof"
"net/url"
"os"
"os/signal"
@@ -37,16 +37,20 @@ import (
"github.com/syncthing/syncthing/lib/dialer"
"github.com/syncthing/syncthing/lib/discover"
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/fs"
"github.com/syncthing/syncthing/lib/logger"
"github.com/syncthing/syncthing/lib/model"
"github.com/syncthing/syncthing/lib/osutil"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/rand"
"github.com/syncthing/syncthing/lib/symlinks"
"github.com/syncthing/syncthing/lib/sha256"
"github.com/syncthing/syncthing/lib/tlsutil"
"github.com/syncthing/syncthing/lib/upgrade"
"github.com/syncthing/syncthing/lib/weakhash"
"github.com/thejerf/suture"
_ "net/http/pprof" // Need to import this to support STPROFILER.
)
var (
@@ -57,8 +61,10 @@ var (
BuildHost = "unknown"
BuildUser = "unknown"
IsRelease bool
IsCandidate bool
IsBeta bool
LongVersion string
BuildTags []string
allowedVersionExp = regexp.MustCompile(`^v\d+\.\d+\.\d+(-[a-z0-9]+)*(\.\d+)*(\+\d+-g[0-9a-f]+)?(-[^\s]+)?$`)
)
@@ -75,7 +81,7 @@ const (
tlsDefaultCommonName = "syncthing"
httpsRSABits = 2048
bepRSABits = 0 // 384 bit ECDSA used instead
pingEventInterval = time.Minute
defaultEventTimeout = time.Minute
maxSystemErrors = 5
initialSystemLog = 10
maxSystemLog = 250
@@ -95,15 +101,26 @@ func init() {
l.Fatalf("Invalid version string %q;\n\tdoes not match regexp %v", Version, allowedVersionExp)
}
}
}
// Check for a clean release build. A release is something like "v0.1.2",
// with an optional suffix of letters and dot separated numbers like
// "-beta3.47". If there's more stuff, like a plus sign and a commit hash
// and so on, then it's not a release. If there's a dash anywhere in
// there, it's some kind of beta or prerelease version.
func setBuildMetadata() {
// Check for a clean release build. A release is something like
// "v0.1.2", with an optional suffix of letters and dot separated
// numbers like "-beta3.47". If there's more stuff, like a plus sign and
// a commit hash and so on, then it's not a release. If it has a dash in
// it, it's some sort of beta, release candidate or special build. If it
// has "-rc." in it, like "v0.14.35-rc.42", then it's a candidate build.
//
// So, every build that is not a stable release build has IsBeta = true.
// This is used to enable some extra debugging (the deadlock detector).
//
// Release candidate builds are also "betas" from this point of view and
// will have that debugging enabled. In addition, some features are
// forced for release candidates - auto upgrade, and usage reporting.
exp := regexp.MustCompile(`^v\d+\.\d+\.\d+(-[a-z]+[\d\.]+)?$`)
IsRelease = exp.MatchString(Version)
IsCandidate = strings.Contains(Version, "-rc.")
IsBeta = strings.Contains(Version, "-")
stamp, _ := strconv.Atoi(BuildStamp)
@@ -111,6 +128,10 @@ func init() {
date := BuildDate.UTC().Format("2006-01-02 15:04:05 MST")
LongVersion = fmt.Sprintf(`syncthing %s "%s" (%s %s-%s) %s@%s %s`, Version, Codename, runtime.Version(), runtime.GOOS, runtime.GOARCH, BuildUser, BuildHost, date)
if len(BuildTags) > 0 {
LongVersion = fmt.Sprintf("%s [%s]", LongVersion, strings.Join(BuildTags, ", "))
}
}
var (
@@ -138,11 +159,11 @@ show time only (2).
Development Settings
--------------------
The following environment variables modify syncthing's behavior in ways that
The following environment variables modify Syncthing's behavior in ways that
are mostly useful for developers. Use with care.
STNODEFAULTFOLDER Don't create a default folder when starting for the first
time. This variable will be ignored anytime after the first
time. This variable will be ignored anytime after the first
run.
STGUIASSETS Directory to load GUI assets from. Overrides compiled in
@@ -151,8 +172,8 @@ are mostly useful for developers. Use with care.
STTRACE A comma separated string of facilities to trace. The valid
facility strings listed below.
STPROFILER Set to a listen address such as "127.0.0.1:9090" to start the
profiler with HTTP access.
STPROFILER Set to a listen address such as "127.0.0.1:9090" to start
the profiler with HTTP access.
STCPUPROFILE Write a CPU profile to cpu-$pid.pprof on exit.
@@ -165,14 +186,33 @@ are mostly useful for developers. Use with care.
STPERFSTATS Write running performance statistics to perf-$pid.csv. Not
supported on Windows.
STDEADLOCK Used for debugging internal deadlocks. Use only under
direction of a developer.
STDEADLOCKTIMEOUT Used for debugging internal deadlocks; sets debug
sensitivity. Use only under direction of a developer.
STDEADLOCKTHRESHOLD Used for debugging internal deadlocks; sets debug
sensitivity. Use only under direction of a developer.
STNORESTART Equivalent to the -no-restart argument. Disable the
Syncthing monitor process which handles restarts for some
configuration changes, upgrades, crashes and also log file
writing (stdout is still written).
STNOUPGRADE Disable automatic upgrades.
STHASHING Select the SHA256 hashing package to use. Possible values
are "standard" for the Go standard library implementation,
"minio" for the github.com/minio/sha256-simd implementation,
and blank (the default) for auto detection.
GOMAXPROCS Set the maximum number of CPU cores to use. Defaults to all
available CPU cores.
GOGC Percentage of heap growth at which to trigger GC. Default is
100. Lower numbers keep peak memory usage down, at the price
of CPU usage (ie. performance).
of CPU usage (i.e. performance).
Debugging Facilities
@@ -185,16 +225,18 @@ The following are valid values for the STTRACE variable:
// Environment options
var (
noUpgrade = os.Getenv("STNOUPGRADE") != ""
innerProcess = os.Getenv("STNORESTART") != "" || os.Getenv("STMONITORED") != ""
noDefaultFolder = os.Getenv("STNODEFAULTFOLDER") != ""
noUpgradeFromEnv = os.Getenv("STNOUPGRADE") != ""
innerProcess = os.Getenv("STNORESTART") != "" || os.Getenv("STMONITORED") != ""
noDefaultFolder = os.Getenv("STNODEFAULTFOLDER") != ""
)
type RuntimeOptions struct {
confDir string
reset bool
resetDatabase bool
resetDeltaIdxs bool
showVersion bool
showPaths bool
showDeviceId bool
doUpgrade bool
doUpgradeCheck bool
upgradeTo string
@@ -203,8 +245,10 @@ type RuntimeOptions struct {
hideConsole bool
logFile string
auditEnabled bool
auditFile string
verbose bool
paused bool
unpaused bool
guiAddress string
guiAPIKey string
generateDir string
@@ -227,7 +271,7 @@ func defaultRuntimeOptions() RuntimeOptions {
}
if os.Getenv("STTRACE") != "" {
options.logFlags = log.Ltime | log.Ldate | log.Lmicroseconds | log.Lshortfile
options.logFlags = logger.DebugFlags
}
if runtime.GOOS != "windows" {
@@ -251,17 +295,21 @@ func parseCommandLineOptions() RuntimeOptions {
flag.IntVar(&options.logFlags, "logflags", options.logFlags, "Select information in log line prefix (see below)")
flag.BoolVar(&options.noBrowser, "no-browser", false, "Do not start browser")
flag.BoolVar(&options.browserOnly, "browser-only", false, "Open GUI in browser")
flag.BoolVar(&options.noRestart, "no-restart", options.noRestart, "Do not restart; just exit")
flag.BoolVar(&options.reset, "reset", false, "Reset the database")
flag.BoolVar(&options.noRestart, "no-restart", options.noRestart, "Disable monitor process, managed restarts and log file writing")
flag.BoolVar(&options.resetDatabase, "reset-database", false, "Reset the database, forcing a full rescan and resync")
flag.BoolVar(&options.resetDeltaIdxs, "reset-deltas", false, "Reset delta index IDs, forcing a full index exchange")
flag.BoolVar(&options.doUpgrade, "upgrade", false, "Perform upgrade")
flag.BoolVar(&options.doUpgradeCheck, "upgrade-check", false, "Check for available upgrade")
flag.BoolVar(&options.showVersion, "version", false, "Show version")
flag.BoolVar(&options.showPaths, "paths", false, "Show configuration paths")
flag.BoolVar(&options.showDeviceId, "device-id", false, "Show the device ID")
flag.StringVar(&options.upgradeTo, "upgrade-to", options.upgradeTo, "Force upgrade directly from specified URL")
flag.BoolVar(&options.auditEnabled, "audit", false, "Write events to audit file")
flag.BoolVar(&options.verbose, "verbose", false, "Print verbose log output")
flag.BoolVar(&options.paused, "paused", false, "Start with all devices paused")
flag.BoolVar(&options.paused, "paused", false, "Start with all devices and folders paused")
flag.BoolVar(&options.unpaused, "unpaused", false, "Start with all devices and folders unpaused")
flag.StringVar(&options.logFile, "logfile", options.logFile, "Log file name (use \"-\" for stdout)")
flag.StringVar(&options.auditFile, "auditfile", options.auditFile, "Specify audit file (use \"-\" for stdout, \"--\" for stderr)")
if runtime.GOOS == "windows" {
// Allow user to hide the console window
flag.BoolVar(&options.hideConsole, "no-console", false, "Hide console window")
@@ -271,10 +319,17 @@ func parseCommandLineOptions() RuntimeOptions {
flag.Usage = usageFor(flag.CommandLine, usage, longUsage)
flag.Parse()
if len(flag.Args()) > 0 {
flag.Usage()
os.Exit(2)
}
return options
}
func main() {
setBuildMetadata()
options := parseCommandLineOptions()
l.SetFlags(options.logFlags)
@@ -287,12 +342,27 @@ func main() {
os.Setenv("STGUIAPIKEY", options.guiAPIKey)
}
// Check for options which are not compatible with each other. We have
// to check logfile before it's set to the default below - we only want
// to complain if they set -logfile explicitly, not if it's set to its
// default location
if options.noRestart && (options.logFile != "" && options.logFile != "-") {
l.Fatalln("-logfile may not be used with -no-restart or STNORESTART")
}
if options.hideConsole {
osutil.HideConsole()
}
if options.confDir != "" {
// Not set as default above because the string can be really long.
if !filepath.IsAbs(options.confDir) {
var err error
options.confDir, err = filepath.Abs(options.confDir)
if err != nil {
l.Fatalln(err)
}
}
baseDirs["config"] = options.confDir
}
@@ -322,6 +392,17 @@ func main() {
return
}
if options.showDeviceId {
cert, err := tls.LoadX509KeyPair(locations[locCertFile], locations[locKeyFile])
if err != nil {
l.Fatalln("Error reading device ID:", err)
}
myID = protocol.NewDeviceID(cert.Certificate[0])
fmt.Println(myID)
return
}
if options.browserOnly {
openGUI()
return
@@ -355,12 +436,12 @@ func main() {
return
}
if options.reset {
if options.resetDatabase {
resetDB()
return
}
if options.noRestart {
if innerProcess || options.noRestart {
syncthingMain(options)
} else {
monitorMain(options)
@@ -368,7 +449,7 @@ func main() {
}
func openGUI() {
cfg, _ := loadConfig()
cfg, _ := loadOrDefaultConfig()
if cfg.GUI().Enabled {
openURL(cfg.GUI().URL())
} else {
@@ -377,7 +458,7 @@ func openGUI() {
}
func generate(generateDir string) {
dir, err := osutil.ExpandTilde(generateDir)
dir, err := fs.ExpandTilde(generateDir)
if err != nil {
l.Fatalln("generate:", err)
}
@@ -407,9 +488,7 @@ func generate(generateDir string) {
l.Warnln("Config exists; will not overwrite.")
return
}
var myName, _ = os.Hostname()
var newCfg = defaultConfig(myName)
var cfg = config.Wrap(cfgFile, newCfg)
var cfg = defaultConfig(cfgFile)
err = cfg.Save()
if err != nil {
l.Warnln("Failed to save config", err)
@@ -439,9 +518,9 @@ func debugFacilities() string {
}
func checkUpgrade() upgrade.Release {
cfg, _ := loadConfig()
releasesURL := cfg.Options().ReleasesURL
release, err := upgrade.LatestRelease(releasesURL, Version)
cfg, _ := loadOrDefaultConfig()
opts := cfg.Options()
release, err := upgrade.LatestRelease(opts.ReleasesURL, Version, opts.UpgradeToPreReleases)
if err != nil {
l.Fatalln("Upgrade:", err)
}
@@ -477,7 +556,7 @@ func performUpgrade(release upgrade.Release) {
}
func upgradeViaRest() error {
cfg, _ := loadConfig()
cfg, _ := loadOrDefaultConfig()
u, err := url.Parse(cfg.GUI().URL())
if err != nil {
return err
@@ -529,7 +608,7 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
l.SetPrefix("[start] ")
if runtimeOptions.auditEnabled {
startAuditing(mainService)
startAuditing(mainService, runtimeOptions.auditFile)
}
if runtimeOptions.verbose {
@@ -542,7 +621,8 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
// Event subscription for the API; must start early to catch the early
// events. The LocalChangeDetected event might overwhelm the event
// receiver in some situations so we will not subscribe to it here.
apiSub := events.NewBufferedSubscription(events.Default.Subscribe(events.AllEvents&^events.LocalChangeDetected), 1000)
defaultSub := events.NewBufferedSubscription(events.Default.Subscribe(defaultEventMask), eventSubBufferSize)
diskSub := events.NewBufferedSubscription(events.Default.Subscribe(diskEventMask), eventSubBufferSize)
if len(os.Getenv("GOMAXPROCS")) == 0 {
runtime.GOMAXPROCS(runtime.NumCPU())
@@ -553,7 +633,7 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
// report the error if there is one.
osutil.MaximizeOpenFileLimit()
// Ensure that that we have a certificate and key.
// Ensure that we have a certificate and key.
cert, err := tls.LoadX509KeyPair(locations[locCertFile], locations[locKeyFile])
if err != nil {
l.Infof("Generating ECDSA key and certificate for %s...", tlsDefaultCommonName)
@@ -568,7 +648,11 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
l.Infoln(LongVersion)
l.Infoln("My ID:", myID)
printHashRate()
// Select SHA256 implementation and report. Affected by the
// STHASHING environment variable.
sha256.SelectAlgo()
sha256.Report()
// Emit the Starting event, now that we know who we are.
@@ -577,7 +661,7 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
"myID": myID.String(),
})
cfg := loadOrCreateConfig()
cfg := loadConfigAtStartup()
if err := checkShortIDs(cfg); err != nil {
l.Fatalln("Short device IDs are in conflict. Unlucky!\n Regenerate the device ID of one of the following:\n ", err)
@@ -605,6 +689,10 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
InsecureSkipVerify: true,
MinVersion: tls.VersionTLS12,
CipherSuites: []uint16{
0xCCA8, // TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, Go 1.8
0xCCA9, // TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, Go 1.8
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
@@ -614,13 +702,27 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
},
}
// If the read or write rate should be limited, set up a rate limiter for it.
// This will be used on connections created in the connect and listen routines.
opts := cfg.Options()
if !opts.SymlinksEnabled {
symlinks.Supported = false
if opts.WeakHashSelectionMethod == config.WeakHashAuto {
perfWithWeakHash := cpuBench(3, 150*time.Millisecond, true)
l.Infof("Hashing performance with weak hash is %.02f MB/s", perfWithWeakHash)
perfWithoutWeakHash := cpuBench(3, 150*time.Millisecond, false)
l.Infof("Hashing performance without weak hash is %.02f MB/s", perfWithoutWeakHash)
if perfWithoutWeakHash*0.8 > perfWithWeakHash {
l.Infof("Weak hash disabled, as it has an unacceptable performance impact.")
weakhash.Enabled = false
} else {
l.Infof("Weak hash enabled, as it has an acceptable performance impact.")
weakhash.Enabled = true
}
} else if opts.WeakHashSelectionMethod == config.WeakHashNever {
l.Infof("Disabling weak hash")
weakhash.Enabled = false
} else if opts.WeakHashSelectionMethod == config.WeakHashAlways {
l.Infof("Enabling weak hash")
weakhash.Enabled = true
}
if (opts.MaxRecvKbps > 0 || opts.MaxSendKbps > 0) && !opts.LimitBandwidthInLan {
@@ -648,6 +750,11 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
l.Fatalln("Cannot open database:", err, "- Is another copy of Syncthing already running?")
}
if runtimeOptions.resetDeltaIdxs {
l.Infoln("Reinitializing delta index IDs")
ldb.DropDeltaIndexIDs()
}
protectedFiles := []string{
locations[locDatabase],
locations[locConfigFile],
@@ -664,8 +771,18 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
}
}
m := model.NewModel(cfg, myID, myDeviceName(cfg), "syncthing", Version, ldb, protectedFiles)
cfg.Subscribe(m)
if cfg.RawCopy().OriginalVersion == 15 {
// The config version 15->16 migration is about handling ignores and
// delta indexes and requires that we drop existing indexes that
// have been incorrectly ignore filtered.
ldb.DropDeltaIndexIDs()
}
if cfg.RawCopy().OriginalVersion < 19 {
// Converts old symlink types to new in the entire database.
ldb.ConvertSymlinkTypes()
}
m := model.NewModel(cfg, myID, "syncthing", Version, ldb, protectedFiles)
if t := os.Getenv("STDEADLOCKTIMEOUT"); len(t) > 0 {
it, err := strconv.Atoi(t)
@@ -676,23 +793,19 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
m.StartDeadlockDetector(20 * time.Minute)
}
if runtimeOptions.paused {
for device := range cfg.Devices() {
m.PauseDevice(device)
}
if runtimeOptions.unpaused {
setPauseState(cfg, false)
} else if runtimeOptions.paused {
setPauseState(cfg, true)
}
// Clear out old indexes for other devices. Otherwise we'll start up and
// start needing a bunch of files which are nowhere to be found. This
// needs to be changed when we correctly do persistent indexes.
// Add and start folders
for _, folderCfg := range cfg.Folders() {
m.AddFolder(folderCfg)
for _, device := range folderCfg.DeviceIDs() {
if device == myID {
continue
}
m.Index(device, folderCfg.ID, nil)
if folderCfg.Paused {
folderCfg.CreateRoot()
continue
}
m.AddFolder(folderCfg)
m.StartFolder(folderCfg.ID)
}
@@ -743,7 +856,7 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
// GUI
setupGUI(mainService, cfg, m, apiSub, cachedDiscovery, connectionsService, errors, systemLog, runtimeOptions)
setupGUI(mainService, cfg, m, defaultSub, diskSub, cachedDiscovery, connectionsService, errors, systemLog, runtimeOptions)
if runtimeOptions.cpuProfile {
f, err := os.Create(fmt.Sprintf("cpu-%d.pprof", os.Getpid()))
@@ -759,43 +872,59 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
}
}
if opts.URAccepted > 0 && opts.URAccepted < usageReportVersion {
l.Infoln("Anonymous usage report has changed; revoking acceptance")
opts.URAccepted = 0
opts.URUniqueID = ""
cfg.SetOptions(opts)
}
if opts.URAccepted >= usageReportVersion {
if opts.URUniqueID == "" {
// Previously the ID was generated from the node ID. We now need
// to generate a new one.
opts.URUniqueID = rand.String(8)
cfg.SetOptions(opts)
cfg.Save()
}
// Candidate builds always run with usage reporting.
if IsCandidate {
l.Infoln("Anonymous usage reporting is always enabled for candidate releases.")
opts.URAccepted = usageReportVersion
// Unique ID will be set and config saved below if necessary.
}
// The usageReportingManager registers itself to listen to configuration
// changes, and there's nothing more we need to tell it from the outside.
// Hence we don't keep the returned pointer.
newUsageReportingManager(cfg, m)
if opts.URUniqueID == "" {
opts.URUniqueID = rand.String(8)
cfg.SetOptions(opts)
cfg.Save()
}
usageReportingSvc := newUsageReportingService(cfg, m, connectionsService)
mainService.Add(usageReportingSvc)
if opts.RestartOnWakeup {
go standbyMonitor()
}
// Candidate builds should auto upgrade. Make sure the option is set,
// unless we are in a build where it's disabled or the STNOUPGRADE
// environment variable is set.
if IsCandidate && !upgrade.DisabledByCompilation && !noUpgradeFromEnv {
l.Infoln("Automatic upgrade is always enabled for candidate releases.")
if opts.AutoUpgradeIntervalH == 0 || opts.AutoUpgradeIntervalH > 24 {
opts.AutoUpgradeIntervalH = 12
// Set the option into the config as well, as the auto upgrade
// loop expects to read a valid interval from there.
cfg.SetOptions(opts)
cfg.Save()
}
// We don't tweak the user's choice of upgrading to pre-releases or
// not, as otherwise they cannot step off the candidate channel.
}
if opts.AutoUpgradeIntervalH > 0 {
if noUpgrade {
if noUpgradeFromEnv {
l.Infof("No automatic upgrades; STNOUPGRADE environment variable defined.")
} else {
go autoUpgrade(cfg)
}
}
if isSuperUser() {
l.Warnln("Syncthing should not run as a privileged or system user. Please consider using a normal user account.")
}
events.Default.Log(events.StartupComplete, map[string]string{
"myID": myID.String(),
})
go generatePingEvents()
cleanConfigDirectory()
@@ -812,15 +941,6 @@ func syncthingMain(runtimeOptions RuntimeOptions) {
os.Exit(code)
}
func myDeviceName(cfg *config.Wrapper) string {
devices := cfg.Devices()
myName := devices[myID].Name
if myName == "" {
myName, _ = os.Hostname()
}
return myName
}
func setupSignalHandling() {
// Exit cleanly with "restarting" code on SIGHUP.
@@ -843,45 +963,31 @@ func setupSignalHandling() {
}()
}
// printHashRate prints the hashing performance in MB/s, formatting it with
// appropriate precision for the value, i.e. 182 MB/s, 18 MB/s, 1.8 MB/s, 0.18
// MB/s.
func printHashRate() {
hashRate := cpuBench(3, 100*time.Millisecond)
decimals := 0
if hashRate < 1 {
decimals = 2
} else if hashRate < 10 {
decimals = 1
}
l.Infof("Single thread hash performance is ~%.*f MB/s", decimals, hashRate)
}
func loadConfig() (*config.Wrapper, error) {
func loadOrDefaultConfig() (*config.Wrapper, error) {
cfgFile := locations[locConfigFile]
cfg, err := config.Load(cfgFile, myID)
if err != nil {
myName, _ := os.Hostname()
newCfg := defaultConfig(myName)
cfg = config.Wrap(cfgFile, newCfg)
cfg = defaultConfig(cfgFile)
}
return cfg, err
}
func loadOrCreateConfig() *config.Wrapper {
cfg, err := loadConfig()
func loadConfigAtStartup() *config.Wrapper {
cfgFile := locations[locConfigFile]
cfg, err := config.Load(cfgFile, myID)
if os.IsNotExist(err) {
cfg = defaultConfig(cfgFile)
cfg.Save()
l.Infof("Defaults saved. Edit %s to taste or use the GUI\n", cfg.ConfigPath())
l.Infof("Default config saved. Edit %s to taste or use the GUI\n", cfg.ConfigPath())
} else if err == io.EOF {
l.Fatalln("Failed to load config: unexpected end of file. Truncated or empty configuration?")
} else if err != nil {
l.Fatalln("Config:", err)
l.Fatalln("Failed to load config:", err)
}
if cfg.Raw().OriginalVersion != config.CurrentVersion {
if cfg.RawCopy().OriginalVersion != config.CurrentVersion {
err = archiveAndSaveConfig(cfg)
if err != nil {
l.Fatalln("Config archive:", err)
@@ -892,24 +998,57 @@ func loadOrCreateConfig() *config.Wrapper {
}
func archiveAndSaveConfig(cfg *config.Wrapper) error {
// To prevent previous config from being cleaned up, quickly touch it too
now := time.Now()
_ = os.Chtimes(cfg.ConfigPath(), now, now) // May return error on Android etc; no worries
archivePath := cfg.ConfigPath() + fmt.Sprintf(".v%d", cfg.Raw().OriginalVersion)
// Copy the existing config to an archive copy
archivePath := cfg.ConfigPath() + fmt.Sprintf(".v%d", cfg.RawCopy().OriginalVersion)
l.Infoln("Archiving a copy of old config file format at:", archivePath)
if err := osutil.Rename(cfg.ConfigPath(), archivePath); err != nil {
if err := copyFile(cfg.ConfigPath(), archivePath); err != nil {
return err
}
// Do a regular atomic config sve
return cfg.Save()
}
func startAuditing(mainService *suture.Supervisor) {
auditFile := timestampedLoc(locAuditLog)
fd, err := os.OpenFile(auditFile, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
func copyFile(src, dst string) error {
bs, err := ioutil.ReadFile(src)
if err != nil {
l.Fatalln("Audit:", err)
return err
}
if err := ioutil.WriteFile(dst, bs, 0600); err != nil {
// Attempt to clean up
os.Remove(dst)
return err
}
return nil
}
func startAuditing(mainService *suture.Supervisor, auditFile string) {
var fd io.Writer
var err error
var auditDest string
var auditFlags int
if auditFile == "-" {
fd = os.Stdout
auditDest = "stdout"
} else if auditFile == "--" {
fd = os.Stderr
auditDest = "stderr"
} else {
if auditFile == "" {
auditFile = timestampedLoc(locAuditLog)
auditFlags = os.O_WRONLY | os.O_CREATE | os.O_EXCL
} else {
auditFlags = os.O_WRONLY | os.O_CREATE | os.O_APPEND
}
fd, err = os.OpenFile(auditFile, auditFlags, 0600)
if err != nil {
l.Fatalln("Audit:", err)
}
auditDest = auditFile
}
auditService := newAuditService(fd)
@@ -919,10 +1058,10 @@ func startAuditing(mainService *suture.Supervisor) {
// ensure we capture all events from the start.
auditService.WaitForStart()
l.Infoln("Audit log in", auditFile)
l.Infoln("Audit log in", auditDest)
}
func setupGUI(mainService *suture.Supervisor, cfg *config.Wrapper, m *model.Model, apiSub events.BufferedSubscription, discoverer discover.CachingMux, connectionsService *connections.Service, errors, systemLog logger.Recorder, runtimeOptions RuntimeOptions) {
func setupGUI(mainService *suture.Supervisor, cfg *config.Wrapper, m *model.Model, defaultSub, diskSub events.BufferedSubscription, discoverer discover.CachingMux, connectionsService *connections.Service, errors, systemLog logger.Recorder, runtimeOptions RuntimeOptions) {
guiCfg := cfg.GUI()
if !guiCfg.Enabled {
@@ -933,27 +1072,33 @@ func setupGUI(mainService *suture.Supervisor, cfg *config.Wrapper, m *model.Mode
l.Warnln("Insecure admin access is enabled.")
}
api := newAPIService(myID, cfg, locations[locHTTPSCertFile], locations[locHTTPSKeyFile], runtimeOptions.assetDir, m, apiSub, discoverer, connectionsService, errors, systemLog)
cpu := newCPUService()
mainService.Add(cpu)
api := newAPIService(myID, cfg, locations[locHTTPSCertFile], locations[locHTTPSKeyFile], runtimeOptions.assetDir, m, defaultSub, diskSub, discoverer, connectionsService, errors, systemLog, cpu)
cfg.Subscribe(api)
mainService.Add(api)
if cfg.Options().StartBrowser && !runtimeOptions.noBrowser && !runtimeOptions.stRestarting {
// Can potentially block if the utility we are invoking doesn't
// fork, and just execs, hence keep it in it's own routine.
// fork, and just execs, hence keep it in its own routine.
<-api.startedOnce
go openURL(guiCfg.URL())
}
}
func defaultConfig(myName string) config.Configuration {
func defaultConfig(cfgFile string) *config.Wrapper {
myName, _ := os.Hostname()
var defaultFolder config.FolderConfiguration
if !noDefaultFolder {
l.Infoln("Default folder created and/or linked to new config")
folderID := rand.String(5) + "-" + rand.String(5)
defaultFolder = config.NewFolderConfiguration(folderID, locations[locDefFolder])
defaultFolder.Label = "Default Folder (" + folderID + ")"
defaultFolder = config.NewFolderConfiguration("default", fs.FilesystemTypeBasic, locations[locDefFolder])
defaultFolder.Label = "Default Folder"
defaultFolder.RescanIntervalS = 60
defaultFolder.MinDiskFreePct = 1
defaultFolder.FSWatcherDelayS = 10
defaultFolder.MinDiskFree = config.Size{Value: 1, Unit: "%"}
defaultFolder.Devices = []config.FolderDeviceConfiguration{{DeviceID: myID}}
defaultFolder.AutoNormalize = true
defaultFolder.MaxConflicts = -1
@@ -989,14 +1134,7 @@ func defaultConfig(myName string) config.Configuration {
}
}
return newCfg
}
func generatePingEvents() {
for {
time.Sleep(pingEventInterval)
events.Default.Log(events.Ping, nil)
}
return config.Wrap(cfgFile, newCfg)
}
func resetDB() error {
@@ -1013,19 +1151,20 @@ func shutdown() {
stop <- exitSuccess
}
func ensureDir(dir string, mode os.FileMode) {
err := osutil.MkdirAll(dir, mode)
func ensureDir(dir string, mode fs.FileMode) {
fs := fs.NewFilesystem(fs.FilesystemTypeBasic, dir)
err := fs.MkdirAll(".", mode)
if err != nil {
l.Fatalln(err)
}
if fi, err := os.Stat(dir); err == nil {
if fi, err := fs.Stat("."); err == nil {
// Apprently the stat may fail even though the mkdirall passed. If it
// does, we'll just assume things are in order and let other things
// fail (like loading or creating the config...).
currentMode := fi.Mode() & 0777
if currentMode != mode {
err := os.Chmod(dir, mode)
err := fs.Chmod(".", mode)
// This can fail on crappy filesystems, nothing we can do about it.
if err != nil {
l.Warnln(err)
@@ -1056,7 +1195,7 @@ func getFreePort(host string, ports ...int) (int, error) {
}
func standbyMonitor() {
restartDelay := time.Duration(60 * time.Second)
restartDelay := 60 * time.Second
now := time.Now()
for {
time.Sleep(10 * time.Second)
@@ -1089,7 +1228,15 @@ func autoUpgrade(cfg *config.Wrapper) {
case <-timer.C:
}
rel, err := upgrade.LatestRelease(cfg.Options().ReleasesURL, Version)
opts := cfg.Options()
checkInterval := time.Duration(opts.AutoUpgradeIntervalH) * time.Hour
if checkInterval < time.Hour {
// We shouldn't be here if AutoUpgradeIntervalH < 1, but for
// safety's sake.
checkInterval = time.Hour
}
rel, err := upgrade.LatestRelease(opts.ReleasesURL, Version, opts.UpgradeToPreReleases)
if err == upgrade.ErrUpgradeUnsupported {
events.Default.Unsubscribe(sub)
return
@@ -1098,13 +1245,13 @@ func autoUpgrade(cfg *config.Wrapper) {
// Don't complain too loudly here; we might simply not have
// internet connectivity, or the upgrade server might be down.
l.Infoln("Automatic upgrade:", err)
timer.Reset(time.Duration(cfg.Options().AutoUpgradeIntervalH) * time.Hour)
timer.Reset(checkInterval)
continue
}
if upgrade.CompareVersions(rel.Tag, Version) != upgrade.Newer {
// Skip equal, older or majorly newer (incompatible) versions
timer.Reset(time.Duration(cfg.Options().AutoUpgradeIntervalH) * time.Hour)
timer.Reset(checkInterval)
continue
}
@@ -1112,7 +1259,7 @@ func autoUpgrade(cfg *config.Wrapper) {
err = upgrade.To(rel)
if err != nil {
l.Warnln("Automatic upgrade:", err)
timer.Reset(time.Duration(cfg.Options().AutoUpgradeIntervalH) * time.Hour)
timer.Reset(checkInterval)
continue
}
events.Default.Unsubscribe(sub)
@@ -1127,32 +1274,35 @@ func autoUpgrade(cfg *config.Wrapper) {
// suitable time after they have gone out of fashion.
func cleanConfigDirectory() {
patterns := map[string]time.Duration{
"panic-*.log": 7 * 24 * time.Hour, // keep panic logs for a week
"audit-*.log": 7 * 24 * time.Hour, // keep audit logs for a week
"index": 14 * 24 * time.Hour, // keep old index format for two weeks
"index*.converted": 14 * 24 * time.Hour, // keep old converted indexes for two weeks
"config.xml.v*": 30 * 24 * time.Hour, // old config versions for a month
"*.idx.gz": 30 * 24 * time.Hour, // these should for sure no longer exist
"backup-of-v0.8": 30 * 24 * time.Hour, // these neither
"panic-*.log": 7 * 24 * time.Hour, // keep panic logs for a week
"audit-*.log": 7 * 24 * time.Hour, // keep audit logs for a week
"index": 14 * 24 * time.Hour, // keep old index format for two weeks
"index-v0.11.0.db": 14 * 24 * time.Hour, // keep old index format for two weeks
"index-v0.13.0.db": 14 * 24 * time.Hour, // keep old index format for two weeks
"index*.converted": 14 * 24 * time.Hour, // keep old converted indexes for two weeks
"config.xml.v*": 30 * 24 * time.Hour, // old config versions for a month
"*.idx.gz": 30 * 24 * time.Hour, // these should for sure no longer exist
"backup-of-v0.8": 30 * 24 * time.Hour, // these neither
"tmp-index-sorter.*": time.Minute, // these should never exist on startup
}
for pat, dur := range patterns {
pat = filepath.Join(baseDirs["config"], pat)
files, err := osutil.Glob(pat)
fs := fs.NewFilesystem(fs.FilesystemTypeBasic, baseDirs["config"])
files, err := fs.Glob(pat)
if err != nil {
l.Infoln("Cleaning:", err)
continue
}
for _, file := range files {
info, err := osutil.Lstat(file)
info, err := fs.Lstat(file)
if err != nil {
l.Infoln("Cleaning:", err)
continue
}
if time.Since(info.ModTime()) > dur {
if err = os.RemoveAll(file); err != nil {
if err = fs.RemoveAll(file); err != nil {
l.Infoln("Cleaning:", err)
} else {
l.Infoln("Cleaned away old file", filepath.Base(file))
@@ -1186,3 +1336,16 @@ func showPaths() {
fmt.Printf("GUI override directory:\n\t%s\n\n", locations[locGUIAssets])
fmt.Printf("Default sync folder directory:\n\t%s\n\n", locations[locDefFolder])
}
func setPauseState(cfg *config.Wrapper, paused bool) {
raw := cfg.RawCopy()
for i := range raw.Devices {
raw.Devices[i].Paused = paused
}
for i := range raw.Folders {
raw.Folders[i].Paused = paused
}
if err := cfg.Replace(raw); err != nil {
l.Fatalln("Cannot adjust paused state:", err)
}
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
// +build solaris

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
// +build freebsd openbsd dragonfly

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -20,9 +20,8 @@ var (
func memorySize() (int64, error) {
var memoryStatusEx [64]byte
binary.LittleEndian.PutUint32(memoryStatusEx[:], 64)
p := uintptr(unsafe.Pointer(&memoryStatusEx[0]))
ret, _, callErr := syscall.Syscall(uintptr(globalMemoryStatusEx), 1, p, 0, 0)
ret, _, callErr := syscall.Syscall(uintptr(globalMemoryStatusEx), 1, uintptr(unsafe.Pointer(&memoryStatusEx[0])), 0, 0)
if ret == 0 {
return 0, callErr
}

View File

@@ -2,13 +2,14 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/util"
)
type mockedConfig struct {
@@ -23,8 +24,10 @@ func (c *mockedConfig) ListenAddresses() []string {
return nil
}
func (c *mockedConfig) Raw() config.Configuration {
return config.Configuration{}
func (c *mockedConfig) RawCopy() config.Configuration {
cfg := config.Configuration{}
util.SetDefaults(&cfg.Options)
return cfg
}
func (c *mockedConfig) Options() config.OptionsConfiguration {
@@ -45,6 +48,14 @@ func (c *mockedConfig) Devices() map[protocol.DeviceID]config.DeviceConfiguratio
return nil
}
func (c *mockedConfig) SetDevice(config.DeviceConfiguration) error {
return nil
}
func (c *mockedConfig) SetDevices([]config.DeviceConfiguration) error {
return nil
}
func (c *mockedConfig) Save() error {
return nil
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -11,3 +11,7 @@ type mockedConnections struct{}
func (m *mockedConnections) Status() map[string]interface{} {
return nil
}
func (m *mockedConnections) NATType() string {
return ""
}

View File

@@ -0,0 +1,13 @@
// Copyright (C) 2017 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
type mockedCPUService struct{}
func (*mockedCPUService) Rate() float64 {
return 42
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,14 +2,18 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import "github.com/syncthing/syncthing/lib/events"
import (
"time"
"github.com/syncthing/syncthing/lib/events"
)
type mockedEventSub struct{}
func (s *mockedEventSub) Since(id int, into []events.Event) []events.Event {
func (s *mockedEventSub) Since(id int, into []events.Event, timeout time.Duration) []events.Event {
select {}
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -21,8 +21,8 @@ func (m *mockedModel) GlobalDirectoryTree(folder, prefix string, levels int, dir
return nil
}
func (m *mockedModel) Completion(device protocol.DeviceID, folder string) float64 {
return 0
func (m *mockedModel) Completion(device protocol.DeviceID, folder string) model.FolderCompletion {
return model.FolderCompletion{}
}
func (m *mockedModel) Override(folder string) {}
@@ -31,8 +31,8 @@ func (m *mockedModel) NeedFolderFiles(folder string, page, perpage int) ([]db.Fi
return nil, nil, nil, 0
}
func (m *mockedModel) NeedSize(folder string) (nfiles int, bytes int64) {
return 0, 0
func (m *mockedModel) NeedSize(folder string) db.Counts {
return db.Counts{}
}
func (m *mockedModel) ConnectionStats() map[string]interface{} {
@@ -95,22 +95,26 @@ func (m *mockedModel) ConnectedTo(deviceID protocol.DeviceID) bool {
return false
}
func (m *mockedModel) GlobalSize(folder string) (nfiles, deleted int, bytes int64) {
return 0, 0, 0
func (m *mockedModel) GlobalSize(folder string) db.Counts {
return db.Counts{}
}
func (m *mockedModel) LocalSize(folder string) (nfiles, deleted int, bytes int64) {
return 0, 0, 0
func (m *mockedModel) LocalSize(folder string) db.Counts {
return db.Counts{}
}
func (m *mockedModel) CurrentLocalVersion(folder string) (int64, bool) {
func (m *mockedModel) CurrentSequence(folder string) (int64, bool) {
return 0, false
}
func (m *mockedModel) RemoteLocalVersion(folder string) (int64, bool) {
func (m *mockedModel) RemoteSequence(folder string) (int64, bool) {
return 0, false
}
func (m *mockedModel) State(folder string) (string, time.Time, error) {
return "", time.Time{}, nil
}
func (m *mockedModel) UsageReportingStats(version int) map[string]interface{} {
return nil
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -35,8 +35,6 @@ const (
)
func monitorMain(runtimeOptions RuntimeOptions) {
os.Setenv("STNORESTART", "yes")
os.Setenv("STMONITORED", "yes")
l.SetPrefix("[monitor] ")
var dst io.Writer = os.Stdout
@@ -70,6 +68,8 @@ func monitorMain(runtimeOptions RuntimeOptions) {
sigHup := syscall.Signal(1)
signal.Notify(restartSign, sigHup)
childEnv := childEnv()
first := true
for {
if t := time.Since(restarts[0]); t < loopThreshold {
l.Warnf("%d restarts in %v; not retrying further", countRestarts, t)
@@ -80,6 +80,7 @@ func monitorMain(runtimeOptions RuntimeOptions) {
restarts[len(restarts)-1] = time.Now()
cmd := exec.Command(args[0], args[1:]...)
cmd.Env = childEnv
stderr, err := cmd.StderrPipe()
if err != nil {
@@ -97,10 +98,6 @@ func monitorMain(runtimeOptions RuntimeOptions) {
l.Fatalln(err)
}
// Let the next child process know that this is not the first time
// it's starting up.
os.Setenv("STRESTART", "yes")
stdoutMut.Lock()
stdoutFirstLines = make([]string, 0, 10)
stdoutLastLines = make([]string, 0, 50)
@@ -150,7 +147,6 @@ func monitorMain(runtimeOptions RuntimeOptions) {
// Restart the monitor process to release the .old
// binary as part of the upgrade process.
l.Infoln("Restarting monitor...")
os.Setenv("STNORESTART", "")
if err = restartMonitor(args); err != nil {
l.Warnln("Restart:", err)
}
@@ -162,6 +158,13 @@ func monitorMain(runtimeOptions RuntimeOptions) {
l.Infoln("Syncthing exited:", err)
time.Sleep(1 * time.Second)
if first {
// Let the next child process know that this is not the first time
// it's starting up.
childEnv = append(childEnv, "STRESTART=yes")
first = false
}
}
}
@@ -178,6 +181,22 @@ func copyStderr(stderr io.Reader, dst io.Writer) {
if panicFd == nil {
dst.Write([]byte(line))
if strings.Contains(line, "SIGILL") {
l.Warnln(`
*******************************************************************************
* Crash due to illegal instruction detected. This is most likely due to a CPU *
* incompatibility with the high performance hashing package. Switching to the *
* standard hashing package instead. Please report this issue at: *
* *
* https://github.com/syncthing/syncthing/issues *
* *
* Include the details of your CPU. *
*******************************************************************************
`)
os.Setenv("STHASHING", "standard")
return
}
if strings.HasPrefix(line, "panic:") || strings.HasPrefix(line, "fatal error:") {
panicFd, err = os.Create(timestampedLoc(locPanicLog))
if err != nil {
@@ -186,8 +205,24 @@ func copyStderr(stderr io.Reader, dst io.Writer) {
}
l.Warnf("Panic detected, writing to \"%s\"", panicFd.Name())
l.Warnln("Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/")
l.Warnln("If no issue with similar panic message exists, please create a new issue with the panic log attached")
if strings.Contains(line, "leveldb") && strings.Contains(line, "corrupt") {
l.Warnln(`
*********************************************************************************
* Crash due to corrupt database. *
* *
* This crash usually occurs due to one of the following reasons: *
* - Syncthing being stopped abruptly (killed/loss of power) *
* - Bad hardware (memory/disk issues) *
* - Software that affects disk writes (SSD caching software and simillar) *
* *
* Please see the following URL for instructions on how to recover: *
* https://docs.syncthing.net/users/faq.html#my-syncthing-database-is-corrupt *
*********************************************************************************
`)
} else {
l.Warnln("Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/")
l.Warnln("If no issue with similar panic message exists, please create a new issue with the panic log attached")
}
stdoutMut.Lock()
for _, line := range stdoutFirstLines {
@@ -378,3 +413,19 @@ func (f *autoclosedFile) closerLoop() {
}
}
}
// Returns the desired child environment, properly filtered and added to.
func childEnv() []string {
var env []string
for _, str := range os.Environ() {
if strings.HasPrefix("STNORESTART=", str) {
continue
}
if strings.HasPrefix("STMONITORED=", str) {
continue
}
env = append(env, str)
}
env = append(env, "STMONITORED=yes")
return env
}

View File

@@ -0,0 +1,88 @@
// Copyright (C) 2017 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"io/ioutil"
"os"
"path/filepath"
"testing"
"time"
)
func TestAutoClosedFile(t *testing.T) {
os.RemoveAll("_autoclose")
defer os.RemoveAll("_autoclose")
os.Mkdir("_autoclose", 0755)
file := filepath.FromSlash("_autoclose/tmp")
data := []byte("hello, world\n")
// An autoclosed file that closes very quickly
ac := newAutoclosedFile(file, time.Millisecond, time.Millisecond)
// Write some data.
if _, err := ac.Write(data); err != nil {
t.Fatal(err)
}
// Wait for it to close
start := time.Now()
for {
time.Sleep(time.Millisecond)
ac.mut.Lock()
fd := ac.fd
ac.mut.Unlock()
if fd == nil {
break
}
if time.Since(start) > time.Second {
t.Fatal("File should have been closed after first write")
}
}
// Write more data, which should be an append.
if _, err := ac.Write(data); err != nil {
t.Fatal(err)
}
// Close.
if err := ac.Close(); err != nil {
t.Fatal(err)
}
// The file should have both writes in it.
bs, err := ioutil.ReadFile(file)
if err != nil {
t.Fatal(err)
}
if len(bs) != 2*len(data) {
t.Fatalf("Writes failed, expected %d bytes, not %d", 2*len(data), len(bs))
}
// Open the file again.
ac = newAutoclosedFile(file, time.Second, time.Second)
// Write something
if _, err := ac.Write(data); err != nil {
t.Fatal(err)
}
// It should now contain only one write, because the first open
// should be a truncate.
bs, err = ioutil.ReadFile(file)
if err != nil {
t.Fatal(err)
}
if len(bs) != len(data) {
t.Fatalf("Write failed, expected %d bytes, not %d", len(data), len(bs))
}
// Close.
if err := ac.Close(); err != nil {
t.Fatal(err)
}
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
// +build !windows

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
// +build windows

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
// +build !solaris,!windows

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -10,6 +10,7 @@ import (
"time"
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/sync"
"github.com/thejerf/suture"
)
@@ -59,7 +60,7 @@ func (c *folderSummaryService) Stop() {
// listenForUpdates subscribes to the event bus and makes note of folders that
// need their data recalculated.
func (c *folderSummaryService) listenForUpdates() {
sub := events.Default.Subscribe(events.LocalIndexUpdated | events.RemoteIndexUpdated | events.StateChanged | events.RemoteDownloadProgress)
sub := events.Default.Subscribe(events.LocalIndexUpdated | events.RemoteIndexUpdated | events.StateChanged | events.RemoteDownloadProgress | events.DeviceConnected)
defer events.Default.Unsubscribe(sub)
for {
@@ -67,8 +68,31 @@ func (c *folderSummaryService) listenForUpdates() {
select {
case ev := <-sub.C():
// Whenever the local or remote index is updated for a given
// folder we make a note of it.
if ev.Type == events.DeviceConnected {
// When a device connects we schedule a refresh of all
// folders shared with that device.
data := ev.Data.(map[string]string)
deviceID, _ := protocol.DeviceIDFromString(data["id"])
c.foldersMut.Lock()
nextFolder:
for _, folder := range c.cfg.Folders() {
for _, dev := range folder.Devices {
if dev.DeviceID == deviceID {
c.folders[folder.ID] = struct{}{}
continue nextFolder
}
}
}
c.foldersMut.Unlock()
continue
}
// The other events all have a "folder" attribute that they
// affect. Whenever the local or remote index is updated for a
// given folder we make a note of it.
data := ev.Data.(map[string]interface{})
folder := data["folder"].(string)
@@ -145,7 +169,7 @@ func (c *folderSummaryService) foldersToHandle() []string {
c.lastEventReqMut.Lock()
last := c.lastEventReq
c.lastEventReqMut.Unlock()
if time.Since(last) > pingEventInterval {
if time.Since(last) > defaultEventTimeout {
return nil
}
@@ -183,9 +207,11 @@ func (c *folderSummaryService) sendSummary(folder string) {
// remote device.
comp := c.model.Completion(devCfg.DeviceID, folder)
events.Default.Log(events.FolderCompletion, map[string]interface{}{
"folder": folder,
"device": devCfg.DeviceID.String(),
"completion": comp,
"folder": folder,
"device": devCfg.DeviceID.String(),
"completion": comp.CompletionPct,
"needBytes": comp.NeedBytes,
"globalBytes": comp.GlobalBytes,
})
}
}

View File

@@ -0,0 +1,17 @@
// Copyright (C) 2017 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
// +build !windows
package main
import (
"os"
)
func isSuperUser() bool {
return os.Geteuid() == 0
}

View File

@@ -0,0 +1,41 @@
// Copyright (C) 2017 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import "syscall"
// https://msdn.microsoft.com/en-us/library/windows/desktop/aa379649(v=vs.85).aspx
const securityLocalSystemRID = "S-1-5-18"
func isSuperUser() bool {
tok, err := syscall.OpenCurrentProcessToken()
if err != nil {
l.Debugln("OpenCurrentProcessToken:", err)
return false
}
defer tok.Close()
user, err := tok.GetTokenUser()
if err != nil {
l.Debugln("GetTokenUser:", err)
return false
}
if user.User.Sid == nil {
l.Debugln("sid is nil")
return false
}
sid, err := user.User.Sid.String()
if err != nil {
l.Debugln("Sid.String():", err)
return false
}
l.Debugf("SID: %q", sid)
return sid == securityLocalSystemRID
}

View File

@@ -0,0 +1,16 @@
// Copyright (C) 2016 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
//+build go1.7
package main
import "runtime/debug"
func init() {
// We want all (our) goroutines in panic traces.
debug.SetTraceback("all")
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main

View File

@@ -2,17 +2,17 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"context"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"encoding/json"
"fmt"
"net"
"net/http"
"runtime"
"sort"
@@ -20,70 +20,26 @@ import (
"time"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/connections"
"github.com/syncthing/syncthing/lib/dialer"
"github.com/syncthing/syncthing/lib/model"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/scanner"
"github.com/syncthing/syncthing/lib/upgrade"
"github.com/thejerf/suture"
)
// Current version number of the usage report, for acceptance purposes. If
// fields are added or changed this integer must be incremented so that users
// are prompted for acceptance of the new report.
const usageReportVersion = 2
type usageReportingManager struct {
cfg *config.Wrapper
model *model.Model
sup *suture.Supervisor
}
func newUsageReportingManager(cfg *config.Wrapper, m *model.Model) *usageReportingManager {
mgr := &usageReportingManager{
cfg: cfg,
model: m,
}
// Start UR if it's enabled.
mgr.CommitConfiguration(config.Configuration{}, cfg.Raw())
// Listen to future config changes so that we can start and stop as
// appropriate.
cfg.Subscribe(mgr)
return mgr
}
func (m *usageReportingManager) VerifyConfiguration(from, to config.Configuration) error {
return nil
}
func (m *usageReportingManager) CommitConfiguration(from, to config.Configuration) bool {
if to.Options.URAccepted >= usageReportVersion && m.sup == nil {
// Usage reporting was turned on; lets start it.
service := newUsageReportingService(m.cfg, m.model)
m.sup = suture.NewSimple("usageReporting")
m.sup.Add(service)
m.sup.ServeBackground()
} else if to.Options.URAccepted < usageReportVersion && m.sup != nil {
// Usage reporting was turned off
m.sup.Stop()
m.sup = nil
}
return true
}
func (m *usageReportingManager) String() string {
return fmt.Sprintf("usageReportingManager@%p", m)
}
const usageReportVersion = 3
// reportData returns the data to be sent in a usage report. It's used in
// various places, so not part of the usageReportingManager object.
func reportData(cfg configIntf, m modelIntf) map[string]interface{} {
func reportData(cfg configIntf, m modelIntf, connectionsService connectionsIntf, version int) map[string]interface{} {
opts := cfg.Options()
res := make(map[string]interface{})
res["urVersion"] = usageReportVersion
res["uniqueID"] = cfg.Options().URUniqueID
res["urVersion"] = version
res["uniqueID"] = opts.URUniqueID
res["version"] = Version
res["longVersion"] = LongVersion
res["platform"] = runtime.GOOS + "-" + runtime.GOARCH
@@ -93,14 +49,14 @@ func reportData(cfg configIntf, m modelIntf) map[string]interface{} {
var totFiles, maxFiles int
var totBytes, maxBytes int64
for folderID := range cfg.Folders() {
files, _, bytes := m.GlobalSize(folderID)
totFiles += files
totBytes += bytes
if files > maxFiles {
maxFiles = files
global := m.GlobalSize(folderID)
totFiles += global.Files
totBytes += global.Bytes
if global.Files > maxFiles {
maxFiles = global.Files
}
if bytes > maxBytes {
maxBytes = bytes
if global.Bytes > maxBytes {
maxBytes = global.Bytes
}
}
@@ -112,7 +68,8 @@ func reportData(cfg configIntf, m modelIntf) map[string]interface{} {
var mem runtime.MemStats
runtime.ReadMemStats(&mem)
res["memoryUsageMiB"] = (mem.Sys - mem.HeapReleased) / 1024 / 1024
res["sha256Perf"] = cpuBench(5, 125*time.Millisecond)
res["sha256Perf"] = cpuBench(5, 125*time.Millisecond, false)
res["hashPerf"] = cpuBench(5, 125*time.Millisecond, true)
bytes, err := memorySize()
if err == nil {
@@ -134,7 +91,7 @@ func reportData(cfg configIntf, m modelIntf) map[string]interface{} {
for _, cfg := range cfg.Folders() {
rescanIntvs = append(rescanIntvs, cfg.RescanIntervalS)
if cfg.Type == config.FolderTypeReadOnly {
if cfg.Type == config.FolderTypeSendOnly {
folderUses["readonly"]++
}
if cfg.IgnorePerms {
@@ -188,7 +145,7 @@ func reportData(cfg configIntf, m modelIntf) map[string]interface{} {
res["deviceUses"] = deviceUses
defaultAnnounceServersDNS, defaultAnnounceServersIP, otherAnnounceServers := 0, 0, 0
for _, addr := range cfg.Options().GlobalAnnServers {
for _, addr := range opts.GlobalAnnServers {
if addr == "default" || addr == "default-v4" || addr == "default-v6" {
defaultAnnounceServersDNS++
} else {
@@ -196,8 +153,8 @@ func reportData(cfg configIntf, m modelIntf) map[string]interface{} {
}
}
res["announce"] = map[string]interface{}{
"globalEnabled": cfg.Options().GlobalAnnEnabled,
"localEnabled": cfg.Options().LocalAnnEnabled,
"globalEnabled": opts.GlobalAnnEnabled,
"localEnabled": opts.LocalAnnEnabled,
"defaultServersDNS": defaultAnnounceServersDNS,
"defaultServersIP": defaultAnnounceServersIP,
"otherServers": otherAnnounceServers,
@@ -218,30 +175,168 @@ func reportData(cfg configIntf, m modelIntf) map[string]interface{} {
"otherServers": otherRelayServers,
}
res["usesRateLimit"] = cfg.Options().MaxRecvKbps > 0 || cfg.Options().MaxSendKbps > 0
res["usesRateLimit"] = opts.MaxRecvKbps > 0 || opts.MaxSendKbps > 0
res["upgradeAllowedManual"] = !(upgrade.DisabledByCompilation || noUpgrade)
res["upgradeAllowedAuto"] = !(upgrade.DisabledByCompilation || noUpgrade) && cfg.Options().AutoUpgradeIntervalH > 0
res["upgradeAllowedManual"] = !(upgrade.DisabledByCompilation || noUpgradeFromEnv)
res["upgradeAllowedAuto"] = !(upgrade.DisabledByCompilation || noUpgradeFromEnv) && opts.AutoUpgradeIntervalH > 0
res["upgradeAllowedPre"] = !(upgrade.DisabledByCompilation || noUpgradeFromEnv) && opts.AutoUpgradeIntervalH > 0 && opts.UpgradeToPreReleases
if version >= 3 {
res["uptime"] = int(time.Now().Sub(startTime).Seconds())
res["natType"] = connectionsService.NATType()
res["alwaysLocalNets"] = len(opts.AlwaysLocalNets) > 0
res["cacheIgnoredFiles"] = opts.CacheIgnoredFiles
res["overwriteRemoteDeviceNames"] = opts.OverwriteRemoteDevNames
res["progressEmitterEnabled"] = opts.ProgressUpdateIntervalS > -1
res["customDefaultFolderPath"] = opts.DefaultFolderPath != "~"
res["weakHashSelection"] = opts.WeakHashSelectionMethod.String()
res["customTrafficClass"] = opts.TrafficClass != 0
res["customTempIndexMinBlocks"] = opts.TempIndexMinBlocks != 10
res["temporariesDisabled"] = opts.KeepTemporariesH == 0
res["temporariesCustom"] = opts.KeepTemporariesH != 24
res["limitBandwidthInLan"] = opts.LimitBandwidthInLan
res["customReleaseURL"] = opts.ReleasesURL != "https://upgrades.syncthing.net/meta.json"
res["restartOnWakeup"] = opts.RestartOnWakeup
res["customStunServers"] = len(opts.StunServers) == 0 || opts.StunServers[0] != "default" || len(opts.StunServers) > 1
folderUsesV3 := map[string]int{
"scanProgressDisabled": 0,
"conflictsDisabled": 0,
"conflictsUnlimited": 0,
"conflictsOther": 0,
"disableSparseFiles": 0,
"disableTempIndexes": 0,
"alwaysWeakHash": 0,
"customWeakHashThreshold": 0,
"fsWatcherEnabled": 0,
}
pullOrder := make(map[string]int)
filesystemType := make(map[string]int)
var fsWatcherDelays []int
for _, cfg := range cfg.Folders() {
if cfg.ScanProgressIntervalS < 0 {
folderUsesV3["scanProgressDisabled"]++
}
if cfg.MaxConflicts == 0 {
folderUsesV3["conflictsDisabled"]++
} else if cfg.MaxConflicts < 0 {
folderUsesV3["conflictsUnlimited"]++
} else {
folderUsesV3["conflictsOther"]++
}
if cfg.DisableSparseFiles {
folderUsesV3["disableSparseFiles"]++
}
if cfg.DisableTempIndexes {
folderUsesV3["disableTempIndexes"]++
}
if cfg.WeakHashThresholdPct < 0 {
folderUsesV3["alwaysWeakHash"]++
} else if cfg.WeakHashThresholdPct != 25 {
folderUsesV3["customWeakHashThreshold"]++
}
if cfg.FSWatcherEnabled {
folderUsesV3["fsWatcherEnabled"]++
}
pullOrder[cfg.Order.String()]++
filesystemType[cfg.FilesystemType.String()]++
fsWatcherDelays = append(fsWatcherDelays, cfg.FSWatcherDelayS)
}
sort.Ints(fsWatcherDelays)
folderUsesV3Interface := map[string]interface{}{
"pullOrder": pullOrder,
"filesystemType": filesystemType,
"fsWatcherDelays": fsWatcherDelays,
}
for key, value := range folderUsesV3 {
folderUsesV3Interface[key] = value
}
res["folderUsesV3"] = folderUsesV3Interface
guiCfg := cfg.GUI()
// Anticipate multiple GUI configs in the future, hence store counts.
guiStats := map[string]int{
"enabled": 0,
"useTLS": 0,
"useAuth": 0,
"insecureAdminAccess": 0,
"debugging": 0,
"insecureSkipHostCheck": 0,
"insecureAllowFrameLoading": 0,
"listenLocal": 0,
"listenUnspecified": 0,
}
theme := make(map[string]int)
if guiCfg.Enabled {
guiStats["enabled"]++
if guiCfg.UseTLS() {
guiStats["useTLS"]++
}
if len(guiCfg.User) > 0 && len(guiCfg.Password) > 0 {
guiStats["useAuth"]++
}
if guiCfg.InsecureAdminAccess {
guiStats["insecureAdminAccess"]++
}
if guiCfg.Debugging {
guiStats["debugging"]++
}
if guiCfg.InsecureSkipHostCheck {
guiStats["insecureSkipHostCheck"]++
}
if guiCfg.InsecureAllowFrameLoading {
guiStats["insecureAllowFrameLoading"]++
}
addr, err := net.ResolveTCPAddr("tcp", guiCfg.Address())
if err == nil {
if addr.IP.IsLoopback() {
guiStats["listenLocal"]++
} else if addr.IP.IsUnspecified() {
guiStats["listenUnspecified"]++
}
}
theme[guiCfg.Theme]++
}
guiStatsInterface := map[string]interface{}{
"theme": theme,
}
for key, value := range guiStats {
guiStatsInterface[key] = value
}
res["guiStats"] = guiStatsInterface
}
for key, value := range m.UsageReportingStats(version) {
res[key] = value
}
return res
}
type usageReportingService struct {
cfg *config.Wrapper
model *model.Model
stop chan struct{}
cfg *config.Wrapper
model *model.Model
connectionsService *connections.Service
forceRun chan struct{}
stop chan struct{}
}
func newUsageReportingService(cfg *config.Wrapper, model *model.Model) *usageReportingService {
return &usageReportingService{
cfg: cfg,
model: model,
stop: make(chan struct{}),
func newUsageReportingService(cfg *config.Wrapper, model *model.Model, connectionsService *connections.Service) *usageReportingService {
svc := &usageReportingService{
cfg: cfg,
model: model,
connectionsService: connectionsService,
forceRun: make(chan struct{}),
stop: make(chan struct{}),
}
cfg.Subscribe(svc)
return svc
}
func (s *usageReportingService) sendUsageReport() error {
d := reportData(s.cfg, s.model)
d := reportData(s.cfg, s.model, s.connectionsService, s.cfg.Options().URAccepted)
var b bytes.Buffer
json.NewEncoder(&b).Encode(d)
@@ -260,53 +355,73 @@ func (s *usageReportingService) sendUsageReport() error {
func (s *usageReportingService) Serve() {
s.stop = make(chan struct{})
l.Infoln("Starting usage reporting")
defer l.Infoln("Stopping usage reporting")
t := time.NewTimer(time.Duration(s.cfg.Options().URInitialDelayS) * time.Second) // time to initial report at start
t := time.NewTimer(time.Duration(s.cfg.Options().URInitialDelayS) * time.Second)
for {
select {
case <-s.stop:
return
case <-s.forceRun:
t.Reset(0)
case <-t.C:
err := s.sendUsageReport()
if err != nil {
l.Infoln("Usage report:", err)
if s.cfg.Options().URAccepted >= 2 {
err := s.sendUsageReport()
if err != nil {
l.Infoln("Usage report:", err)
} else {
l.Infof("Sent usage report (version %d)", s.cfg.Options().URAccepted)
}
}
t.Reset(24 * time.Hour) // next report tomorrow
}
}
}
func (s *usageReportingService) VerifyConfiguration(from, to config.Configuration) error {
return nil
}
func (s *usageReportingService) CommitConfiguration(from, to config.Configuration) bool {
if from.Options.URAccepted != to.Options.URAccepted || from.Options.URUniqueID != to.Options.URUniqueID || from.Options.URURL != to.Options.URURL {
s.forceRun <- struct{}{}
}
return true
}
func (s *usageReportingService) Stop() {
close(s.stop)
close(s.forceRun)
}
func (usageReportingService) String() string {
return "usageReportingService"
}
// cpuBench returns CPU performance as a measure of single threaded SHA-256 MiB/s
func cpuBench(iterations int, duration time.Duration) float64 {
func cpuBench(iterations int, duration time.Duration, useWeakHash bool) float64 {
dataSize := 16 * protocol.BlockSize
bs := make([]byte, dataSize)
rand.Reader.Read(bs)
var perf float64
for i := 0; i < iterations; i++ {
if v := cpuBenchOnce(duration); v > perf {
if v := cpuBenchOnce(duration, useWeakHash, bs); v > perf {
perf = v
}
}
blocksResult = nil
return perf
}
func cpuBenchOnce(duration time.Duration) float64 {
chunkSize := 100 * 1 << 10
h := sha256.New()
bs := make([]byte, chunkSize)
rand.Reader.Read(bs)
var blocksResult []protocol.BlockInfo // so the result is not optimized away
func cpuBenchOnce(duration time.Duration, useWeakHash bool, bs []byte) float64 {
t0 := time.Now()
b := 0
for time.Since(t0) < duration {
h.Write(bs)
b += chunkSize
r := bytes.NewReader(bs)
blocksResult, _ = scanner.Blocks(context.TODO(), r, protocol.BlockSize, int64(len(bs)), nil, useWeakHash)
b += len(bs)
}
h.Sum(nil)
d := time.Since(t0)
return float64(int(float64(b)/d.Seconds()/(1<<20)*100)) / 100
}

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -66,7 +66,7 @@ func (s *verboseService) WaitForStart() {
func (s *verboseService) formatEvent(ev events.Event) string {
switch ev.Type {
case events.Ping, events.DownloadProgress, events.LocalIndexUpdated:
case events.DownloadProgress, events.LocalIndexUpdated:
// Skip
return ""
@@ -94,9 +94,12 @@ func (s *verboseService) formatEvent(ev events.Event) string {
case events.LocalChangeDetected:
data := ev.Data.(map[string]string)
// Local change detected in folder "foo": modified file /Users/jb/whatever
return fmt.Sprintf("Local change detected in folder %q: %s %s %s", data["folder"], data["action"], data["type"], data["path"])
case events.RemoteChangeDetected:
data := ev.Data.(map[string]string)
return fmt.Sprintf("Remote change detected in folder %q: %s %s %s", data["folder"], data["action"], data["type"], data["path"])
case events.RemoteIndexUpdated:
data := ev.Data.(map[string]interface{})
return fmt.Sprintf("Device %v sent an index update for %q with %d items", data["device"], data["folder"], data["items"])
@@ -151,7 +154,7 @@ func (s *verboseService) formatEvent(ev events.Event) string {
if total > 0 {
pct = 100 * current / total
}
return fmt.Sprintf("Scanning folder %q, %d%% done (%.01f MB/s)", folder, pct, rate)
return fmt.Sprintf("Scanning folder %q, %d%% done (%.01f MiB/s)", folder, pct, rate)
case events.DevicePaused:
data := ev.Data.(map[string]string)
@@ -163,6 +166,18 @@ func (s *verboseService) formatEvent(ev events.Event) string {
device := data["device"]
return fmt.Sprintf("Device %v was resumed", device)
case events.FolderPaused:
data := ev.Data.(map[string]string)
id := data["id"]
label := data["label"]
return fmt.Sprintf("Folder %v (%v) was paused", id, label)
case events.FolderResumed:
data := ev.Data.(map[string]string)
id := data["id"]
label := data["label"]
return fmt.Sprintf("Folder %v (%v) was resumed", id, label)
case events.ListenAddressesChanged:
data := ev.Data.(map[string]interface{})
address := data["address"]

View File

@@ -2,7 +2,7 @@
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
// You can obtain one at https://mozilla.org/MPL/2.0/.
// +build ignore

View File

@@ -1,5 +0,0 @@
{{.name}} ({{.version}}); urgency=medium
* Packaging of {{.version}}.
-- Syncthing Release Management <release@syncthing.net> {{.date}}

View File

@@ -1 +0,0 @@
9

View File

@@ -1,16 +0,0 @@
Package: syncthing
Version: {{.version}}
Priority: optional
Section: net
Architecture: {{.arch}}
Depends: libc6, procps
Homepage: https://syncthing.net/
Maintainer: Syncthing Release Management <release@syncthing.net>
Description: Open Source Continuous File Synchronization
Syncthing is an application that lets you synchronize your files across
multiple devices. This means the creation, modification or deletion of files
on one machine will automatically be replicated to your other devices. We
believe your data is your data alone and you deserve to choose where it is
stored. Therefore Syncthing does not upload your data to the cloud but
exchanges your data across your machines as soon as they are online at the
same time.

Some files were not shown because too many files have changed in this diff Show More