Compare commits

...

769 Commits

Author SHA1 Message Date
Simon Frei
f6f696c6c5 lib/config: Prevent nil deref in debug logging (fixes #5955) (#5956) 2019-08-15 15:51:09 +02:00
Simon Frei
cf40ed6cec lib/connections: Return exported intf from exported function (#5947) 2019-08-13 09:33:33 +02:00
Simon Frei
6fa02d5081 lib/model: Fix a few more problematic locks (ref #5929) (#5944) 2019-08-13 09:04:43 +02:00
Simon Frei
86e35f1879 lib/model: Less locking in ClusterConfig (#5943) 2019-08-11 19:30:24 +02:00
Jakob Borg
720a6bf62e lib/tlsutil: Remove hardcoded curve preferences (fixes #5940) (#5942)
They are arguable outdated and we are better off trusting the standard
library than trying to keep up with it ourselves.
2019-08-11 19:01:57 +02:00
Simon Frei
4a619e74f2 lib/model: Fix incorrect locking (#5939) 2019-08-11 16:10:30 +02:00
Audrius Butkevicius
58ef5368f8 lib/connections: Validate device id before assuming success (fixes #5934) (#5935)
* lib/connections: Validate device id before assuming success (fixes #5934)

* Vet
2019-08-09 12:31:42 +01:00
Cromefire_
7b37d453f9 build, etc: Add systemd units and ufw rules for relay and discovery (fixes #5115) (#5350) 2019-08-08 18:04:52 +02:00
Oliver Freyermuth
edf2399ce6 Add NATSymmetricUDPFirewall to punchable NATs (#5931)
NATSymmetricUDPFirewall actually is not NAT at all, but means the machine has a global IP address and an UDP firewall in front (RFC calls it Symmetric UDP Firewall). This is punchable fine, both theoretically and also practically in testing.
2019-08-06 12:26:02 +01:00
dependabot-preview[bot]
d43b0a4395 build(deps): bump github.com/urfave/cli from 1.20.0 to 1.21.0 (#5928)
Bumps [github.com/urfave/cli](https://github.com/urfave/cli) from 1.20.0 to 1.21.0.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/master/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v1.20.0...v1.21.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-08-05 12:18:35 +02:00
Jakob Borg
f2efd08e0f Merge branch 'release'
* release:
  cmd/syncthing: Print version information early (fixes #5891) (#5893)
2019-08-05 07:16:07 +02:00
Jakob Borg
61b9f7bd55 lib/config: Format bytes in insufficient space errors (fixes #5920) (#5921) 2019-08-02 14:43:05 +02:00
Simon Frei
1475c0344a gui: Parse strings before copying options object (fixes #5824) (#5922) 2019-08-02 12:46:27 +02:00
Simon Frei
77cc87dfca lib/connections: Log errors in relay clients (#5917) 2019-08-01 17:37:58 +02:00
Jakob Borg
bc7dd02e2b authors: Update carstenhag (Moter8) (fixes #5919) 2019-08-01 17:35:45 +02:00
Simon Frei
8a06cf0973 lib/model: Unflake TestPullInvalidIgnored (#5918) 2019-08-01 11:07:41 +02:00
Simon Frei
05835ed81f all: Remove potentially problematic errors from panics (fixes #5839) (#5912) 2019-07-31 10:53:35 +02:00
Simon Frei
df522576ac lib/model: Don't call t.Fatal in goroutines (fixes #5901) (#5903) 2019-07-30 17:50:51 +02:00
Simon Frei
d681ac11fe lib/config: Handle empty Fstype for mtime-window (#5906) 2019-07-30 15:23:00 +02:00
Simon Frei
7d5f7d508d lib/syncthing: Stop only once (fixes #5908) (#5909) 2019-07-29 20:07:19 +02:00
Simon Frei
1d182e4631 lib/fs: Use gopsutils for disk usage (#5905) 2019-07-29 20:06:17 +02:00
Simon Frei
710f5c199f lib/db: Don't use global fileset in benchmarks (#5902) 2019-07-28 22:31:24 +02:00
Simon Frei
fd847d4efe lib/model: Fix flakyness of TestRequestRemoteRenameChanged (#5904) 2019-07-28 22:29:31 +02:00
Jakob Borg
15e51fc045 cmd/stcrashreceiver: Store and serve compressed reports (#5892)
This changes the on disk format for new raw reports to be gzip
compressed. Also adds the ability to serve these reports in plain text,
to insulate web browsers from the change (previously we just served the
raw reports from disk using Caddy).
2019-07-28 11:13:04 +02:00
Jakob Borg
c1c976aa2b lib/model: Don't panic on failed chmod-back on directory (fixes #5836) (#5896)
* lib/model: Don't panic on failed chmod-back on directory (fixes #5836)

This makes the "in writable dir"-wrapper log chmod-back errors instead
of panicking. To do that we need a logger so the function moved into the
model package which is also the only place it's used. The tests came
along.

(The test also exercised osutil.RenameOrCopy like some sort of
piggybacking. I removed that.)
2019-07-28 10:25:05 +02:00
Jakob Borg
159d1a68e1 lib/api: Don't log random stuff in the HTTP server (fixes #5738) (#5897) 2019-07-28 09:49:07 +02:00
Simon Frei
dd850f66bb lib/api: Close server before exiting serve (fixes #5866) (#5895) 2019-07-28 08:03:55 +02:00
Jakob Borg
d0c3697152 cmd/syncthing: Print version information early (fixes #5891) (#5893) 2019-07-27 20:36:33 +02:00
Jakob Borg
669bcb748f lib/config, lib/model: Don't save on every pending folder/device update (fixes #5888) (#5890)
Wrapper methods generally don't save by themselves.
2019-07-27 11:05:00 +01:00
Jakob Borg
4e22a96602 cmd/syncthing: Print version information early (fixes #5891) (#5893) 2019-07-27 10:58:39 +01:00
Jakob Borg
a992559abc lib/db: Add hacky way to adjust database parameters (#5889)
This adds a set of magical environment variables that can be used to
tweak the database parameters. It's totally undocumented and not
intended to be a long term or supported thing.

It's ugly, but there is a backstory. I have a couple of large
installations where the database options are inefficient or otherwise
suboptimal (24/7 compaction going on and stuff like that). I don't
*know* the correct database parameters, nor yet the formula or method to
derive them by, so this requires experimentation. Experimentation needs
to happen partly in production, and rolling out new builds for every
tweak isn't practical. This provides override points for all reasonable
values, while not changing anything by default.

Ideally, at the end of such experimentation, we'll know which values are
relevant to change and in what manner, and can provide a more user
friendly knob to do so - or do it automatically based on the database
size.
2019-07-26 22:18:42 +02:00
Simon Frei
46e72d76b5 cmd/syncthing, lib/syncthing: Create library utils (ref #4085) (#5871) 2019-07-23 23:39:20 +02:00
Simon Frei
7a4c88d4e4 lib: Add mtime window when comparing files (#5852) 2019-07-23 21:48:53 +02:00
Simon Frei
35f40e9a58 lib/model: Create new file-set after stopping folder (fixes #5882) (#5883) 2019-07-23 20:39:25 +02:00
Simon Frei
5de9b677c2 lib/fs: Fix kqueue event list (fixes #5308) (#5885) 2019-07-23 14:11:15 +02:00
Simon Frei
6f08162376 lib/model: Remove incorrect/useless panics (#5881) 2019-07-23 10:51:16 +02:00
Simon Frei
7b3d9a8dca lib/syncthing: Refactor to use util.AsService (#5858) 2019-07-23 10:50:37 +02:00
Simon Frei
942659fb06 lib/model, lib/nat: More service termination speedup (#5884) 2019-07-23 10:49:22 +02:00
dependabot-preview[bot]
15c262184b build(deps): bump github.com/maruel/panicparse from 1.2.1 to 1.3.0 (#5879)
Bumps [github.com/maruel/panicparse](https://github.com/maruel/panicparse) from 1.2.1 to 1.3.0.
- [Release notes](https://github.com/maruel/panicparse/releases)
- [Commits](https://github.com/maruel/panicparse/compare/v1.2.1...v1.3.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-22 19:46:52 +01:00
dependabot-preview[bot]
484fa0592e build(deps): bump github.com/lib/pq from 1.1.1 to 1.2.0 (#5878)
Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.1.1 to 1.2.0.
- [Release notes](https://github.com/lib/pq/releases)
- [Commits](https://github.com/lib/pq/compare/v1.1.1...v1.2.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-07-22 08:07:21 +01:00
Simon Frei
b5b54ff057 lib/model: No watch-error on missing folder (fixes #5833) (#5876) 2019-07-19 19:41:16 +02:00
Simon Frei
4d3432af3e lib: Ensure timely service termination (fixes #5860) (#5863) 2019-07-19 19:40:40 +02:00
Simon Frei
1cb55904bc lib/model: Prevent panic in NeedFolderFiles (fixes #5872) (#5875) 2019-07-19 19:39:52 +02:00
Simon Frei
2b622d0774 lib/model: Close conn on dev pause (fixes #5873) (#5874) 2019-07-19 19:37:29 +02:00
Simon Frei
1894123d3c lib/syncthing: Modify exit status before stopping (fixes #5869) (#5870) 2019-07-18 20:49:00 +02:00
Jakob Borg
e7e177a6fa lib/relay: Prevent spurious relay error message (fixes #5861) (#5864) 2019-07-17 10:55:28 +02:00
Simon Frei
eed1edcca0 cmd/syncthing: Ensure myID is set by making it local (fixes #5859) (#5862) 2019-07-17 07:19:14 +02:00
Simon Frei
0025e9ccfb all: Refactor cmd/syncthing creating lib/syncthing (ref #4085) (#5805)
* add skeleton for lib/syncthing

* copy syncthingMain to lib/syncthing (verbatim)

* Remove code to deduplicate copies of syncthingMain

* fix simple build errors

* move stuff from main to syncthing with minimal mod

* merge runtime options

* actually use syncthing.App

* pass io.writer to lib/syncthing for auditing

* get rid of env stuff in lib/syncthing

* add .Error() and comments

* review: Remove fs interactions from lib

* and go 1.13 happened

* utility functions
2019-07-14 11:43:13 +01:00
Simon Frei
82b70b9fae lib/model, lib/protocol: Track closing connections (fixes #5828) (#5829) 2019-07-14 11:03:55 +02:00
Simon Frei
def4b8cee5 lib/config: Error on empty folder path (fixes #5853) (#5854) 2019-07-14 11:03:14 +02:00
Aurélien Rainone
f1a7dd766e all: Add comment to ensure correct atomics alignment (fixes #5813)
Per the sync/atomic bug note:

> On ARM, x86-32, and 32-bit MIPS, it is the caller's
> responsibility to arrange for 64-bit alignment of 64-bit words
> accessed atomically. The first word in a variable or in an
> allocated struct, array, or slice can be relied upon to be
> 64-bit aligned.

All atomic accesses of 64-bit variables in syncthing code base are
currently ok (i.e they are all 64-bit aligned).

Generally, the bug is triggered because of incorrect alignement
of struct fields. Free variables (declared in a function) are
guaranteed to be 64-bit aligned by the Go compiler.

To ensure the code remains correct upon further addition/removal
of fields, which would change the currently correct alignment, I
added the following comment where required:

     // atomic, must remain 64-bit aligned

See https://golang.org/pkg/sync/atomic/#pkg-note-BUG.
2019-07-13 14:05:39 +01:00
Simon Frei
20c8dbd9ed lib/model: Fix integer conversion (fixes #5837) (#5851) 2019-07-12 16:37:12 +02:00
xduugu
4b3f9b1af9 lib/versioner: Replace multiple placeholders in a single token in external command (fixes #5849)
* lib/versioner: Add placeholder to provide the absolute file path to external commands

This commit adds support for a new placeholder, %FILE_PATH_FULL%, to the
command of the external versioner. The placeholder will be replaced by
the absolute path of the file that should be deleted.

* Revert "lib/versioner: Add placeholder to provide the absolute file path to external commands"

This reverts commit fb48962b94.

* lib/versioner: Replace all placeholders in external command (fixes #5849)

Before this commit, only these placeholders were replaced that span a
whole word, for example "%FOLDER_PATH%". Words that consisted of more
than one placeholder or additional characters, for example
"%FOLDER_PATH%/%FILE_PATH%", were left untouched.

* fixup! lib/versioner: Replace all placeholders in external command (fixes #5849)
2019-07-12 08:45:39 +01:00
Simon Frei
3446d50201 lib/model: Remove pointless error that watch hasn't started (fixes #5833) (#5834) 2019-07-10 11:00:06 +02:00
Simon Frei
9fef1552fc lib/db, lib/model: Remove folder info from panics (ref #5839) (#5840) 2019-07-10 10:57:49 +02:00
Simon Frei
85318f3b82 gui: On update setting don't show RC msg when disabled (fixes #5803) (#5842) 2019-07-09 22:30:22 +01:00
Simon Frei
485acda63b lib/relay: Call the proper Error method (ref #5806) (#5841) 2019-07-09 22:29:19 +01:00
Simon Frei
05e9e0bfa9 build: Update notify dependency (#5838) 2019-07-09 21:33:22 +01:00
Simon Frei
ba056578ec lib: Add util.Service as suture.Service template (fixes #5801) (#5806) 2019-07-09 11:40:30 +02:00
Jakob Borg
d0ab65a178 cmd/stcrashreceiver: Don't leak clients
Use a global raven.Client because they allocate an http.Client for each,
with a separate CA bundle and infinite connection idle time. Infinite
connection idle time means that if the client is never used again it
will always keep the connection around, not verifying whether it's
closed server side or not. This leaks about a megabyte of memory for
each client every created.

client.Close() doesn't help with this because the http.Client is still
around, retained by its own goroutines.

The thing with the map is just to retain the API on sendReport, even
though there will in practice only ever be one DSN per process
instance...
2019-07-09 11:11:06 +02:00
Simon Frei
4cba433852 build: Add go major version to go.mod (#5822) 2019-06-30 13:18:34 +02:00
Simon Frei
863fe23347 gui, lib/model: Fix download progress accounting (fixes #5811) (#5815) 2019-06-30 09:23:47 +02:00
Jakob Borg
43b6ac9501 cmd/stcrashreceiver: Add source code loader (#5779) 2019-06-29 08:50:09 +02:00
Simon Frei
1cf352a722 lib/model: NewFileSet outside fmut (#5818) 2019-06-29 08:49:30 +02:00
Simon Frei
b58f6ca886 lib/model: Correct/unify check if item changed (#5819) 2019-06-29 07:45:41 +02:00
Jakob Borg
5cbc9089fd Merge branch 'release'
* release:
  go.mod: Update AudriusButkevicius/pfilter (fixes #5820)
2019-06-28 08:21:00 +02:00
Jakob Borg
20eab36a33 go.mod: Update AudriusButkevicius/pfilter (fixes #5820) 2019-06-28 08:03:29 +02:00
Jakob Borg
2b4df6b874 go.mod: Update AudriusButkevicius/pfilter (fixes #5820) 2019-06-28 07:38:52 +02:00
Simon Frei
3c7e7e971d lib/model: Make jobQueue.Jobs paginated (fixes #5754) (#5804)
* lib/model: Make jobQueue.Jobs paginated (fixes #5754)

* fix, no test yet

* add test
2019-06-27 19:25:38 +01:00
Audrius Butkevicius
afde0727fe lib/versioner: Revert naming change (fixes #5807) (#5808) 2019-06-25 08:56:11 +03:00
Simon Frei
bf744ded31 cmd/syncthing, lib/db: Exit/close db faster (fixes #5781) (#5782)
This adds a 10s timeout on closing the db and additionally cancels active
db iterators and waits for them to terminate before closing the db.
2019-06-17 15:27:25 +03:00
Wulf Weich
0d86166890 gui: Optimize folder/device info for small screens (fixes #5774) (#5787)
break table layout and add button margin on small screens
2019-06-17 15:24:45 +03:00
Simon Frei
cea5962417 lib/model: Unflake TestPullInvalidIgnoredSR/SO (fixes #5796) (#5799) 2019-06-17 15:23:28 +03:00
Simon Frei
02752af862 lib/protocol: Don't block on Close (fixes #5794) (#5795) 2019-06-14 19:04:41 +02:00
Simon Frei
6b1d7ac727 gui: Check data before calling .reverse() (#5793) 2019-06-14 13:14:15 +02:00
Simon Frei
abd363e8bb lib/model: Don't error on pulling deletion of invalid file (fixes #5791) (#5792) 2019-06-14 08:48:14 +02:00
Jakob Borg
bff1a5f5e4 build: Upgrade github.com/syndtr/goleveldb 2019-06-14 06:56:52 +02:00
Simon Frei
38302270d4 build: Include cross-package test coverage (#5735) 2019-06-13 19:28:14 +02:00
desbma
1b4fe39a89 etc: Remove Systemd hardening options unsupported in user mode (#5788) 2019-06-12 10:31:19 +02:00
Jakob Borg
6b74cdc613 gui, man, authors: Update docs, translations, and contributors 2019-06-12 07:45:26 +02:00
Simon Frei
13a746e0fb lib/model: Prevent nil deref if folder stopped (fixes #5780) (#5778) 2019-06-11 11:48:51 +02:00
Audrius Butkevicius
21f50e2f8f lib/versioner: Use mtime for version cleanup (fixes #5765) (#5769) 2019-06-11 09:16:55 +02:00
Audrius Butkevicius
b7c70a9817 lib/fs: Enhance mtimefs, use everywhere (fixes #5777) (#5776) 2019-06-11 08:27:12 +02:00
Jakob Borg
42ce6be9b9 lib/ur: Implement crash (panic) reporting (fixes #959) (#5702)
* lib/ur: Implement crash (panic) reporting (fixes #959)

This implements a simple crash reporting method. It piggybacks on the
panic log files created by the monitor process, picking these up and
uploading them from the usage reporting routine.

A new config value points to the crash receiver base URL, which defaults
to "https://crash.syncthing.net/newcrash" (following the pattern of
"https://data.syncthing.net/newdata" for usage reports, but allowing us
to separate the service as required).
2019-06-11 08:19:11 +02:00
dependabot-preview[bot]
93e57bd357 build(deps): bump github.com/prometheus/client_golang (#5775)
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 0.9.3 to 0.9.4.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/master/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v0.9.3...v0.9.4)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-06-10 19:06:25 +02:00
Mingxuan Lin
eb4fe808c5 lib/fs: Fallback EvalSymlinks method on windows (fixes #5609) (#5611) 2019-06-10 14:33:53 +02:00
Simon Frei
1054ce9354 lib/model: Refactor sending indexes as suture service (#5757) 2019-06-10 13:27:22 +02:00
Simon Frei
97ad575b1f gui: Enable rescan button with failed items (fixes #5770) (#5771) 2019-06-10 10:12:30 +02:00
Audrius Butkevicius
ee746263fb lib/connections: Do not leak FDs, fix address copy (fixes #5767) (#5768)
* lib/connections: Do not leak FDs, fix address copy (fixes #5767)

* build

* Update quic_listen.go

* Update quic_listen.go
2019-06-09 22:14:00 +01:00
Jakob Borg
41ff4b323e lib/api: Correct logic for errors.jons in support bundle (fixes #5766)
The err check should always be done with != and have the error case
first, to follow the pattern and avoid mistakes like these.
2019-06-09 09:35:05 +02:00
Jakob Borg
997bb5e7e1 all: Remove "large blocks" config (#5763)
We now always use large / variable blocks.
2019-06-06 15:57:38 +01:00
Jakob Borg
ca2fa8de4e lib/build: Version 1.2 will be the Fermium Flea 2019-06-06 14:45:07 +02:00
Jakob Borg
64185484b2 readme: Remove old, dead link (fixes #5760) 2019-06-06 08:35:03 +02:00
Simon Frei
5541697d18 gui: Don't call API when no modal is open (fixes #5652) (#5759) 2019-06-05 14:03:52 +08:00
Simon Frei
e39d3f95dd lib/protocol: Prioritize close msg and add close timeout (#5746) 2019-06-05 14:01:59 +08:00
Jakob Borg
6e8aa0ec25 gui, man, authors: Update docs, translations, and contributors 2019-06-05 07:45:26 +02:00
dependabot-preview[bot]
97057eb9de build(deps): bump github.com/lucas-clemente/quic-go (#5761)
Bumps [github.com/lucas-clemente/quic-go](https://github.com/lucas-clemente/quic-go) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/lucas-clemente/quic-go/releases)
- [Changelog](https://github.com/lucas-clemente/quic-go/blob/master/Changelog.md)
- [Commits](https://github.com/lucas-clemente/quic-go/compare/v0.11.1...v0.11.2)
2019-06-03 12:42:23 +01:00
Simon Frei
6664e01acf lib/protocol: Return from ClusterConfig when closed (#5752) 2019-05-29 12:14:00 +02:00
dependabot-preview[bot]
e2a647a6a4 build(deps): bump golang.org/x/text from 0.3.0 to 0.3.2 (#5751)
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.3.0 to 0.3.2.
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.3.0...v0.3.2)
2019-05-29 11:42:07 +02:00
Jakob Borg
5ce5b2c94a lib/config: Refactor migrations a bit (#5750)
This breaks out config migrations to a separate concept, making it
(imho) slightly easier to maintain and get an overview.
2019-05-29 11:37:44 +02:00
Audrius Butkevicius
e714df013f lib/connections: Add QUIC protocol support (fixes #5377) (#5737) 2019-05-29 09:56:40 +02:00
Kalle Laine
d8fa61e27c github: Create SECURITY.md (#5749)
Now that github has added the security section why not use it.
2019-05-29 09:39:00 +02:00
Jakob Borg
d3f583c8c9 gui, man, authors: Update docs, translations, and contributors 2019-05-29 07:45:25 +02:00
Jakob Borg
e096a14ff5 golang-ci: Disable confused scopelint check 2019-05-27 12:54:08 +02:00
Simon Frei
3775a64d5c lib/protocol: Don't send anything else before cluster config (#5741) 2019-05-27 11:15:34 +01:00
Simon Frei
129df0613b lib/model: Unflake folder restart with blocking conn test (ref #5707) (#5744) 2019-05-27 10:58:09 +01:00
Simon Frei
486230768e lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697) (#5734)
* lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697)

* forgot unsupported watch

* and more non(-standard)-unixy fixes

* and windows test

* review
2019-05-25 20:08:26 +01:00
Simon Frei
9e6db72535 lib/protocol: Don't call receiver after calling Closed (fixes #4170) (#5742)
* lib/protocol: Don't call receiver after calling Closed (fixes #4170)

* review
2019-05-25 20:08:07 +01:00
Simon Frei
d91da8feee lib/model: Readd special handling of conn close in TestIssue5063 (#5743)
This partially reverts commit 64518b0f7e.
2019-05-25 18:51:13 +01:00
Simon Frei
64518b0f7e lib/model: Close connections when model is stopped (#5733) 2019-05-25 16:00:32 +02:00
Simon Frei
5d35b2c540 lib/protocol: Test for Close on blocking send deadlock (ref #5442) (#5732) 2019-05-23 21:42:02 +01:00
Jakob Borg
224775ca81 golang-ci: Not as good at modules as you'd hope 2019-05-23 17:34:32 +02:00
Jakob Borg
98cda96b1c gui, man, authors: Update docs, translations, and contributors 2019-05-22 07:45:27 +02:00
Simon Frei
5b306510a0 lib/model: Consistently cleanup model in tests (#5724) 2019-05-19 14:29:07 +02:00
Jakob Borg
f593ac387c golang-ci: Turn up the heat 2019-05-18 12:51:49 +02:00
Jakob Borg
eb8df7f632 cmd/ursrv: Lint fixes 2019-05-18 11:59:32 +02:00
dependabot[bot]
78d6eee74a build(deps): bump github.com/prometheus/client_golang (#5729)
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 0.9.2 to 0.9.3.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/master/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v0.9.2...v0.9.3)

Signed-off-by: dependabot[bot] <support@dependabot.com>
2019-05-18 09:57:35 +02:00
Simon Frei
1b2b970f32 lib/model, lib/testutils: Test closing a connection on folder restart (#5707) 2019-05-18 08:53:59 +02:00
Simon Frei
5ffbb7668d lib/model: Fix test flakyness regression (ref #5592) (#5718) 2019-05-18 08:52:50 +02:00
Simon Frei
1df8701c46 test: Report time per MiB on transfer benchs (#5711) 2019-05-18 08:46:08 +02:00
Tom Jakubowski
cc36621b11 docker: Create entrypoint script (fixes #5631) (#5635) 2019-05-18 08:43:53 +02:00
Simon Frei
441ea109a1 lib/model: Refactor file deletions when pulling (#5699) 2019-05-17 18:29:54 +02:00
Jakob Borg
8015f3f937 cmd/ursrv: Add F-droid and some third party distributors to the distrubtion summary 2019-05-17 10:18:37 +02:00
Jakob Borg
2c866277a2 lib/api, lib/connections, gui: Show connection error for disconnected devices (fixes #3345) (#5727)
* lib/api, lib/connections, gui: Show connection error for disconnected devices (fixes #3345)

This adds functionality in the connetions service to track the last
error per address. That is in turn exposed in the /rest/system/status
API method, as that is also where we already show the listener status
from the connection service.

The GUI uses this info where it lists addresses, showing errors (if any)
in red underneath each address.

I also slightly refactored the existing status method on the connection
service to have a better name and return typed information.

* ok

* review

* formatting

* review
2019-05-16 22:11:45 +01:00
Jakob Borg
1da9317a09 authors: Update acolomb (fixes #5726) 2019-05-15 14:47:35 +02:00
Jakob Borg
e16a65bacb cmd/ursrv: Summarize known distribution channels
That is, whether the binary was downloaded from GitHub, from our APT
repository, etc.
2019-05-15 13:42:55 +02:00
Jakob Borg
c02aed0a21 gui, man, authors: Update docs, translations, and contributors 2019-05-15 07:45:24 +02:00
André Colomb
e4956358fb lib/model: Remove superfluous check for IndexID in remote ClusterConfig (#5717)
The check in ClusterConfig() when iterating through announced devices
in a folder explicitly skips entries without a non-zero IndexID.
Therefore, the check for IndexID == 0 just below will never be true
and the intended cleanup of local index data will not happen.

Plainly remove that check to make the intended case distinction work.
2019-05-12 21:17:55 +02:00
dependabot[bot]
ad44d77a21 build(deps): bump github.com/lib/pq from 1.0.0 to 1.1.1 (#5714)
Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.0.0 to 1.1.1.
- [Release notes](https://github.com/lib/pq/releases)
- [Commits](https://github.com/lib/pq/compare/v1.0.0...v1.1.1)

Signed-off-by: dependabot[bot] <support@dependabot.com>
2019-05-11 22:15:30 +02:00
dependabot[bot]
6bb5d140fa build(deps): bump github.com/oschwald/geoip2-golang from 1.1.0 to 1.3.0 (#5713)
Bumps [github.com/oschwald/geoip2-golang](https://github.com/oschwald/geoip2-golang) from 1.1.0 to 1.3.0.
- [Release notes](https://github.com/oschwald/geoip2-golang/releases)
- [Commits](https://github.com/oschwald/geoip2-golang/compare/v1.1.0...v1.3.0)

Signed-off-by: dependabot[bot] <support@dependabot.com>
2019-05-11 21:02:49 +01:00
dependabot[bot]
d082b2ba4c build(deps): bump github.com/mattn/go-isatty from 0.0.4 to 0.0.7 (#5712)
Bumps [github.com/mattn/go-isatty](https://github.com/mattn/go-isatty) from 0.0.4 to 0.0.7.
- [Release notes](https://github.com/mattn/go-isatty/releases)
- [Commits](https://github.com/mattn/go-isatty/compare/v0.0.4...v0.0.7)

Signed-off-by: dependabot[bot] <support@dependabot.com>
2019-05-11 21:00:00 +01:00
dependabot[bot]
638789899c build(deps): bump github.com/gogo/protobuf from 1.2.0 to 1.2.1 (#5715)
Bumps [github.com/gogo/protobuf](https://github.com/gogo/protobuf) from 1.2.0 to 1.2.1.
- [Release notes](https://github.com/gogo/protobuf/releases)
- [Commits](https://github.com/gogo/protobuf/compare/v1.2.0...v1.2.1)

Signed-off-by: dependabot[bot] <support@dependabot.com>
2019-05-11 20:59:44 +01:00
Simon Frei
dfbbb286fc lib/fs: Consider win83 for root path as well when watching (ref #5706) (#5709) 2019-05-11 10:06:04 +02:00
Simon Frei
fbd445fe0a lib/api: Clean up after test, enables test caching (#5710) 2019-05-11 08:41:32 +02:00
Simon Frei
2b246eeb52 lib/model: Move test utilities to separate files (#5694) 2019-05-10 13:33:45 +02:00
Simon Frei
2558b021e5 lib/fs: Remove \\?\ for drive letters when watching (fixes #5578) (#5701) 2019-05-10 09:09:58 +02:00
Jakob Borg
31be810eb6 lib/model: Don't fail operation when fsync() fails (fixes #5704) (#5705) 2019-05-09 21:20:29 +01:00
Jakob Borg
62a6d619e7 Merge branch 'release'
* release:
  lib/fs: Revert removal of \\?\ for drive letters (fixes #5695)
2019-05-08 17:38:08 +02:00
Jakob Borg
a04fcfe749 lib/fs: Revert removal of \\?\ for drive letters (fixes #5695)
This reverts commit ca823bd591 from #5633.

This un-fixes bug #5578.
2019-05-08 17:31:52 +02:00
Simon Frei
283f39ae5f lib/protocol: Revert unreleased changes related to closing connections (#5688)
This reverts commits:
    ec7c88ca55
    19b51c9b92
    5da41f75fa
    04b927104f
2019-05-08 08:08:26 +02:00
Jakob Borg
59e1349499 gui, man, authors: Update docs, translations, and contributors 2019-05-08 07:45:28 +02:00
Simon Frei
b45d77b6be lib/model: Fix regression deleting directories on pull (ref #5690) (#5691) 2019-05-06 20:55:26 +02:00
Simon Frei
79e67b7f79 lib/model: Remove individual pull errors on retry (fixes #5659) (#5690) 2019-05-06 17:21:56 +02:00
Jakob Borg
36a4a9fd34 Merge branch 'release'
* release:
  etc: Revert systemd After=multi-user.target addition (fixes #5676)
2019-05-06 12:24:39 +02:00
Nitroretro
92ed31fe21 gui: Fix typo in log viewer modal (#5687) 2019-05-03 07:28:17 +02:00
Simon Frei
5fa8467756 lib/model: Consistent use of locks (#5684) 2019-05-02 18:55:39 +01:00
Simon Frei
5954b105cd lib/model: Let fakeConnection call Model.Closed on close (#5682) 2019-05-02 14:24:55 +02:00
Simon Frei
defc5dca65 lib/model: Use correct lock (#5683) 2019-05-02 14:09:42 +02:00
Simon Frei
9f358ecae0 lib/protocol: Refactor to pass only relevant argument to writeMessage (#5681) 2019-05-02 14:09:19 +02:00
Simon Frei
fe4daf242b cmd, lib/db: Actually close goleveldb (fixes #5505) (#5671) 2019-05-02 11:15:00 +02:00
Simon Frei
ec7c88ca55 lib/protocol: Fix yet another deadlock (fixes #5678) (#5679)
* lib/protocol: Fix yet another deadlock (fixes #5678)

* more consistency

* read deadlock

* naming

* more naming
2019-05-02 09:21:07 +01:00
Jakob Borg
26e6d94c00 gui, man, authors: Update docs, translations, and contributors 2019-05-01 07:45:25 +02:00
Jakob Borg
a54d7a32d1 etc: Revert systemd After=multi-user.target addition (fixes #5676)
This reverts commit 84fe285659.
2019-04-29 21:48:28 +02:00
Jakob Borg
e2470c8bc8 etc: Revert systemd After=multi-user.target addition (fixes #5676)
This reverts commit 84fe285659.
2019-04-29 21:45:01 +02:00
Simon Frei
19b51c9b92 lib/protocol: Don't close asyncMessage.done on success (fixes #5674) (#5675) 2019-04-29 17:52:57 +02:00
Audrius Butkevicius
0ca1f26ff8 lib/versioner: Restore for all versioners, cross-device support (#5514)
* lib/versioner: Restore for all versioners, cross-device support

Fixes #4631
Fixes #4586
Fixes #1634
Fixes #5338
Fixes #5419
2019-04-28 23:30:16 +01:00
Jakob Borg
2984d40641 lib/ignore: Additional test case (#5672) 2019-04-28 21:20:11 +01:00
Simon Frei
5da41f75fa lib/model, lib/protocol: Wait for reader/writer loops on close (fixes #4170) (#5657)
* lib/protocol: Wait for reader/writer loops on close (fixes #4170)

* waitgroup

* lib/model: Don't hold lock while closing connection

* fix comments

* review (lock once, func argument) and naming
2019-04-28 11:58:51 +01:00
Jakob Borg
32dec4a00d gui, man, authors: Update docs, translations, and contributors 2019-04-24 07:45:24 +02:00
Simon Frei
04b927104f lib/protocol: Don't send any messages before cluster config (#5646)
* lib/model: Send cluster config before releasing pmut

* reshuffle

* add model.connReady to track cluster-config status

* Corrected comments/strings

* do it in protocol
2019-04-23 20:47:11 +01:00
Simon Frei
110806842c lib/model: Pass the old not new fileinfo to deleteItemOnDisk (fixes #5654) (#5655) 2019-04-23 20:46:28 +01:00
Jonas Thelemann
d9b3415dec gui: Semicolon insertion (#5666) 2019-04-23 18:37:51 +01:00
Jonas Thelemann
d3d43d90f6 gui: Remove superfluous trailing argument (#5668) 2019-04-23 18:37:20 +01:00
Jonas Thelemann
e7d11adf3c gui: Remove unused variables (#5665) 2019-04-23 18:36:12 +01:00
Jonas Thelemann
afeb606b5b gui: Use of AngularJS markup in URL-valued attribute (#5667)
Using AngularJS markup (that is, AngularJS expressions enclosed in double curly braces) in HTML attributes that reference URLs is not recommended: the browser may attempt to fetch the URL before the AngularJS compiler evaluates the markup, resulting in a request for an invalid URL.
2019-04-23 18:33:40 +01:00
Jonas Thelemann
c6a179fa4d cmd/strelaypoolsrv: Missing explicit dependency injection (#5669)
https://lgtm.com/rules/1505800326162/
2019-04-23 12:17:27 +01:00
Simon Frei
e302ccf4b4 lib/fs: Improve IsParent (#5658) 2019-04-22 11:12:32 +02:00
Simon Frei
926e9228ed lib/scanner, lib/model: File -> item when logging error (#5664) 2019-04-21 16:19:59 +01:00
Simon Frei
86e72d9973 lib/model: Use RLock and legacy polish (#5663) 2019-04-21 13:21:36 +01:00
otbutz
952c8becf5 gui: Adjust table column width to content size (fixes #4531) (#5565) 2019-04-18 18:35:44 +02:00
Jakob Borg
32ee8d783d gui, man, authors: Update docs, translations, and contributors 2019-04-17 07:45:24 +02:00
Simon Frei
3bea59b0d9 lib/model: Refactor progressEmitter to de-/activate by config (fixes #4613) (#5623) 2019-04-13 14:20:51 +02:00
Simon Frei
8a4b65b937 gui: Fix setting page size on failed and locally changed modals (fixes #5421) (#5650) 2019-04-13 14:05:39 +02:00
Simon Frei
fca895a632 lib/model: Fix block index calculation for recheckFile (fixes #5649) (#5648) 2019-04-12 15:21:07 +02:00
Simon Frei
79360e2205 lib/fs: Add test that symlinks are skipped on walk (#5644) 2019-04-10 22:36:37 +02:00
Simon Frei
9b2a73f9ab lib/model: Pause pull for at least as long as failed pull took (fixes #5641) (#5643) 2019-04-10 16:22:23 +02:00
Simon Frei
c305265c62 lib/model: Request errors conforming to BEP specs (#5642) 2019-04-10 11:47:24 +02:00
Jakob Borg
1954239ffa gui, man, authors: Update docs, translations, and contributors 2019-04-10 07:45:25 +02:00
Simon Frei
ca823bd591 lib/fs: When watching remove \\?\ for drive letters (fixes #5578) (#5633) 2019-04-09 09:02:04 +02:00
Jakob Borg
eabd972667 docker: Update build image version 2019-04-07 16:13:55 +02:00
Simon Frei
395e524e2d lib/model: Update db on scan/pull in folder (#5608) 2019-04-07 13:29:17 +02:00
Jakob Borg
48b1a2b264 gui, man, authors: Update docs, translations, and contributors 2019-04-03 07:45:31 +02:00
Evgeny Kuznetsov
58953d799c gui: Add licensing information to aboutModalView (fixes #1223) (#5605) 2019-03-28 07:49:29 +01:00
Jakob Borg
f0f8bf7784 lib/config: Round times stored for pending folders/devices (fixes #5554) 2019-03-27 20:35:42 +01:00
Jakob Borg
bf3834e367 lib/protocol: Use constants instead of init time hashing (fixes #5624) (#5625)
This constructs the map of hashes of zero blocks from constants instead
of calculating it at startup time. A new test verifies that the map is
correct.
2019-03-27 20:20:30 +01:00
Simon Frei
8d1eff7e41 lib/model: Missing queue.Done and reset between pulls (fixes #5332) (#5626) 2019-03-27 20:19:35 +01:00
Simon Frei
0cff66fcbc lib/model: Optimize puller for meta only changes (#5622) 2019-03-27 08:36:58 +00:00
Jakob Borg
d23e8be39f gui, man, authors: Update docs, translations, and contributors 2019-03-27 07:45:25 +01:00
Simon Frei
43a5be1c4b lib/model: Send item finished even after deregistering (fixes #5362) (#5620) 2019-03-26 21:31:33 +01:00
Simon Frei
b50039a920 cmd/syncthing, lib/api: Separate api/gui into own package (ref #4085) (#5529)
* cmd/syncthing, lib/gui: Separate gui into own package (ref #4085)

* fix tests

* Don't use main as interface name (make old go happy)

* gui->api

* don't leak state via locations and use in-tree config

* let api (un-)subscribe to config

* interface naming and exporting

* lib/ur

* fix tests and lib/foldersummary

* shorter URVersion and ur debug fix

* review

* model.JsonCompletion(FolderCompletion) -> FolderCompletion.Map()

* rename debug facility https -> api

* folder summaries in model

* disassociate unrelated constants

* fix merge fail

* missing id assignement
2019-03-26 19:53:58 +00:00
Simon Frei
d4e81fff8a lib/model: Remove unnecessary arguments in pulling functions (#5619) 2019-03-25 12:59:22 +01:00
Jakob Borg
3a557a43cd Merge branch 'release'
* release:
  lib/model: Pass correct file info to deleteItemOnDisk (fixes #5616) (#5617)
  gui: Add missing quote char
2019-03-25 12:58:12 +01:00
Simon Frei
bc53782f88 test: Update conflict integration test (ref #5511) (#5618)
The change in 225c0dda80 (#5511) results in
conflicts being immediately scanned and synced, while the test expects just one
conflict on either side. Change it to expect the same conflict on both sides.
2019-03-25 12:53:06 +01:00
Simon Frei
e4ab9d3312 lib/model: Pass correct file info to deleteItemOnDisk (fixes #5616) (#5617) 2019-03-25 12:43:21 +01:00
Simon Frei
e31a116e6e lib/model: Pass correct file info to deleteItemOnDisk (fixes #5616) (#5617) 2019-03-25 12:42:39 +01:00
Jakob Borg
24f41e169a gui: Add missing quote char 2019-03-25 12:37:10 +01:00
Simon Frei
675f289aef gui: Add new folder state "Failed Items" (fixes #5456) (#5614) 2019-03-22 17:57:53 +00:00
Simon Frei
e7ae851900 lib/model: Debug and test fixes (#5613) 2019-03-22 14:43:47 +01:00
Jakob Borg
73fa7a6e5b go.mod: Update golang.org/x/sys
There was a vulnerability in salsa20, which we don't use, but better
safe than sorry.
2019-03-21 08:08:15 +01:00
Simon Frei
1a6d023ba8 lib/model: Run recvonly tests in temporary dirs (#5587) 2019-03-20 17:38:03 +00:00
Jakob Borg
9207535028 gui: Add missing quote char 2019-03-14 07:37:18 +01:00
Jakob Borg
24967e99a7 gui, man, authors: Update docs, translations, and contributors 2019-03-13 07:45:25 +01:00
Simon Frei
50d8c43e7c lib/config: Set UseLargeBlocks to true by default (fixes #5599) (#5600) 2019-03-12 12:59:26 +00:00
Evgeny Kuznetsov
90d9b2de2b cmd/syncthing: Add a check for particular far-future version of config (fixes #1101) 2019-03-12 08:20:47 +00:00
Evgeny Kuznetsov
04f05f102d cmd: Add check for newer config file and an option to override it (fixes #4921) (#5597)
* Add check for newer config file and override option

* Expanded error message

* Polish previous commits

* Make it newER
2019-03-12 07:12:08 +00:00
Simon Frei
289a02e994 lib/model: Integrate stat refs in folder (#5596) 2019-03-11 16:57:21 +00:00
georgespatton
84fe285659 etc: Systemd unit should declare after=multiuser.target (fixes #5346) (#5593) 2019-03-11 15:50:34 +01:00
Simon Frei
445637ebec lib/model: Pass fset & ignores on folder creation (#5592) 2019-03-11 07:28:54 +01:00
Simon Frei
3f3d2c814b lib/model: Remove unused code (#5591) 2019-03-10 17:05:39 +01:00
Simon Frei
189e44488e lib/model: Introduce must test utility (#5586)
* lib/model: Introduce must test utility

* nice
2019-03-09 18:45:36 +00:00
Simon Frei
27ff20faa3 lib/model: Introduce waitForState test utility (#5585)
* lib/model: Introduce waitForState test utility

* folder id as param to waitForState
2019-03-09 10:36:55 +00:00
Simon Frei
b1564e53e4 lib/model: Improve test utilities (#5584) 2019-03-08 20:29:09 +00:00
Simon Frei
3a75b63776 lib/model: Use temp dir from osutils in tests (#5581) 2019-03-07 16:34:41 +01:00
Simon Frei
8e238c8e48 lib/model: Check before replacing existing file on pull (fixes #5571) (#5567) 2019-03-07 15:15:14 +01:00
Jakob Borg
3d5af675db gui, man, authors: Update docs, translations, and contributors 2019-03-06 07:45:23 +01:00
Jakob Borg
9da3273eb8 lib/model: Clarify fileInfoBatch.flushIfFull criteria 2019-03-05 21:34:04 +01:00
Simon Frei
bd37f6da17 lib/model: Optimize dbUpdaterRoutine (#5576) 2019-03-05 21:32:37 +01:00
Simon Frei
6940d79f5b lib/model: Use errors.Wrap for pull errors (#5563) 2019-03-04 13:01:52 +00:00
Simon Frei
0f80318ef6 lib/scanner: Consistenlty use CreateFileInfo and remove outdated comment (#5574) 2019-03-04 13:27:33 +01:00
Simon Frei
d6622b1f68 lib/model: setUp -> setup (#5573) 2019-03-04 12:27:10 +00:00
Simon Frei
43bcb3d5a5 lib/model: Refactor conflict name handling (#5572) 2019-03-04 12:20:40 +00:00
Evgeny Kuznetsov
e2e8f6e940 gui: Update copyright notices (fixes #5569) (#5570) 2019-03-03 13:15:17 +01:00
Jakob Borg
88b0ce892d gui, man, authors: Update docs, translations, and contributors 2019-02-27 07:45:23 +01:00
otbutz
55cd4b3d9b gui: Use handshake icon for "Introduced by" (fixes #5560) (#5561) 2019-02-26 14:18:35 +01:00
Jakob Borg
f24676ba5a lib/tlsutil: Enable TLS 1.3 when available, on test builds (fixes #5065) (#5558)
* lib/tlsutil: Enable TLS 1.3 when available, on test builds (fixes #5065)

This enables TLS 1.3 negotiation on Go 1.12 by setting the GODEBUG
variable. For now, this just gets enabled on test versions (those with a
dash in the version number).

Users wishing to enable this on production builds can set GODEBUG
manually.

The string representation of connections now includes the TLS version
and cipher suite. This becomes part of the log output on connections.
That is, when talking to an old client:

    Established secure connection .../TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

and now potentially:

    Established secure connection .../TLS1.3-TLS_AES_128_GCM_SHA256

(The cipher suite was there previously in the log output, but not the
TLS version.)

I also added this info as a new Crypto() method on the connection, and
propagate this out to the API and GUI, where it can be seen in the
connection address hover (although with bad word wrapping sometimes).

* wip

* wip
2019-02-26 11:49:02 +01:00
Simon Frei
722b3fce6a all: Hide implementations behind interfaces for mocked testing (#5548)
* lib/model: Hide implementations behind interfaces for mocked testing

* review
2019-02-26 08:09:25 +00:00
Jonas Thelemann
8a05492622 docs: Correct Docker README (#5480) (#5545)
Plus some formatting.
2019-02-26 00:37:59 +04:00
Jakob Borg
63c4e7f6d0 Merge branch 'release'
* release:
  lib/scanner: Use standard adler32 when we don't need rolling (#5556)
2019-02-25 19:36:41 +01:00
Audrius Butkevicius
f0f79a3e3e lib/scanner: Use standard adler32 when we don't need rolling (#5556)
* lib/scanner: Use standard adler32 when we don't need rolling

Seems the rolling adler32 implementation is super slow when executed on large blocks, even tho I can't explain why.

BenchmarkFind1MFile-16    				     100	  18991667 ns/op	  55.21 MB/s	  398844 B/op	      20 allocs/op
BenchmarkBlock/adler32-131072/#00-16     		     200	   9726519 ns/op	1078.06 MB/s	 2654936 B/op	     163 allocs/op
BenchmarkBlock/bozo32-131072/#00-16      		      20	  73435540 ns/op	 142.79 MB/s	 2654928 B/op	     163 allocs/op
BenchmarkBlock/buzhash32-131072/#00-16   		      20	  61482005 ns/op	 170.55 MB/s	 2654928 B/op	     163 allocs/op
BenchmarkBlock/buzhash64-131072/#00-16   		      20	  61673660 ns/op	 170.02 MB/s	 2654928 B/op	     163 allocs/op
BenchmarkBlock/vanilla-adler32-131072/#00-16         	     300	   4377307 ns/op	2395.48 MB/s	 2654935 B/op	     163 allocs/op
BenchmarkBlock/adler32-16777216/#00-16               	       2	 544010100 ns/op	  19.27 MB/s	   65624 B/op	       5 allocs/op
BenchmarkBlock/bozo32-16777216/#00-16                	       1	4678108500 ns/op	   2.24 MB/s	51970144 B/op	      24 allocs/op
BenchmarkBlock/buzhash32-16777216/#00-16             	       1	3880370700 ns/op	   2.70 MB/s	51970144 B/op	      24 allocs/op
BenchmarkBlock/buzhash64-16777216/#00-16             	       1	3875911700 ns/op	   2.71 MB/s	51970144 B/op	      24 allocs/op
BenchmarkBlock/vanilla-adler32-16777216/#00-16       	     300	   4010279 ns/op	2614.72 MB/s	   65624 B/op	       5 allocs/op
BenchmarkRoll/adler32-131072/#00-16                  	    2000	    974279 ns/op	 134.53 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/bozo32-131072/#00-16                   	    2000	    791770 ns/op	 165.54 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/buzhash32-131072/#00-16                	    2000	    917409 ns/op	 142.87 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/buzhash64-131072/#00-16                	    2000	    881125 ns/op	 148.76 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/adler32-16777216/#00-16                	      10	 124000400 ns/op	 135.30 MB/s	 7548937 B/op	       0 allocs/op
BenchmarkRoll/bozo32-16777216/#00-16                 	      10	 118008080 ns/op	 142.17 MB/s	 7548928 B/op	       0 allocs/op
BenchmarkRoll/buzhash32-16777216/#00-16              	      10	 126794440 ns/op	 132.32 MB/s	 7548928 B/op	       0 allocs/op
BenchmarkRoll/buzhash64-16777216/#00-16              	      10	 126631960 ns/op	 132.49 MB/s	 7548928 B/op	       0 allocs/op

* Update benchmark_test.go

* gofmt

* fixup benchmark
2019-02-25 19:25:08 +01:00
Audrius Butkevicius
fafd30f804 lib/scanner: Use standard adler32 when we don't need rolling (#5556)
* lib/scanner: Use standard adler32 when we don't need rolling

Seems the rolling adler32 implementation is super slow when executed on large blocks, even tho I can't explain why.

BenchmarkFind1MFile-16    				     100	  18991667 ns/op	  55.21 MB/s	  398844 B/op	      20 allocs/op
BenchmarkBlock/adler32-131072/#00-16     		     200	   9726519 ns/op	1078.06 MB/s	 2654936 B/op	     163 allocs/op
BenchmarkBlock/bozo32-131072/#00-16      		      20	  73435540 ns/op	 142.79 MB/s	 2654928 B/op	     163 allocs/op
BenchmarkBlock/buzhash32-131072/#00-16   		      20	  61482005 ns/op	 170.55 MB/s	 2654928 B/op	     163 allocs/op
BenchmarkBlock/buzhash64-131072/#00-16   		      20	  61673660 ns/op	 170.02 MB/s	 2654928 B/op	     163 allocs/op
BenchmarkBlock/vanilla-adler32-131072/#00-16         	     300	   4377307 ns/op	2395.48 MB/s	 2654935 B/op	     163 allocs/op
BenchmarkBlock/adler32-16777216/#00-16               	       2	 544010100 ns/op	  19.27 MB/s	   65624 B/op	       5 allocs/op
BenchmarkBlock/bozo32-16777216/#00-16                	       1	4678108500 ns/op	   2.24 MB/s	51970144 B/op	      24 allocs/op
BenchmarkBlock/buzhash32-16777216/#00-16             	       1	3880370700 ns/op	   2.70 MB/s	51970144 B/op	      24 allocs/op
BenchmarkBlock/buzhash64-16777216/#00-16             	       1	3875911700 ns/op	   2.71 MB/s	51970144 B/op	      24 allocs/op
BenchmarkBlock/vanilla-adler32-16777216/#00-16       	     300	   4010279 ns/op	2614.72 MB/s	   65624 B/op	       5 allocs/op
BenchmarkRoll/adler32-131072/#00-16                  	    2000	    974279 ns/op	 134.53 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/bozo32-131072/#00-16                   	    2000	    791770 ns/op	 165.54 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/buzhash32-131072/#00-16                	    2000	    917409 ns/op	 142.87 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/buzhash64-131072/#00-16                	    2000	    881125 ns/op	 148.76 MB/s	     270 B/op	       0 allocs/op
BenchmarkRoll/adler32-16777216/#00-16                	      10	 124000400 ns/op	 135.30 MB/s	 7548937 B/op	       0 allocs/op
BenchmarkRoll/bozo32-16777216/#00-16                 	      10	 118008080 ns/op	 142.17 MB/s	 7548928 B/op	       0 allocs/op
BenchmarkRoll/buzhash32-16777216/#00-16              	      10	 126794440 ns/op	 132.32 MB/s	 7548928 B/op	       0 allocs/op
BenchmarkRoll/buzhash64-16777216/#00-16              	      10	 126631960 ns/op	 132.49 MB/s	 7548928 B/op	       0 allocs/op

* Update benchmark_test.go

* gofmt

* fixup benchmark
2019-02-25 13:29:31 +04:00
Simon Frei
ad5a046843 lib/fs: Rename fsFile* to basicFile* (#5546) 2019-02-24 18:02:02 +01:00
Jakob Borg
4b8853bfde gui, man, authors: Update docs, translations, and contributors 2019-02-20 07:45:22 +01:00
Evgeny Kuznetsov
648fcf2c45 etc: Remove unnecessary quotes in .desktop (#5540) 2019-02-15 07:35:19 +00:00
Simon Frei
ca3ae64bbf lib/db: Flush batch based on size and refactor (fixes #5531) (#5536)
Flush the batch when exceeding a certain size, instead of when reaching a number
of batched operations.
Move batch to lowlevel to be able to use it in NamespacedKV.
Increase the leveldb memory buffer from 4 to 16 MiB.
2019-02-14 23:15:13 +00:00
Simon Frei
e2204d0071 build: Add option to get test coverage (#5539) 2019-02-14 22:38:47 +00:00
Simon Frei
d5ff2c41dc all: Get rid of fatal logging (#5537)
* cleanup Fatal in lib/config/config.go

* cleanup Fatal in lib/config/folderconfiguration.go

* cleanup Fatal in lib/model/model.go

* cleanup Fatal in cmd/syncthing/monitor.go

* cleanup Fatal in cmd/syncthing/main.go

* cleanup Fatal in lib/api

* remove Fatal methods from logger

* lowercase in errors.Wrap

* one less channel
2019-02-14 20:29:14 +00:00
Simon Frei
7a40c42e8b cmd/syncthing: Introduce exiter to handle termination (#5532) 2019-02-13 23:07:27 +00:00
Simon Frei
905c3594b0 lib/model: Various model test fixes and polish (#5528)
* lib/model: Various model test fixes and polish

Missing calls to m.Stop()
Don't fail test if temporary test dir cleanup fails

* drop lazyness
2019-02-13 18:54:04 +00:00
Jakob Borg
5fd333e4f7 gui, man, authors: Update docs, translations, and contributors 2019-02-13 07:45:23 +01:00
Simon Frei
225c0dda80 lib/model: Scan conflicts after creation (#5511)
Also unflakes and improve TestRequestRemoteRenameChanged.
2019-02-12 16:05:20 +01:00
Simon Frei
5fd2cab102 lib/model: Run more tests in tmp dir (#5527) 2019-02-12 16:04:04 +01:00
Simon Frei
4299af1c63 lib/config, lib/model: Use path from locations to check disk space for db (#5525) 2019-02-12 12:25:11 +00:00
Simon Frei
d85ef949be lib/model: Introduce setupModel test utility (#5524) 2019-02-12 12:18:13 +00:00
Audrius Butkevicius
dc929946fe all: Use new reflect based CLI (#5487) 2019-02-12 07:58:24 +01:00
Simon Frei
7bac927ac8 lib/model: Use functions to generate config (#5513) 2019-02-12 07:50:07 +01:00
Matt Robenolt
93b4597d1a gui: Localize items counts (#5520) 2019-02-12 07:49:14 +01:00
Evgeny Kuznetsov
0928628e7b etc: Add keywords to .desktop files (fixes #5365) (#5521) 2019-02-11 14:33:05 +01:00
Audrius Butkevicius
c5a79bdfe6 Fix builds on Windows without CGO (#5518) 2019-02-10 15:58:29 +01:00
Jakob Borg
04fdafa280 lib/db, lib/model: Remove dead code (#5517) 2019-02-08 16:42:58 +01:00
Jakob Borg
cb49136269 gui: Add missing translation string (fixes #5515) 2019-02-07 07:30:22 +01:00
Simon Frei
2f415d8f09 lib/model: Don't use LocalDeviceID as normal id in tests (#5512) 2019-02-06 09:32:03 +01:00
Jakob Borg
b076031bfa gui, man, authors: Update docs, translations, and contributors 2019-02-06 07:45:24 +01:00
Simon Frei
e538797ce1 lib/model: Improve TestIssue5063 (#5509)
* lib/model: Improve TestIssue5063

* add not that helpful issue comment
2019-02-05 23:07:21 +00:00
Simon Frei
5df8219bcb lib/model: Add progressEmitter to supervisor (model) (#5510) 2019-02-05 18:02:36 +00:00
Simon Frei
af4fb97538 lib/model: Fail test instead of panic due to closing channel twice (#5508) 2019-02-05 18:01:56 +00:00
Simon Frei
5d9d87f770 lib/model: Helperize test os and remove error return value (#5507) 2019-02-05 18:01:05 +00:00
Jakob Borg
c75cfcfd06 gui: Enable large blocks by default, add to config dialog (#5405)
Also moves a couple of </div> that were out of balance and reindents
accordingly, so try a white space insensitive diff when reviewing...
2019-02-02 13:06:01 +01:00
Jakob Borg
6cdd1c5158 cmd/syncthing: Fixup previous commit 2019-02-02 12:54:26 +01:00
Simon Frei
41d037da1f build: Remove outdated&non-functional setup command (fixes #5454) (#5455) 2019-02-02 12:47:46 +01:00
Simon Frei
82afe73a9a cmd/syncthing, lib/config: Update default config creation (#5492)
Also remove dead code in config.Wrapper.
2019-02-02 12:43:57 +01:00
Jakob Borg
6452e16f15 golangci: Add config file 2019-02-02 12:34:57 +01:00
Iskander (Alex) Sharipov
ca47b4c218 cmd/syncthing: Correct strings.HasPrefix args order (#5498) 2019-02-02 12:18:19 +01:00
Jakob Borg
c2ddc83509 all: Revert the underscore sillyness 2019-02-02 12:16:27 +01:00
Jakob Borg
9fd270d78e all: A few more interesting linter fixes (#5502)
A couple of minor bugs and simplifications
2019-02-02 12:09:07 +01:00
Jakob Borg
0b2cabbc31 all: Even more boring linter fixes (#5501) 2019-02-02 11:45:17 +01:00
Jakob Borg
df5c1eaf01 all: Bunch of more linter fixes (#5500) 2019-02-02 11:02:28 +01:00
Jakob Borg
2111386ee4 all: Fix some linter errors (#5499)
I'm working through linter complaints, these are some fixes. Broad
categories:

1) Ignore errors where we can ignore errors: add "_ = ..." construct.
you can argue that this is annoying noise, but apart from silencing the
linter it *does* serve the purpose of highlighting that an error is
being ignored. I think this is OK, because the linter highlighted some
error cases I wasn't aware of (starting CPU profiles, for example).

2) Untyped constants where we though we had set the type.

3) A real bug where we ineffectually assigned to a shadowed err.

4) Some dead code removed.

There'll be more of these, because not all packages are fixed, but the
diff was already large enough.
2019-02-02 10:11:42 +01:00
Simon Frei
583172dc8d lib/db: Fix race in NamespacedKV (#5496) 2019-02-01 09:54:21 +01:00
Jakob Borg
1529563332 docker: Build outside GOPATH (fixes #5495) 2019-01-31 20:38:33 +01:00
Simon Frei
5605877625 cmd/syncthing: Pass SIGTERM on in monitor (fixes #5493) (#5494) 2019-01-31 18:35:20 +01:00
Simon Frei
7236d56731 lib/model: In tests disable watching for changes by default (fixes #5246) (#5485) 2019-01-30 16:38:10 +01:00
Jakob Borg
47d68a0aec gui, man, authors: Update docs, translations, and contributors 2019-01-30 07:45:27 +01:00
Simon Frei
657be162dd test, lib/rc: Integration test fixes and polish (#5488) 2019-01-29 16:59:00 +01:00
Simon Frei
8815ef922b mod: Update dependencies and tidy (fixes #5311) (#5486) 2019-01-28 22:45:56 +01:00
Simon Frei
79d109a386 lib/config: Add omitempty to DeprecatedMinHomeDiskFreePct (fixes #5482) (#5484) 2019-01-28 11:46:28 +01:00
Jakob Borg
75dcff0a0e all: Copy owner/group from parent (fixes #5445) (#5479)
This adds a folder option "CopyOwnershipFromParent" which, when set,
makes Syncthing attempt to retain the owner/group information when
syncing files. Specifically, at the finisher stage we look at the parent
dir to get owner/group and then attempt a Lchown call on the temp file.
For this to succeed Syncthing must be running with the appropriate
permissions. On Linux this is CAP_FOWNER, which can be granted by the
service manager on startup or set on the binary in the filesystem. Other
operating systems do other things, but often it's not required to run as
full "root". On Windows this patch does nothing - ownership works
differently there and is generally less of a deal, as permissions are
inherited as ACLs anyway.

There are unit tests on the Lchown functionality, which requires the
above permissions to run. There is also a unit test on the folder which
uses the fake filesystem and hence does not need special permissions.
2019-01-25 09:52:21 +01:00
Jakob Borg
0e07f6bef4 vendor: rm -rf vendor (#5478)
This removes our vendored dependencies. They provide no value for our
own build or development processes. For our source releases, the build
job can accomplish the same thing by a "go mod vendor" to recreate the
vendor dir (from the cryptographically verified dependencies).
2019-01-24 09:03:00 +01:00
Simon Frei
a45ba70467 lib/model: Improve errors while pulling (#5474) 2019-01-24 08:18:55 +01:00
Simon Frei
42bd42df5a lib/db: Do all update operations on a single item at once (#5441)
To do so the BlockMap struct has been removed. It behaves like any other prefixed
part of the database, but was not integrated in the recent keyer refactor. Now
the database is only flushed when files are in a consistent state.
2019-01-23 10:22:33 +01:00
Jakob Borg
6421693cce gui, man, authors: Update docs, translations, and contributors 2019-01-23 07:45:23 +01:00
Simon Frei
a371b15398 lib/db: Various polish (#5471)
naming: buf -> keyBuf
dedup: use getFileTrunc
manually inline insertFile
2019-01-20 10:24:39 +01:00
Jakob Borg
29e4b417f2 vendor: Update vendor dir from a80c0fda 2019-01-20 08:50:40 +01:00
Simon Frei
00fa77dd47 lib/db: Consistent use of buffers (#5470) 2019-01-20 08:47:20 +01:00
Simon Frei
df4d754197 lib/db: Minor polish (#5469) 2019-01-19 20:26:46 +01:00
Simon Frei
f3d735c56a lib/db: Fix, optimize and extend benchmarks (#5467) 2019-01-19 20:24:44 +01:00
Simon Frei
1d99db9bc6 lib/db: Deduplicate getFile* code (#5468) 2019-01-19 20:21:58 +01:00
Audrius Butkevicius
96bd691f55 lib/fs: Skip some tests on OpenBSD (fixes #5077) (#5466) 2019-01-19 08:28:57 +01:00
Simon Frei
22e133cce6 lib/db: Deduplicate comparing old and new items (#5465) 2019-01-18 21:19:56 +00:00
Audrius Butkevicius
a80c0fdae8 vendor: Update minio/sha256-simd 2019-01-18 20:30:40 +00:00
Simon Frei
1f87b874af lib/db: Add "dirty" function terminology to getGlobal (ref #5462) (#5463) 2019-01-18 13:01:39 +01:00
Jakob Borg
1e69997ecd lib/db: Fix iterating sequence index (fixes #5340) (#5462)
There was a problem in iterating the sequence index that could result
in missing updates. The issue is that while the index was (correctly)
iterated in a snapshot, the actual file infos were read dirty outside of
the snapshot. This fixes this by doing the reads inside the snapshot,
and also updates a couple of other places that did the same thing more
or less harmfully (I didn't investigate).

To avoid similar issues in the future I did some renaming of the
getFile* methods - the ones in a transaction are just getFile, while the
ones directly on the database are variants of getFileDirty to highlight
what's going on.
2019-01-18 11:34:18 +01:00
Jakob Borg
76af0cf07b lib/protocol: Remove support for v0.13 hello messages (#5461) 2019-01-17 20:48:43 +01:00
Jakob Borg
c2cc1dadea gui, man, authors: Update docs, translations, and contributors 2019-01-16 07:45:23 +01:00
Jakob Borg
f4bf18c1b4 cmd/syncthing, gui: Settings dialog upgrades/reporting for candidates (fixes #4216) (#5457)
This adds booleans to the /system/version response to advice the GUI
whether the running version is a candidate release or not. (We could
parse it from the version string, but why duplicate the logic.)

Additionally the settings dialog locks down the upgrade and usage
reporting options on candidate releases. This matches the current
behavior, it just makes it obvious what actually *can* be chosen.
2019-01-15 08:44:46 +01:00
Jakob Borg
b01edca420 all: Update protobuf package 1.0.0 -> 1.2.0 (#5452)
Also adds a few file global options to keep the generated code similar
to what we already had.
2019-01-14 11:53:36 +01:00
Simon Frei
c87411851d lib/protocol: Fix potential deadlock when closing connection (ref #5440) (#5442) 2019-01-14 08:32:37 +01:00
Simon Frei
51f65bd23a lib/model: Reset pull errors when succeeding (fix #5450) (#5451) 2019-01-14 08:30:52 +01:00
Audrius Butkevicius
801e9b57eb Update config (#5444)
* gui: Update config from remote after saving (fixes #5354)

* Refresh is async :(
2019-01-13 23:28:17 +00:00
Jakob Borg
93ae9a1c2e lib/logger: Clean up LOGGER_DISCARD handing to unbreak tests
When using newLogger() with an io.Writer, that writer should in fact be
used...
2019-01-13 10:22:01 +01:00
Audrius Butkevicius
d4f0544d9e build: Add -cc argument that is not used by build.go (#5447) 2019-01-11 23:07:53 +00:00
Simon Frei
0b03b6a9ec lib/model: Improve filesystem operations during tests (fixes #5422)
* lib/fs, lib/model: Improve filesystem operations during tests (fixes #5422)

Introduces MustFilesystem that panics on errors and should be used for operations
during testing which must never fail.
Create temporary directories outside of testdata.

* don't do a filesystem, just a wrapper around os for testing

* fix copyright
2019-01-11 12:56:05 +00:00
Simon Frei
24ffd8be99 all: Send Close BEP msg on intentional disconnect (#5440)
This avoids waiting until next ping and timeout until the connection is actually
closed both by notifying the peer of the disconnect and by immediately closing
the local end of the connection after that. As a nice side effect, info level
logging about dropped connections now have the actual reason in it, not a generic
timeout error which looks like a real problem with the connection.
2019-01-09 17:31:09 +01:00
Jakob Borg
d924bd7bd9 gui, man, authors: Update docs, translations, and contributors 2019-01-09 07:45:23 +01:00
Jakob Borg
1e71b00936 lib/model: Sanitize paths used for auto accepted folders (fixes #5411) (#5435) 2019-01-05 18:10:02 +01:00
Jakob Borg
f3630a69f1 gui: Fix bar chart icon (fixes #5389) 2019-01-05 11:36:25 +01:00
Jakob Borg
5503175854 lib/logger: Strip control characters from log output (fixes #5428) (#5434) 2019-01-05 11:31:02 +01:00
Audrius Butkevicius
ad30192dca vendor: Update minio/sha256-simd (#5433)
* vendor: Update minio/sha256-simd

* Add go module stuff
2019-01-05 10:21:42 +01:00
Simon Frei
158559023e lib/db: Fix sequence updating for remote invalid items (#5420)
* lib/db: Fix sequence updating for remote invalid items

* fix for the unit test introduced in the previous commit

* lib/db: Polish blockmap
2019-01-04 20:19:10 +01:00
Jakob Borg
04070b4848 lib/logger: Add test for stack level (#5430) 2019-01-04 17:51:58 +01:00
Jakob Borg
597cee67d3 cmd/ur*: Updates for 1.0 2019-01-03 21:46:02 +01:00
Jakob Borg
78ccedfeec gui, man, authors: Update docs, translations, and contributors 2019-01-02 07:45:22 +01:00
Ben S
69fe471dfa gui: Make sync status accessible by screen readers (fixes #2697) (#5381) 2019-01-01 10:18:08 +01:00
Simon Frei
47e08797cb lib/model: Don't pull if ignores failed to load and cleanup (#5418) 2019-01-01 10:17:14 +01:00
Jakob Borg
48dab8f201 Merge branch 'release'
* release:
  cmd/syncthing: Updated code name
2019-01-01 09:38:30 +01:00
Simon Frei
643cfe2e98 lib/config: Revert #5415 (#5417)
This reverts commit 9d075781ad:
"cmd/syncthing: Improve messages when free space is running out (#5415)"
2018-12-30 21:57:41 +01:00
Simon Frei
8bb9878f26 lib/model: Check folder context before setting error state (#5416) 2018-12-30 21:56:16 +01:00
Maurizio Tomasi
9d075781ad cmd/syncthing: Improve messages when free space is running out (#5415) 2018-12-29 20:48:20 +01:00
Hugo Locurcio
2ad40734f8 gui: Use a modern font stack inspired by Bootstrap 4 (#5407)
See https://github.com/twbs/bootstrap/pull/19098 for more information.
2018-12-26 23:05:20 +01:00
Jakob Borg
952ab7db1c cmd/syncthing: Updated code name 2018-12-26 17:23:05 +01:00
Jakob Borg
84ca86f095 gui, man, authors: Update docs, translations, and contributors 2018-12-26 07:45:23 +01:00
Jakob Borg
0ac6ea6f1e gui: Reformat HTML & JS for consistent white space
(white space only change)
2018-12-25 14:26:46 +01:00
Jakob Borg
41469c5393 lib/rc: Remove dead lock (fixes #5403) 2018-12-23 09:11:15 +01:00
Simon Frei
4783294a09 lib/model: Don't pass nil *Matcher to performFinish (fixes #5401) (#5402) 2018-12-22 21:58:17 +01:00
Jakob Borg
c8123bda28 cmd/syncthing: Don't rewrite config on startup unless necessary (#5399) 2018-12-21 14:26:36 +00:00
Simon Frei
99c9d65ddf lib/scanner: Check ignore patterns before reporting error (fixes #5397) (#5398) 2018-12-21 12:08:15 +01:00
Simon Frei
fc81e2b3d7 lib/model: Do folder watch operations under lock (fixes #5392) (#5395) 2018-12-21 12:06:21 +01:00
Simon Frei
c3b3e02f21 test: Update configs and revert changes during testing (#5393) 2018-12-21 11:50:28 +01:00
Simon Frei
2626143fc5 lib/rc: Fix hangups when Syncthing process ends unexpectedly (#5383) 2018-12-21 11:49:04 +01:00
Stefan Tatschner
ae0dfcd7ca build: Allow specifying the go command (#5396) 2018-12-20 15:24:35 +01:00
Jakob Borg
58f3e56729 gui, man, authors: Update docs, translations, and contributors 2018-12-19 07:45:22 +01:00
Jakob Borg
944ddcf768 all: Become a Go module (fixes #5148) (#5384)
* go mod init; rm -rf vendor

* tweak proto files and generation

* go mod vendor

* clean up build.go

* protobuf literals in tests

* downgrade gogo/protobuf
2018-12-18 12:36:38 +01:00
Simon Frei
3cc8918eb4 lib/db: Defer unlock to avoid hangup on panic (#5388) 2018-12-17 14:59:09 +01:00
Simon Frei
c40c9a8d6a lib/scanner: Don't report error on missing items (fixes #5385) (#5387) 2018-12-17 14:52:15 +01:00
Jakob Borg
abb3fb8a31 lib/events: Become a service (fixes #5372) (#5373)
Here the event Logger is rewritten as a service with a main loop instead
of mutexes. This loop has a select with essentially two legs: incoming
events, and subscription changes. When both are possible select will
chose one randomly, thus ensuring that in practice unsubscribes will
happen timely and not block the system.
2018-12-13 13:42:28 +01:00
Jakob Borg
fc860df514 lib/watchaggregator: Properly unsubscribe from events when stopping (ref #5372) (#5374) 2018-12-13 08:11:51 +01:00
Simon Frei
c934918347 lib/scanner: Fix empty path on normalization error (fixes #5369) (#5370) 2018-12-12 12:09:39 +01:00
Jakob Borg
c98a34a5d4 gui, man, authors: Update docs, translations, and contributors 2018-12-12 07:45:23 +01:00
Jörg Thalheim
bdcffe703b build: Don't use go build -i (#5367) 2018-12-11 13:37:46 +01:00
Simon Frei
a09079ed25 all: Display list of locally changed items in UI (fixes #5336) (#5337) 2018-12-11 09:59:04 +01:00
Jakob Borg
1b59960ff9 lib/model: Attempt to unflake Deregister tests (fixes #5362) (#5366) 2018-12-11 09:42:03 +01:00
Jakob Borg
f04d054b5a lib/db: Document current keyspace 2018-12-10 09:55:21 +01:00
desbma
132789785d etc: Add hardening options to syncthing systemd services (fixes #5286) (#5351) 2018-12-07 14:58:12 +01:00
Jakob Borg
002de7b6a0 Merge branch 'release'
* release:
  gui: Don't use newfangled ES6 features (fixes #5348)
2018-12-05 12:24:36 +01:00
Jakob Borg
9e725026d1 gui: Don't use newfangled ES6 features (fixes #5348) 2018-12-05 12:24:27 +01:00
Jakob Borg
da39dfada3 gui: Don't use newfangled ES6 features (fixes #5348) 2018-12-05 12:18:45 +01:00
Audrius Butkevicius
ff2cde469e lib/model: Allow limiting number of concurrent scans (fixes #2760) (#4888) 2018-12-05 08:40:05 +01:00
Jakob Borg
0fe4c01a28 lib/db: Fix broken test on Windows (#5347) 2018-12-05 08:24:14 +01:00
Jakob Borg
351e855464 gui, man, authors: Update docs, translations, and contributors 2018-12-05 07:45:21 +01:00
Jakob Borg
7203ccb73d gui, man, authors: Update docs, translations, and contributors 2018-11-28 07:45:22 +01:00
Jakob Borg
e89c4c053a lib/db: Simple case sensitivity test 2018-11-26 17:58:00 +01:00
Simon Frei
0391c57a37 cmd/syncthing: Improve logged device information on startup (#5321) 2018-11-22 11:39:01 +01:00
Simon Frei
2f9840ddae lib: Introduce fs.IsParent (fixes #5324) (#5326) 2018-11-22 11:16:45 +01:00
Ben S
513d3bc374 gui: Display rate limit (fixes #5320) (#5328)
Display rate limit for own device and connected devices
2018-11-22 10:59:04 +01:00
Jakob Borg
c0a26c918a lib/scanner, vendor: Update github.com/chmduquesne/rollinghash (fixes #5334) (#5335)
Updates the package and fixes a test that depended on the old behavior
of Write() being equivalent to Reset()+Write() which is no longer the
case. The scanner already did resets after each block write, so this is
fine.
2018-11-22 08:50:06 +01:00
Jakob Borg
d1704d5304 gui, man, authors: Update docs, translations, and contributors 2018-11-21 07:45:22 +01:00
Jakob Borg
d3d0161fb9 gui, man, authors: Update docs, translations, and contributors 2018-11-14 07:45:23 +01:00
Simon Frei
db8777c29e lib/model: Add test for #5323 (#5325) 2018-11-13 08:36:16 +00:00
Ben S
ba4554f053 gui: Sharing tab for folders (#5313)
Sharing tab for folders (like sharing tab for devices)
2018-11-13 08:57:45 +01:00
Simon Frei
33bed5b1ec lib/model: Don't compare permissions if IgnorePerms is true (fixes #5323) (#5322) 2018-11-13 08:54:49 +01:00
Simon Frei
4f27bdfc27 lib/model, lib/protocol: Handle request concurrency in model (#5216) 2018-11-13 08:53:55 +01:00
Cromefire_
9212303906 gui: Defer jsTree initialisation until next digest cycle (fixes #4738) 2018-11-11 12:42:53 +00:00
Simon Frei
603da2dce2 cmd/syncthing, lib/relay: Fixes regarding stopping of services (#5293) 2018-11-07 11:05:07 +01:00
Simon Frei
d510e3cca3 all: Display errors while scanning in web UI (fixes #4480) (#5215) 2018-11-07 11:04:41 +01:00
BAHADIR YILMAZ
f51514d0e7 gui: Select / Deselect all folders / devices (#fixes 4000) (#5307) 2018-11-07 09:44:52 +01:00
Jakob Borg
e67be59c5f gui, man, authors: Update docs, translations, and contributors 2018-11-07 07:45:25 +01:00
Benno Fünfstück
add12b43aa cmd/syncthing: Make directory auto-complete case insensitive (fixes #1347) 2018-11-01 19:13:11 +00:00
Simon Frei
aec91d8f32 cmd/ursrv: Add google maps api key (fixes #5296) (#5303) 2018-11-01 06:21:28 +01:00
Simon Frei
53b0f36be6 lib/fs: Use os.FileMode.String for fs.FileMode (#5302) 2018-10-31 12:49:50 +01:00
Jakob Borg
830bde2c83 gui, man, authors: Update docs, translations, and contributors 2018-10-31 07:45:23 +01:00
Jakob Borg
be1744a481 cmd/strelaypoolsrv: Hardcode a usable maps API key (fixes #5296)
Yeah it's not the most beautiful solution but it works for now.
2018-10-31 07:39:38 +01:00
Simon Frei
01ade9c8ae lib/connections: Don't panic on removed device (fixes #5299) (#5300) 2018-10-30 10:34:19 +01:00
Simon Frei
b1acc37c16 lib/db: Update local need on device removal (fixes #5294) (#5295) 2018-10-30 05:40:51 +01:00
Simon Frei
64a591610b lib/model: Check if files from queue are invalid (fixes #5291) (#5292) 2018-10-26 19:13:35 +01:00
Jakob Borg
9a07b22d4a gui, man, authors: Update docs, translations, and contributors 2018-10-24 07:45:24 +02:00
Simon Frei
406bedf1e3 lib/logger: Disable debug flags without debugging (fixes #4780) (#5278) 2018-10-23 15:17:40 +02:00
Simon Frei
7f55fbbe84 gui: Show usage reporting title regardless of RC (#5284) 2018-10-23 17:30:13 +09:00
Alexandre Viau
75f9ea623c cmd: Update prometheus_client (fixes #5280) (#5282) 2018-10-21 16:11:26 +01:00
Alexandre Viau
9745679c63 lib: chmod -x on progressemitter.go and errors.go (#5281) 2018-10-21 16:08:14 +01:00
Jakob Borg
8519a24ba6 cmd/*, lib/tlsutil: Refactor TLS stuff (fixes #5256) (#5276)
This changes the TLS and certificate handling in a few ways:

- We always use TLS 1.2, both for sync connections (as previously) and
  the GUI/REST/discovery stuff. This is a tightening of the requirements
  on the GUI. AS far as I can tell from caniusethis.com every browser from
  2013 and forward supports TLS 1.2, so I think we should be fine.

- We always greate ECDSA certificates. Previously we'd create
  ECDSA-with-RSA certificates for sync connections and pure RSA
  certificates for the web stuff. The new default is more modern and the
  same everywhere. These certificates are OK in TLS 1.2.

- We use the Go CPU detection stuff to choose the cipher suites to use,
  indirectly. The TLS package uses CPU capabilities probing to select
  either AES-GCM (fast if we have AES-NI) or ChaCha20 (faster if we
  don't). These CPU detection things aren't exported though, so the tlsutil
  package now does a quick TLS handshake with itself as part of init().
  If the chosen cipher suite was AES-GCM we prioritize that, otherwise we
  prefer ChaCha20. Some might call this ugly. I think it's awesome.
2018-10-21 14:17:50 +09:00
Simon Frei
c0be9987d0 build: Add desktop files and icons to .deb (fixes #3439) (#5277) 2018-10-20 08:25:59 +02:00
Jakob Borg
c1f1fd71fe cmd/stdiscosrv: Unflake test (fixes #5247) 2018-10-18 20:39:36 +09:00
Jakob Borg
3c657d1749 gui, man, authors: Update docs, translations, and contributors 2018-10-17 07:45:24 +02:00
Nico Stapelbroek
53f80fdf73 Update golint import path (#5274)
Fixes error "code in directory GOPATH/src/github.com/golang/lint/golint expects import golang.org/x/lint/golint" during the
go run build.go setup command.

See https://github.com/golang/lint/issues/415
2018-10-16 19:53:03 +01:00
Simon Frei
6325ae070c cmd/syncthing: Correct type assertion in verboseService (fixes #5270) (#5272) 2018-10-16 01:09:24 +02:00
Simon Frei
089c283ca6 lib/config: Disable folder free disk check when configured (fixes #5267) (#5268) 2018-10-12 12:34:56 +01:00
Simon Frei
8d7ea0424d vendor: Update github.com/syncthing/notify (#5263) 2018-10-12 12:24:25 +02:00
Jakob Borg
f12d5771dc cmd/syncthing: We can use Go 1.8 constants now 2018-10-12 07:55:54 +02:00
Jakob Borg
7b0c49a1b6 cmd/stindex: Add index checking mode ("idxck") (#5262) 2018-10-11 20:48:39 +01:00
Simon Frei
0690fe7585 lib/model: Unnecessary return (#5264) 2018-10-11 15:08:37 +02:00
Jakob Borg
3bc918ff78 lib/db: Properly remove FileInfos when dropping folder (#5260) 2018-10-11 12:09:44 +02:00
Simon Frei
3e50edf46f lib/db: More info in sequence panic msg (#5261) 2018-10-11 12:05:57 +02:00
Simon Frei
1b10607def lib/model: Don't check folder health if there is nothing to pull (fixes #2497) (#5255) 2018-10-11 11:33:21 +02:00
Jakob Borg
9d7a811e72 lib/model: Don't flake out on shortcutting files (ref #5258, #5257, #5234) (#5259)
In a recent change (#5201) this return disappeared. The effect is that
we first shortcut the file and then also treat it normally. This results
in to database updates after each other, which are bound to end up in
the same batch. This means we remove one sequence entry and add two.

Not marking the issues as fixed, because I need to do more testing and
there are other discrepancies...
2018-10-11 11:07:52 +02:00
Jakob Borg
4f7d43a6dd cmd/syncthing: Minor support bundle tweaks
- Name the zip file with the short device ID
- Include the log on disk, if there is one
2018-10-11 08:13:52 +02:00
Jakob Borg
83675e7a1d Merge branch 'release'
* release:
  gui: Close unclosed tag (fixes #5253)
2018-10-11 07:12:49 +02:00
Jakob Borg
34fee05a1d gui: Update device name properly when saving config (fixes #5249) (#5254) 2018-10-10 19:14:23 +01:00
Simon Frei
d10773c311 lib/db, lib/model: Resolve identical recv only items (fixes #5130) (#5230) 2018-10-10 12:43:07 +02:00
Jakob Borg
caa2356409 lib/db: Rename things (ref #5198)
This renames a couple of files to better reflect their current contents,
and moves a type. No lines of code actually changed.
2018-10-10 11:48:21 +02:00
Simon Frei
523ac45456 lib/model: Treat failed rename like del&update (#5203) 2018-10-10 11:37:20 +02:00
Jakob Borg
b50d57b7fd lib/db: Refactor: use a Lowlevel type underneath Instance (ref #5198) (#5212)
This adds a thin type that holds the state associated with the
leveldb.DB, leaving the huge Instance type more or less stateless. Also
moves some keying stuff into the DB package so that other packages need
not know the keying specifics.

(This does not, yet, fix the cmd/stindex program, in order to keep the
diff size down. Hence the keying constants are still exported.)
2018-10-10 11:34:24 +02:00
Jakob Borg
666314a9d9 gui: Close unclosed tag (fixes #5253) 2018-10-10 09:20:06 +02:00
Jakob Borg
8e645ab782 gui: Close unclosed tag (fixes #5253) 2018-10-10 08:16:02 +02:00
Jakob Borg
dfc1101307 gui, man, authors: Update docs, translations, and contributors 2018-10-10 07:45:20 +02:00
Jakob Borg
f12ca95af2 lib/model: Unflake TestFolderRestartZombies (fixes #5244) 2018-10-07 13:58:25 +02:00
Jakob Borg
f528923d1e lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes #5233) (#5239)
* lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes #5233)

This is the somewhat ugly - but on the other hand clear - fix for what
is really a somewhat thorny issue. To avoid zombie folder runners a new
mutex is introduced that protects the RestartFolder operation. I hate
adding more mutexes but the alternatives I can think of are worse.

The other part of it is that the POST /rest/system/config operation now
waits for the config commit to complete. The point of this is that until
the commit has completed we should not accept another config commit. If
we did, we could end up with two separate RestartFolders queued in the
background. While they are both correct, and will run without
interfering with each other, we can't guarantee the order in which they
will run. Thus it could happen that the newer config got committed
first, and the older config commited after that, leaving us with the
wrong config running.

* test

* wip

* hax

* hax

* unflake test

* per folder mutexes

* paranoia

* race
2018-10-05 09:26:25 +01:00
Simon Frei
c9d6366d75 lib/connections: Don't info log about LAN if there are no rate limits (#5242) 2018-10-05 08:22:47 +02:00
Simon Frei
714a47ffb0 lib/config: Add context to the home disk out of space error (#5241) 2018-10-05 08:21:39 +02:00
Peter Badida
86a22b8086 gui: Add missing 'Close' button on remote devices' files (#5232) 2018-10-03 09:00:02 +02:00
Alexandre Viau
a4d27282ae gui: Migrate to fork-awesome (fixes #5236) (#5237) 2018-10-03 08:56:54 +02:00
Jakob Borg
2e425c4386 gui, man, authors: Update docs, translations, and contributors 2018-10-03 07:45:28 +02:00
Jakob Borg
d27463268d lib/fs: Add fakefs (#5235)
* lib/fs: Add fakefs

This adds a new fake filesystem type. It's described rather extensively
in fakefs.go, but the main point is that it's for testing: when you want
to spin up a Syncthing and have a terabyte or two of random files that
can be synced somewhere, or an inifitely large filesystem to sync files
into.

It has pseudorandom properties such that data read from one fakefs can
be written into another fakefs and read back and it will look
consistent, without any of the data actually being stored.

To use:

    <folder id="default" path="whatever" ...>
        <filesystemType>fake</filesystemType>

This will create an empty fake filesystem. You can also specify that it
should be prefilled with files:

    <folder id="default" path="whatever?size=2000000" ...>
        <filesystemType>fake</filesystemType>

This will create a filesystem filled with 2TB of random data that can be
scanned and synced. There are more options, see fakefs.go.

Prefilled data is based on a deterministic seed, so you can index the
data and restart Syncthing and the index is still correct for all the
stored data.
2018-10-02 19:29:06 +01:00
Jakob Borg
3d74ff97af cmd/syncthing: Fix support bundle zip name pattern 2018-10-02 05:32:54 +02:00
BAHADIR YILMAZ
675846ac1e cmd/syncthing, gui: Implement download of "support bundle" (fixes #5142) (#5145) 2018-10-01 17:23:46 +02:00
Simon Frei
c2b0d309fb lib/model: Prevent repeat db update (#5231) 2018-09-27 07:41:40 +02:00
Simon Frei
cb0950b3fd lib/db: Improve VersionList.String (#5229) 2018-09-26 23:30:22 +02:00
Simon Frei
03d0f0dc34 lib/fs: Try EvalSymlinks without '\\?\' prefix on failure (fixes #5226) (#5227) 2018-09-26 19:28:20 +01:00
Jakob Borg
91c3218a0c gui, man, authors: Update docs, translations, and contributors 2018-09-26 07:45:25 +02:00
Jakob Borg
2ced65b9e7 Merge branch 'release'
* release:
  gui: Use right variable for device ID in share dialog (fixes #5213)
2018-09-24 14:58:28 +02:00
Jakob Borg
03821d8bd3 gui: Use right variable for device ID in share dialog (fixes #5213) 2018-09-24 14:50:31 +02:00
Jakob Borg
92405ad1a6 gui: Use right variable for device ID in share dialog (fixes #5213) 2018-09-24 14:49:06 +02:00
Jakob Borg
5a69e85e80 cmd/syncthing: Listen on UNIX socket (fixes #3616) (#5210)
This adds the ability to listen on (only) a UNIX socket.
2018-09-21 14:28:57 +02:00
Jakob Borg
4f034a01ed gui, man, authors: Update docs, translations, and contributors 2018-09-19 07:45:25 +02:00
Jakob Borg
d7bc0659e4 cmd/syncthing: Use lower strength factor in auth tests (fixes #5206) (#5211)
Newly added auth tests uses a high strength factor for bcrypt, which
takes ages under -race. No reason for that.
2018-09-18 21:56:52 +02:00
Simon Frei
4f4781d254 lib/scanner: Centralise protocol.FileInfo creation (#5202) 2018-09-18 15:34:17 +02:00
Jakob Borg
6a87aac84f lib/db: Refactor key handling (ref #5198) (#5199)
This breaks out the key generation stuff into a separate type. It's
cleaner on its own, and it prepares for future stuff.
2018-09-18 10:41:06 +02:00
Jakob Borg
797a999585 test: Terminology only 2018-09-17 12:39:28 +02:00
Simon Frei
272fb3b444 all: Adjust windows perms in fs package (#5200) 2018-09-16 16:09:56 +02:00
Simon Frei
60eb9088ff lib/model: Extend shortcutFile (#5201) 2018-09-16 10:29:06 +01:00
Simon Frei
c8652222ef all: Check files on disk/in db when deleting/renaming (fixes #5194) (#5195) 2018-09-16 09:48:14 +02:00
Jakob Borg
84494edab4 cmd/syncthing: Return "Forbidden" for REST API debug endpoints when debugging disabled
The previous "Bad Request" was really confusing as it implies it's
somethign wrong with the request, which there isn't - the problem is
that server configuration forbids the request.
2018-09-16 09:32:24 +02:00
Boris Rybalkin
b19885fdb3 cmd/syncthing: Add explicit -help flag (fixes #5193) (#5197) 2018-09-15 08:17:31 +02:00
Boris Rybalkin
e39dafb584 gui: LDAP counts as authentication (#5196) 2018-09-15 08:12:00 +02:00
Audrius Butkevicius
710dba7f84 gui: Add arrays for pending/ignored folders (fixes #5190) (#5192) 2018-09-13 22:52:16 +02:00
Simon Frei
a57fa9cfab lib/model: Polish (#5189) 2018-09-13 18:17:13 +02:00
Simon Frei
49d5eae66a lib/config: Actually modify config element (fixes #5185) (#5186) 2018-09-12 12:16:52 +01:00
Jakob Borg
d9d2cf74a0 gui, man, authors: Update docs, translations, and contributors 2018-09-12 07:45:26 +02:00
Boris Rybalkin
1b1741de64 cmd/syncthing: Add LDAP authentication for GUI (fixes #5163) (#5169) 2018-09-11 23:25:24 +02:00
Simon Frei
50ba0fd079 lib/fs: Case insensitive conversion to rel path on windows (fixes #5183) (#5176) 2018-09-11 22:30:32 +02:00
Jakob Borg
60a6a40175 dockerfile: Improve UID/GID handling (fixes #5180) (#5181)
This removes the user and group juggling, which would fail when given
for example a PGID that already existed as the "syncthing" group could
then not be created with that PGID. It's not reasonable to expect the
user to know which group/user names/IDs are already present in the
Docker image.

Instead we now just launch under the specified IDs, while manually
setting the HOME env var to give us a home directory - the only thing we
needed the user entry for anyway.

Also updates to Go 1.11 and building without upgrades instead of
disabling by env var.
2018-09-11 19:46:20 +01:00
Jakob Borg
323195be0e cmd/uraggregate, cmd/ursrv: Add missing copyright headers 2018-09-09 15:52:59 +02:00
Jakob Borg
ae76f1e999 cmd/ursrv: Handle send only / recv only accounting correctly 2018-09-09 15:39:54 +02:00
Simon Frei
0a29fa65ab cmd/ursrv: Display more relevant data (#17) 2018-09-09 14:28:48 +02:00
Jakob Borg
37cd5a0bec Merge github.com/syncthing/usage-reporting into main repo 2018-09-09 14:26:56 +02:00
Jakob Borg
db0c85318b cmd/syncthing: Account receive only folders in usage report 2018-09-09 13:55:19 +02:00
Jakob Borg
9e00b619ab all, vendor: Switch back to non-forked thejerf/suture (#5171) 2018-09-08 12:56:56 +03:00
Jakob Borg
8aa2d8d92c gui, man, authors: Update docs, translations, and contributors 2018-09-05 07:45:24 +02:00
Jakob Borg
22d8a379c7 Merge branch 'release'
* release:
  lib/model: Fixes on receive-only test setup and pulling (#5136)
  lib/fs: Don't add path separators at end of path (fixes #5144) (#5146)
  lib/fs: Evaluate root when watching not on fs creation (fixes #5043) (#5105)
2018-09-02 21:51:49 +02:00
Simon Frei
09aff7bb14 lib/model: Fixes on receive-only test setup and pulling (#5136) 2018-09-02 21:36:07 +02:00
Simon Frei
b068c8f346 lib/fs: Don't add path separators at end of path (fixes #5144) (#5146) 2018-09-02 21:31:18 +02:00
Simon Frei
c2ff49ed43 lib/fs: Evaluate root when watching not on fs creation (fixes #5043) (#5105) 2018-09-02 21:31:04 +02:00
Jakob Borg
4cb6bb6f64 Merge branch 'release'
* release:
  lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
2018-09-02 21:05:53 +02:00
Jakob Borg
b80da29b23 lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.

This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).

The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
2018-09-02 21:02:28 +02:00
Jakob Borg
836ca50570 lib/db: Fix inconsistency in sequence index (fixes #5149) (#5158)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.

This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).

The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
2018-09-02 20:58:32 +02:00
Jakob Borg
e384c822b6 cmd/stdiscosrv: Be more picky about allowed addresses (fixes #5151) (#5153)
Filter out ludicrous stuff both from explicitly announced addresses and
potential automatic replacements.
2018-08-30 18:06:35 +01:00
Jakob Borg
916182bc73 gui, man, authors: Update docs, translations, and contributors 2018-08-29 07:45:24 +02:00
Simon Frei
c62ce007ea lib/fs: Don't add path separators at end of path (fixes #5144) (#5146) 2018-08-28 08:18:55 +02:00
Audrius Butkevicius
aec66045ef lib/config: Rewrite pending notifications (fixes #2291) 2018-08-25 11:36:10 +01:00
Simon Frei
03c0537340 lib/model: Fix regressions detecting deletes/ignores (fixes #5125, fixes #5127) (#5129) 2018-08-25 10:32:35 +02:00
Jakob Borg
7dde6c7e3c lib/model: Clear out-of-space-errored files from queue (fixes #5143) 2018-08-25 10:26:10 +02:00
Simon Frei
cb0f4ce55a lib/model: Don't stop folder if out of disk space (fixes #2370) (#5099)
This removes the out of disk space check from CheckHealth. The disk space is now
only checked if there are files to pull, in which case pulling those files is
stopped, but everything else (dirs, links, deletes) keeps running -> can recover
disk space through pulling.
2018-08-25 10:16:38 +02:00
Ben S
d86d5e8bb2 gui: Add icons to settings tabs (#5140)
Add icons to tabs in settings modal like in folder and device modal
2018-08-23 12:45:28 +01:00
Jakob Borg
d2c68fd163 gui, man, authors: Update docs, translations, and contributors 2018-08-22 07:45:24 +02:00
Jakob Borg
a02db70a63 lib/model: Process download progress messages for all folder types (fixes #5131) (#5139) 2018-08-21 18:49:35 +01:00
Simon Frei
165417c462 lib/model: Fixes on receive-only test setup and pulling (#5136) 2018-08-19 22:34:26 +01:00
Simon Frei
ff3cbdc90d lib/model: Check availability later to catch renames (#5097) 2018-08-19 19:03:20 +01:00
Simon Frei
9028969617 lib/model: Small fixes to test convenience functions (#5128) 2018-08-16 12:11:48 +02:00
Paweł Rozlach
f6da436f4b cmd/stdiscosrv: Add replication heartbeats (fixes #5117) (#5120) 2018-08-15 16:52:20 +02:00
Jakob Borg
166a33037f Merge branch 'release'
* release:
  lib/model: Always release the lock (#5126)
  lib/model: Catch racy nil deref in ClusterConfig (#5106)
2018-08-15 16:49:03 +02:00
Simon Frei
7c0798b622 lib/model: Always release the lock (#5126) 2018-08-15 16:44:59 +02:00
Simon Frei
ee42c46bd3 lib/model: Catch racy nil deref in ClusterConfig (#5106) 2018-08-15 16:44:20 +02:00
Simon Frei
885f6fcf28 lib/model: Always release the lock (#5126) 2018-08-15 16:33:03 +02:00
Andrew Rabert
ada5ab74d2 Dockerfile: Reduce number of container layers for final image (#5124) 2018-08-15 08:33:35 +01:00
Jakob Borg
0bc3ae15ae gui, man, authors: Update docs, translations, and contributors 2018-08-15 07:45:28 +02:00
Simon Frei
dc57bab107 lib/model: Don't enter pulling state if we need nothing (fixes #4782) (#5118) 2018-08-13 20:39:25 +02:00
Jakob Borg
48795dba07 all: Don't let Suture capture panics (fixes #4758) (#5119)
Fork with new option.
2018-08-13 20:39:08 +02:00
Simon Frei
c55c0c8c28 lib/watchaggregator: Don't delay mixed events only (#5094)
Also fix a minor bug in testing failure output.
2018-08-13 09:14:03 +02:00
Jakob Borg
2c9d96375b build: Fix LICENSE distribution for stdiscosrv/strelaysrv 2018-08-11 22:45:46 +02:00
Simon Frei
e20679afe1 lib/fs: Evaluate root when watching not on fs creation (fixes #5043) (#5105) 2018-08-11 22:24:36 +02:00
Jakob Borg
b37c05c6b8 lib/model: Don't run watcher on recvonly tests (fixes #5110) (#5112) 2018-08-11 22:19:37 +02:00
Simon Frei
dfe4008607 lib/model: Catch racy nil deref in ClusterConfig (#5106) 2018-08-11 09:10:29 +02:00
Jakob Borg
54d610878d gui, man, authors: Update docs, translations, and contributors 2018-08-08 07:45:31 +02:00
Adam Piggott
229b777f30 gui: Fix incorrect label (#5101) 2018-08-06 23:08:17 +01:00
Jakob Borg
7812c2c937 vendor: Point github.com/syncthing/notify at master again (metadata only) 2018-08-06 21:44:53 +02:00
Jakob Borg
500e02cdbb vendor: Patch github.com/syncthing/notify for Go 1.11 2018-08-06 20:15:43 +02:00
Simon Frei
82c9e23206 vendor: Remove unused vendor packages (fixes #3595) (#5096) 2018-08-04 16:29:13 +01:00
Simon Frei
705b7d18e8 build: Also copy gui to temporary GOPATH (#5095) 2018-08-02 16:13:17 +02:00
Simon Frei
f4bde023aa build: Build and set GOPATH before generating assets (#5093) 2018-08-01 21:22:47 +02:00
Jakob Borg
af48b069cc gui, man, authors: Update docs, translations, and contributors 2018-08-01 07:45:28 +02:00
Jakob Borg
5cb4a9acf6 lib/db: Don't account remote invalid files (fixes #5089) (#5090) 2018-07-31 13:00:03 +02:00
Oyebanji Jacob Mayowa
adc5bf6604 lib/upnp: Don’t log unknown device types (fixes #5038) (#5087) 2018-07-30 16:34:35 +02:00
Audrius Butkevicius
24d307531d gui: Use one instead of on to have callbacks fire one (#5085) 2018-07-29 21:15:24 +02:00
Audrius Butkevicius
93fdd1c012 cmd/strelaypoolsrv: Prevent scraped metrics moving backwards (#5068) 2018-07-27 07:59:55 +02:00
Audrius Butkevicius
5161f03f02 lib/config: Fix aliased append, copy config inputs and outputs (fixes #5063) (#5069) 2018-07-26 23:14:12 +02:00
Adam Piggott
682ffcb8ed github: Mention auxiliary projects and db corruption in issue template (#5081)
Add a section on using the correct issue tracker
Add a section on database corruption
2018-07-26 20:05:56 +02:00
Jakob Borg
524ffe3fa5 gui, man, authors: Update docs, translations, and contributors 2018-07-25 07:45:29 +02:00
Jakob Borg
d8366e4a88 authors: Enable auto updates (#5074)
Removes the manual handling of the AUTHORS file, giving every committer automatic credit for their contribution.

Manual tweaks to the file are still accepted and retained by the scripts.
2018-07-23 17:41:59 +02:00
Jakob Borg
b83c5b32bf authors: Add bebehei 2018-07-20 15:46:38 +02:00
Benedikt Heine
3102e36a45 dockerfile: Create a dedicated syncthing user (#5072)
A dedicated user is necessary to create relative references via
~/<folder> or $HOME/<folder>. Having the syncthing process just running
under a unprivileged UID/GID, will remove the home folder relation and
therefore will result in nonexistent shares after update.

Signed-off-by: Benedikt Heine <bebe@bebehei.de>
2018-07-20 15:45:40 +02:00
Jakob Borg
3d8344003e lib/logger: Add missing dots (fixes #5073) 2018-07-19 20:49:57 +02:00
Jakob Borg
a4415bce10 gui, man: Update docs & translations 2018-07-18 07:45:27 +02:00
Audrius Butkevicius
9f87fd1fcf lib/model: More auto accept tests (#5014) 2018-07-15 20:26:20 +03:00
Simon Frei
5592b8b190 lib/model: Record error for unavailable files (#5066) 2018-07-14 14:09:23 +01:00
Jakob Borg
f822b10550 all: Add receive only folder type (#5027)
Adds a receive only folder type that does not send changes, and where the user can optionally revert local changes. Also changes some of the icons to make the three folder types distinguishable.
2018-07-12 11:15:57 +03:00
Jakob Borg
1a6c7587c2 gui, man: Update docs & translations 2018-07-11 07:45:26 +02:00
Simon Frei
6b82538e62 lib/model: Also handle missing parent dir non-regular items (#5048)
This is an improvement of PR #4493 and related to (and maybe fixing) #4961
and #4475. Maybe fixing, because there is no clear reproducer for that
problem.

The previous PR added a mechanism to resurrect missing parent directories,
if there is a valid child file to be pulled. The same mechanism does not
exist for dirs and symlinks, even though a missing parent can happen for
those items as well. Therefore this PR extends the resurrection to all types
of pulled items.

In addition I moved the IsDeleted branch while iterating over
processDirectly to the existing IsDeleted branch in the WithNeed iteration.
This saves one pointless assignment and IsDeleted query. Also
2018-07-10 18:40:06 +03:00
Simon Frei
3f17bda786 lib/db: Catch unignored/conflicting files as needed (fixes #5053) (#5054) 2018-07-10 18:32:34 +03:00
rubenbe
e10d7260c2 dockerfile: Install su-exec without updating. (#5051)
* Using --no-cache instead prevents unnecessarily
  adding about 1.3MB of Alpine package data
* This reduces the uncompressed image size with about 6%.
2018-07-09 21:09:04 +03:00
Simon Frei
409cb2beb8 lib/fs: Catch size-preserving changes on windows (fixes #5050) (#5056) 2018-07-09 18:29:22 +01:00
Simon Frei
742ab17a2a authors: Patch xor-gate (#5057) 2018-07-09 16:22:51 +03:00
Jerry Jacobs
9f254df091 cmd/stcli: Add config command with pretty printed JSON (#5049) 2018-07-06 00:37:13 +02:00
Jakob Borg
290dcf0610 authors: Add xor-gate 2018-07-05 23:58:00 +02:00
Simon Frei
0f0290d574 lib/model, lib/weakhash: Abort pulling quicker on folder stop (ref #5028) 2018-07-04 08:07:33 +01:00
Andrew Rabert
5bb72dfe5d docker: Add configurable UID and GID (#5041)
Allows for configuring the UID and GID Syncthing runs as in the container. Uses su-exec from the Alpine repos to accomplish this. Addition of su-exec results in <2MB increase in image size.
2018-07-04 08:42:29 +02:00
Jakob Borg
0b73a66516 authors: Add nvllsvm 2018-07-04 08:41:09 +02:00
Simon Frei
0631fd2440 vendor: Add missing file to github.com/syncthing/notify (#5042) 2018-07-04 08:37:58 +02:00
Jakob Borg
1e2e8650e1 Merge branch 'release'
* release:
  vendor: Update github.com/thejerf/suture
  lib/model: Release both locks when waiting for services to stop (fixes #5028)
2018-06-30 10:40:36 +02:00
Jakob Borg
21035b0c1a vendor: Update github.com/thejerf/suture 2018-06-29 08:58:53 +02:00
Jakob Borg
c884955ff7 vendor: Update github.com/thejerf/suture 2018-06-29 08:54:57 +02:00
Simon Frei
b91ff430db lib/model: Release both locks when waiting for services to stop (fixes #5028) 2018-06-28 13:29:41 +02:00
Alexandre Viau
b09c50bf9c gui: Ship license files (#5037) 2018-06-27 19:09:57 +02:00
Alexandre Viau
7f0603effa gui: Add license verbiage to vendored angular.js (#5035) 2018-06-27 08:29:44 +02:00
Jakob Borg
ff441d3b3e lib/connections: Don't spin on accept failures (fixes #5025) (#5036) 2018-06-27 08:24:30 +02:00
Jakob Borg
950f4a8672 authors: Patch anonymouse64 2018-06-27 08:10:06 +02:00
Ian Johnson
95eb81467c build: Improve snap generation (fixes #4863, fixes #5000) (#5034)
This makes the environment variables easier to read/change.

Also expand the list so it's not inline, more readable that way.

The new architecture syntax for snapcraft allows specifying both the building architecture and the running architecture of the snap, so if we specify the build-on architecture as the host architecture and the run-on architecture as the target architecture, then snapcraft shouldn't need to install any cross-compilers, etc.
2018-06-27 08:08:57 +02:00
Jakob Borg
55ed883648 authors: Add anonymouse64 2018-06-27 08:07:09 +02:00
Jakob Borg
0c2cebb775 gui, man: Update docs & translations 2018-06-27 07:45:25 +02:00
Ben S
645588902b gui: Use info icon for folder ID (#5031) 2018-06-26 21:52:34 +01:00
Simon Frei
881e923105 cmd/syncthing, lib/db: Abort execution if db version is too high (fixes #4994) (#5022) 2018-06-26 11:40:34 +02:00
Jakob Borg
ef5ca0c218 build: Let "go generate" create assets 2018-06-26 10:29:36 +02:00
Simon Frei
406b394704 vendor: Update github.com/syncthing/notify (fixes #4854) (#5032) 2018-06-26 10:13:39 +02:00
Simon Frei
7b0d8c2e77 lib/model: Release both locks when waiting for services to stop (fixes #5028) 2018-06-24 16:55:28 +01:00
Jakob Borg
b1b68ceedb Add LocalFlags to FileInfo (#4952)
We have the invalid bit to indicate that a file isn't good. That's enough for remote devices. For ourselves, it would be good to know sometimes why the file isn't good - because it's an unsupported type, because it matches an ignore pattern, or because we detected the data is bad and we need to rescan it.

Or, and this is the main future reason for the PR, because it's a change detected on a receive only device. We will want something like the invalid flag for those changes, but marking them as invalid today means the scanner will rehash them. Hence something more fine grained is required.

This introduces a LocalFlags fields to the FileInfo where we can stash things that we care about locally. For example,

    FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
    FlagLocalIgnored     = 1 << 1 // Matches local ignore patterns
    FlagLocalMustRescan  = 1 << 2 // Doesn't match content on disk, must be rechecked fully

The LocalFlags fields isn't sent over the wire; instead the Invalid attribute is calculated based on the flags at index sending time. It's on the FileInfo anyway because that's what we serialize to database etc.

The actual Invalid flag should after this just be considered when building the global state and figuring out availability for remote devices. It is not used for local file index entries.
2018-06-24 09:50:18 +02:00
Jakob Borg
6df3940c26 conduct: Upgrade to Contributor Covenant
It's pretty much the GitHub standard, says the obvious, and does it
better than our old one which was a little bit rambling.

Intentionally just doing it instead of discussing it, in the hope of
avoiding trolling. So sue me.
2018-06-20 23:53:06 +02:00
Simon Frei
238476bfcd gui: kiB -> KiB (fixes #5017) (#5021) 2018-06-20 16:51:30 +02:00
Jakob Borg
d69a9d52a9 gui, man: Update docs & translations 2018-06-20 07:45:26 +02:00
Audrius Butkevicius
b3007c03bd gui: Stop log tailing on scroll (#5013) 2018-06-18 12:27:54 +01:00
Simon Frei
c2784d76e4 lib/db: Remove updated invalid files from need bucket (fixes #5007) (#5008) 2018-06-18 08:23:40 +02:00
Simon Frei
8ff7ceeddc lib/ignore, lib/scanner: Fix recursion to catch included paths (fixes #5009) (#5010) 2018-06-18 08:22:19 +02:00
qepasa
84c8964865 gui: Improve validation of rate limits (fixes #5012) (#5015) 2018-06-18 08:18:52 +02:00
xjtdy888
7d3f94911f cmd/syncthing: Correctly compare If-Modified-Since in HTTP server (#5016) 2018-06-18 08:14:17 +02:00
Simon Frei
54e17c8bf4 lib/model: Don't set watch error on folder creation (fixes 5005) (#5006) 2018-06-15 23:33:23 +01:00
Jakob Borg
793b86e604 script: Use strings instead of byte slice literals in asset generation
It compiles literally 50 times faster and generates the same code. Also
add the little header that indicates auto generated code.
2018-06-13 23:48:50 +02:00
Jakob Borg
7b47692ec0 build: MacOS X is now macOS; don't presume to know all possible goarchs 2018-06-13 23:20:52 +02:00
Jakob Borg
35a75a95dc lib/model: Don't panic when rechecking file (fixes #5002) (#5003) 2018-06-13 18:07:52 +01:00
Jakob Borg
c7c7fbe9d9 gui, man: Update docs & translations 2018-06-13 07:45:27 +02:00
Simon Frei
9e0e04f4de all: Fix FS watcher restarting and web UI indication (fixes #4923) (#4962) 2018-06-11 15:47:54 +02:00
Simon Frei
1e2732aa21 lib/config, lib/model: Don't warn and return error (#4997) 2018-06-10 15:41:20 +02:00
Simon Frei
b7234785f8 lib/model: Wait for folder to stop (fixes #4981) (#4982) 2018-06-10 13:24:59 +02:00
Simon Frei
30056cd1ae lib/db: Move database schema migration into its own file (#4985) 2018-06-08 12:46:00 +02:00
Jakob Borg
4781e1d86f test: Migrate test configs to current version 2018-06-08 12:39:17 +02:00
Jakob Borg
a467f6908c lib/protocol: Test for IsEquivalent (#4996) 2018-06-08 12:02:16 +02:00
qepasa
9fc20b41c6 gui: Add tabs in device editor (#4986) 2018-06-07 21:01:19 +01:00
Simon Frei
e2c44f519c lib/config, lib/model: Handle shared with information in config (fixes #4870) (#4974) 2018-06-06 23:34:11 +02:00
Simon Frei
53a029a796 gui: Restrict shown decimals and restrict size of header columns (#4973) 2018-06-06 23:33:31 +02:00
Jakob Borg
02da7414ab cmd/stfindignored: Default to current directory 2018-06-06 22:24:36 +02:00
Jakob Borg
d1f953b0dd cmd/stfindignored: Make that 2018 2018-06-06 22:10:17 +02:00
Jakob Borg
47cefb2ab2 cmd/stfindignored: Add utility 2018-06-06 22:08:41 +02:00
Ben S
4de8920642 gui: Hide allowed networks if unused (#4989) 2018-06-06 19:56:54 +01:00
Jakob Borg
76f9e5c5db lib/protocol: Correct block size calculation on 32 bit archs (fixes #4990) (#4991) 2018-06-06 09:59:33 +02:00
Jakob Borg
27d675a793 lib/upgrade: Tests should pass on darwin-386 2018-06-06 09:47:13 +02:00
Jakob Borg
dd9e1d61eb gui, man: Update docs & translations 2018-06-06 07:45:29 +02:00
Simon Frei
b670b5550a gui: Don't remove the slash from path '/' (fixes #4983) (#4988) 2018-06-05 23:47:47 +02:00
Simon Frei
ee6516aa31 lib/fs: Resolve 8.3 filenames from watcher (ref #3800) (#4975) 2018-06-04 13:41:03 +02:00
Jakob Borg
8b15624f7d lib/scanner: Skip block size hysteresis test in -short mode
This burns like a percent of my laptop battery on every invocation...
2018-06-02 13:10:05 +00:00
Simon Frei
5baa432906 lib/db: Add index to track locally needed files (#4958)
To optimize WithNeed, which is called for the local device whenever an index
update is received. No tracking for remote devices to conserve db space, as
WithNeed is only queried for completion.
2018-06-02 15:08:32 +02:00
Ben S
d3a02a1663 gui: Disable rescan button while scanning (fixes #4977) (#4979) 2018-06-01 15:40:44 +02:00
Jakob Borg
e187a5f5cb test: Add another variant of API timeout to skip when benchmarking 2018-05-30 13:23:52 +02:00
Jakob Borg
e73631c5f0 gui, man: Update docs & translations 2018-05-30 07:45:27 +02:00
Jakob Borg
b87e5ed13d build: Use commit date as assets change date 2018-05-29 08:49:25 +02:00
Jakob Borg
c51365c634 script: Use source data from environment when generating assets 2018-05-29 08:39:46 +02:00
Audrius Butkevicius
d60f0e734c lib/scanner: Copy execute bits from previous version on Windows (fixes #4969) (#4970) 2018-05-29 07:01:23 +01:00
Simon Frei
a83176c77a lib/watchaggregator: Speedup propagation of removals (fixes #4953) (#4955) 2018-05-26 10:08:23 +01:00
Ben S
eb31be0432 gui: Update to Font Awesome v5 (#4889) 2018-05-24 19:59:32 +01:00
Simon Frei
07bf24a3b4 lib/watchaggregator: Prevent race on config update (#4938) 2018-05-24 19:47:15 +01:00
Simon Frei
ef1633ac76 lib/db: Update global count when removing the previous global version (#4968) 2018-05-24 18:17:45 +02:00
Simon Frei
9f305f674a gui: Check if folder exists in folderLabel (fixes #4965) (#4966) 2018-05-23 12:41:04 +02:00
Simon Frei
9d2b744c12 lib/model: Move String method to folder (#4964) 2018-05-23 08:23:21 +01:00
Jakob Borg
91f1f3067a gui, man: Update docs & translations 2018-05-23 07:45:25 +02:00
Jakob Borg
119335930c lib/model: Refactor override implementation into sendOnlyFolder
I'm trying to slowly clean this up a bit, and moving functionality out
into the folder types and having those methods not reach into model is
part of it. That can mean takign some odd arguments in the meantime,
some of those should probably become interfaces or properties on folder
in the long term.
2018-05-21 09:09:47 +02:00
Jakob Borg
370a3549e7 lib/model: Refactor folderScanner into folder
The functionality was anyway mostly implemented there and not isolated
in the folderScanner type. The attempt to refactor it out in the other
direction wouldn't work given that the event loop and stuff is on
`folder`.
2018-05-21 08:47:58 +02:00
Simon Frei
d64d954721 lib/db: Fix prefixed walks (fixes #4925) (#4940) 2018-05-17 09:26:40 +02:00
Simon Frei
81f9a81c7d gui: Change rescan interval only if fs watcher was actually toggled (fixes #4944) (#4946) 2018-05-17 09:21:10 +02:00
Jakob Borg
127c891526 cmd/stdiscosrv: Delete records for abandoned devices (#4957)
Once a device has been missing for a long time, and noone has asked
about it for a long time, delete the record.
2018-05-16 09:26:20 +02:00
Jakob Borg
c08024ebb8 lib/db: Add db and test with invalid files (#4954)
This adds a couple of utilities for transporting databases in JSON and a
test to load a database and verify a couple of invalid bits. The test
itself is quite pointless at the moment, but it lays the groundwork for
testing the migration of this data in the next step (after the invalid
bit should be changed to local flags for local files).

When that happens we need to have a database in the old format already
there in order to be able to test the migration.
2018-05-16 08:44:08 +02:00
Jakob Borg
677fde50b3 gui, man: Update docs & translations 2018-05-16 07:45:25 +02:00
Simon Frei
eabce48761 lib/model: Also wait for ItemFinished in TestDeregister (#4951) 2018-05-14 20:01:35 +01:00
Simon Frei
d59aecba31 lib/ignore, lib/scanner: Catch included items below ignored ones (#4811) 2018-05-14 09:47:23 +02:00
Jakob Borg
22b0bd8e13 Merge branch 'release'
* release:
  gui: Fix scopes in notification directive (fixes #4941) (#4948)
  gui: Fallback to folder ID in recent changes (fixes #4947) (#4949)
2018-05-14 08:55:24 +02:00
Jakob Borg
9260248543 gui: Fix scopes in notification directive (fixes #4941) (#4948)
The directive lives in its own isolated scope (where we put the visible() function). The stuff transcluded into the notification directive lives in the root scope and doesn't have access to the directive scope. Hence we cannot call dismiss() from inside the directive.

Similarly, config does not exist by itself in the directive scope, we need $parent to access the root scope.

Reference: https://docs.angularjs.org/guide/directive#isolating-the-scope-of-a-directive

How this worked before is a mystery. My guess is Angular bug with directive scope that was fixed in 1.3. One possibility is that transclude plus scope: true (which doesn't make sense as that is supposed to be an object) resulted in the root scope being used in the directive itself. This would then "work" as long as there is only one notification, as visible() and dismiss() would then be registered on the root scope, thus accessible from within the notification but also overridden by any notification rendered.
2018-05-14 08:53:06 +02:00
Simon Frei
c4a348db67 gui: Fallback to folder ID in recent changes (fixes #4947) (#4949) 2018-05-14 08:53:01 +02:00
Jakob Borg
20aa53486a all: Serialize folder types to new names (#4942)
It's been a year and a half since we started accepting the new names.
It's time we start producing them.
2018-05-13 09:58:00 +02:00
Jakob Borg
ed4807d9a4 gui: Fix scopes in notification directive (fixes #4941) (#4948)
The directive lives in its own isolated scope (where we put the visible() function). The stuff transcluded into the notification directive lives in the root scope and doesn't have access to the directive scope. Hence we cannot call dismiss() from inside the directive.

Similarly, config does not exist by itself in the directive scope, we need $parent to access the root scope.

Reference: https://docs.angularjs.org/guide/directive#isolating-the-scope-of-a-directive

How this worked before is a mystery. My guess is Angular bug with directive scope that was fixed in 1.3. One possibility is that transclude plus scope: true (which doesn't make sense as that is supposed to be an object) resulted in the root scope being used in the directive itself. This would then "work" as long as there is only one notification, as visible() and dismiss() would then be registered on the root scope, thus accessible from within the notification but also overridden by any notification rendered.
2018-05-13 07:44:19 +02:00
Simon Frei
7d07b19734 gui: Fallback to folder ID in recent changes (fixes #4947) (#4949) 2018-05-13 07:43:51 +02:00
Simon Frei
a7e30c925f all: Use Executable from os instead of osext (fixes #4900) (#4950) 2018-05-13 07:43:24 +02:00
Jakob Borg
0c81fa4191 cmd/syncthing: Return formatted JSON in API (#4945)
Because machines don't care, and sometimes humans read the output.
2018-05-12 15:14:41 +02:00
Jakob Borg
9e696a154b lib/model: Rename {ro,rw}folder.go
To newer names better reflecting their types and yet sorting together
with folder.go. Doing it now without asking because there are no open
PRs that will get killed by it, and to avoid bikeshedding the names.
2018-05-11 11:00:02 +02:00
Jakob Borg
344e52e311 lib/model: Refactor {ro,rw}folder serve loop into just folder (#4939)
The actual pull method (which is really the only thing that differs
between them) is now an interface member which gets overridden by the
subclass.

"Subclass?!" Well, this is dynamic dispatch with overriding, I guess.
2018-05-11 10:45:13 +02:00
Simon Frei
a2f51c85c2 lib/osutil: Add test for IsDeleted (ref #4925) (#4936) 2018-05-10 21:39:33 +02:00
Simon Frei
00f4900ba7 build: Increase test timeout to 2min (#4937) 2018-05-10 21:38:49 +02:00
Audrius Butkevicius
e125f8b05b gui: Enable proper asset caching (#4931) 2018-05-10 07:53:39 +02:00
Jakob Borg
fb198a0645 lib/db: Actually delete the correct sequence prefix 2018-05-09 12:06:29 +02:00
xjtdy888
506181599c lib/db: Remove all sequences related to the folder (fixes #4928) (#4929) 2018-05-09 08:57:42 +02:00
Jakob Borg
15d0bdcba9 authors: Add xjtdy888 2018-05-09 08:42:27 +02:00
Jakob Borg
b6b0d0664d gui, man: Update docs & translations 2018-05-09 07:45:15 +02:00
Audrius Butkevicius
baa3aa4bad gui: Don't save ignores if they haven't been loaded (fixes #4915) (#4930) 2018-05-08 23:39:17 +02:00
Audrius Butkevicius
a48a31e3f5 lib/ignores: Fix ignore loading, report errors to UI (fixes #4901) (#4932) 2018-05-08 23:37:13 +02:00
Jakob Borg
2343c82c33 lib/model: Don't include unshared folders in ClusterConfig (fixes #4926)
Also fixes a data race where ClusterConfig would access folderFiles
without a lock. Tweaked the ClusterConfig unit test to verify the
behavior.
2018-05-08 08:19:34 +01:00
Jakob Borg
616883304e Revert "build: Only the first dash in the Debian version should become a tilde"
This reverts commit 1cb09b7d2d.

I misunderstood and acted too quickly.
2018-05-05 15:33:54 +02:00
Jakob Borg
1cb09b7d2d build: Only the first dash in the Debian version should become a tilde 2018-05-05 15:29:51 +02:00
Jakob Borg
3b5e1fa0fc gui: Update Angular to 1.3.20, enable $applyAsync (fixes #4918) 2018-05-05 11:13:16 +01:00
Simon Frei
a94aceb22f lib/model: Don't create folder root when paused (fixes #4903) (#4904) 2018-05-05 10:30:39 +02:00
Nils Jakobi
f5d8243f15 dockerfile: Incorporate exposed ports, add volume (#4908)
Added EXPOSE to Dockerfile. this way these ports will show up in docker GUIs like cockpit.
Added VOLUME parameter, this renders creating the folder (/var/syncthing) obsolete.
2018-05-05 10:28:19 +02:00
Jakob Borg
c9c2e69f49 authors: Add thunderstorm99 2018-05-05 10:27:26 +02:00
Audrius Butkevicius
ef0dcea6a4 lib/model: Verify request content against weak (and possibly strong) hash (#4767) 2018-05-05 10:24:44 +02:00
Jakob Borg
678c80ffe4 authors: Add harrisonhjones (fixes #4909) 2018-05-02 08:40:33 +02:00
Jakob Borg
0f1d0380dc build: Syso stuff needs to happen on build (ref #4909) 2018-05-02 08:37:53 +02:00
Jakob Borg
58bc722a1f gui, man: Update docs & translations 2018-05-02 07:45:15 +02:00
Simon Frei
53dc346583 lib/model: Fix test function for introducer (#4898) 2018-05-01 22:56:20 +01:00
Jakob Borg
c2f498fc82 lib/model: Units are hard (fixes #4910) 2018-05-01 22:50:23 +01:00
Simon Frei
a548014755 lib/db, lib/model: Add sequence->deviceKey to db for sending indexes (#4906)
Instead of walking and unmarshalling the entire db and sorting the resulting
file infos by sequence, add store device keys by sequence number in the
database. Thus only the required file infos need be unmarshalled and are already
sorted by index.
2018-05-01 23:39:15 +02:00
Wulf Weich
2c18640386 gui: Localize number formatting (fixes #4896) (#4902) 2018-04-25 10:26:49 +02:00
Jakob Borg
78094fa0cb gui, man: Update docs & translations 2018-04-25 07:45:13 +02:00
Simon Frei
f6458d1b8f lib/config, lib/model: Include paused folders in cluster config (fixes #4897) 2018-04-22 17:01:52 +01:00
Jakob Borg
56cf2db68b Merge branch 'release'
* release:
  vendor: Update github.com/syncthing/notify (fixes #4885) (#4894)
2018-04-21 13:43:56 +02:00
Simon Frei
a1b5a3d5c0 vendor: Update github.com/syncthing/notify (fixes #4885) (#4894) 2018-04-21 13:42:08 +02:00
Iain Barnett
4d3b5348ae lib/protocol: Add note about non-standard Luhn calculation (#4895)
Skip-check: authors
2018-04-20 18:52:03 +02:00
Simon Frei
3d02fcd473 vendor: Update github.com/syncthing/notify (fixes #4885) (#4894) 2018-04-20 17:01:03 +02:00
Jakob Borg
25bf406f0e authors: Patch pramodhkp 2018-04-20 08:03:11 +02:00
pramodhkp
071910b255 cmd/syncthing-cli: Fix errors show command (#4849) 2018-04-18 21:08:10 +01:00
Jakob Borg
bed6fb8baf authors: Add pramodhkp 2018-04-18 21:37:40 +02:00
Jakob Borg
d903bf79c5 gui, man: Update docs & translations 2018-04-18 07:45:16 +02:00
Jakob Borg
49cfe57edc Merge branch 'release'
* release:
  lib/osutil: Fix TraversesSymlink with symlinked fs root on windows (fixes #4875) (#4886)
2018-04-18 07:33:01 +02:00
Simon Frei
1f70f7e37c lib/osutil: Fix TraversesSymlink with symlinked fs root on windows (fixes #4875) (#4886) 2018-04-18 07:28:35 +02:00
Simon Frei
eca076cf7d lib/osutil: Fix TraversesSymlink with symlinked fs root on windows (fixes #4875) (#4886) 2018-04-17 22:53:06 +02:00
Jakob Borg
dbcf7a02a0 lib/model: Increase the default pull limit (fixes #4883)
Bumping the limit to 2 * the max block size (16 MiB) is a slight
increase compared to previously. Nonetheless I think it's good to allow
us to queue one request and have one on the way in, or conversely have
one large block on the way in and be able to ask for smaller blocks from
others at the same time.
2018-04-17 07:55:49 +01:00
Jakob Borg
7be53bbb88 Merge branch 'release'
* release:
  lib/fs: Fix watcher panic due to casing on windows (fixes #4877)  (#4878)
2018-04-16 20:14:54 +02:00
Simon Frei
17e3608865 lib/fs: Fix watcher panic due to casing on windows (fixes #4877) (#4878) 2018-04-16 20:10:11 +02:00
Jakob Borg
19c7cd99f5 all: Implement variable sized blocks (fixes #4807) 2018-04-16 19:08:50 +01:00
Simon Frei
01aef75c96 lib/fs: Fix watcher panic due to casing on windows (fixes #4877) (#4878) 2018-04-16 20:07:00 +02:00
Jakob Borg
8f6d587ecb authors: Patch wwwutz 2018-04-14 14:39:41 +02:00
Peter Marquardt
3ef19411f9 cmd/syncthing: Remove spurious <nil> in debug output
Remove spurious gui.go:985: DEBUG: <nil> on successful call time.Parse() in getSystemLog and getSystemLogTxt.
2018-04-14 10:46:33 +02:00
Jakob Borg
ea43e089d4 authors: Add wwwutz 2018-04-14 10:43:42 +02:00
Jakob Borg
5fa9237a62 script: Don't base64 encode the assets (#4874)
We did this to minimize source size and make it compile faster. Nowadays
byte slice literals compile as fast as anything, and the source isn't
checked in so it doesn't matter how big it is.

Not using base64 avoids a decode step at startup and makes the binary
about 400k smaller.
2018-04-11 22:02:14 +01:00
Jakob Borg
fa367e92b0 gui, man: Update docs & translations 2018-04-11 07:45:15 +02:00
Simon Frei
4072ae4d05 lib/model: Prevent warning on request in paused folder (fixes #4870) 2018-04-09 20:55:52 +01:00
Jakob Borg
e9c6795ef8 docker: Add README from old Docker repo (fixes #4868) (#4869)
With slight modifications
2018-04-09 10:48:37 +02:00
Audrius Butkevicius
afb27f7f02 cmd/strelaypoolsrv: Move metric scraping to the server itself (#4866) 2018-04-08 20:13:55 +01:00
Jakob Borg
cf4d7ff50f gui, man: Update docs & translations 2018-04-04 07:45:15 +02:00
Audrius Butkevicius
29e7e54bb4 assets: Use icon from synctrayzor (ref #4839) (#4859) 2018-04-02 19:57:01 -04:00
Jakob Borg
6982c06261 cmd/strelaypoolsrv: Handle portless X-Forwarded-For (#4856) 2018-04-01 21:29:34 -04:00
Simon Frei
26d87ec3bb lib/fs: Don't panic when watching a folder with symlinked root (#4846) 2018-03-28 22:01:25 +01:00
John Rinehart
c51591b308 cmd/syncthing: Set Content-Type header regardless of asset location (#4847) 2018-03-28 15:51:24 -04:00
Jakob Borg
8b052a950d authors: Add fuzzybear3965 2018-03-28 15:48:04 -04:00
Audrius Butkevicius
1ff5fc9620 cmd/syncthing: Correct --paths in some cases (#4845) 2018-03-28 18:02:43 +01:00
Jakob Borg
4aaf8d4ceb gui, man: Update docs & translations 2018-03-28 07:45:15 +02:00
Audrius Butkevicius
720e8dedbc lib/osutil: Use unix lowprio implementation on Android (#4844) 2018-03-27 22:03:09 +01:00
Jakob Borg
8ac11f19bd Merge branch 'release'
* release:
  lib/scanner, lib/model: Actually assign version when un-ignoring (fixes #4841) (#4842)
2018-03-27 16:27:53 -04:00
Simon Frei
bea6ecaf35 lib/scanner, lib/model: Actually assign version when un-ignoring (fixes #4841) (#4842)
This fixes a mistake introduced in #4750 and #4776 and is relevant to v0.14.46-rc1
2018-03-27 16:25:36 -04:00
Simon Frei
69f2c26d50 lib/scanner, lib/model: Actually assign version when un-ignoring (fixes #4841) (#4842)
This fixes a mistake introduced in #4750 and #4776 and is relevant to v0.14.46-rc1
2018-03-27 16:24:20 -04:00
Jakob Borg
59802c3981 cmd/stdiscosrv, vendor: Remove remnants of golang.org/x/net/context (#4843) 2018-03-27 07:37:34 -04:00
Jakob Borg
bdbaa84989 lib/connections: Wrong context snuck in somehow 2018-03-27 07:18:26 -04:00
Harrison Jones
8208bfa2b9 build: Add icon & file info to syncthing.exe (#4839) 2018-03-26 19:44:44 +01:00
Jakob Borg
c49d864f14 lib/connections: Slightly refactor limiter juggling
Two small behavior changes: don't "charge" the data to the global rate
limit until it's been accepted by the device specific limiter, and fix
the send/recv direction in the log print on per device rate limits.
2018-03-26 11:45:02 -04:00
qepasa
2621c6fd2f lib/connections, lib/config: Bandwidth throttling per remote device (fixes #4516) (#4603) 2018-03-26 12:01:59 +02:00
Audrius Butkevicius
3c920c61e9 gui: Add folder label to global changes, use bootstrap table (fixes #4828) (#4838) 2018-03-26 11:29:06 +02:00
Audrius Butkevicius
a5e40563de lib/model: Don't log ignored files (fixes #4832) (#4837) 2018-03-25 22:12:50 +02:00
Simon Frei
a557d62c4a all: Transition to using fs watcher by default (fixes #4552) 2018-03-25 21:05:47 +01:00
Jakob Borg
d6bb8e6e06 docker: Build using Go 1.10 2018-03-24 09:21:22 +01:00
Simon Frei
69d05a3637 Handle versions as returned from transformVersion (#16) 2018-03-24 09:08:58 +01:00
Simon Frei
da3b38ccce lib/fs: Fix and update error about inotify watch limit (fixes #4833) (#4835) 2018-03-23 12:56:38 +01:00
Graham Miln
e7dc2f9190 lib/nat: Fix clearAddresses/notify deadlock (ref #4601) (#4829)
clearAddresses write locks the struct and then calls notify. notify in turn tries to obtain a read lock on the same mutex. The result was a deadlock. This change unlocks the struct before calling notify.
2018-03-21 08:02:32 +01:00
Jakob Borg
ac2e9b0e09 authors: Update grahammiln 2018-03-21 08:00:41 +01:00
Jakob Borg
6cb4ac9ec2 gui, man: Update docs & translations 2018-03-21 07:45:15 +01:00
Audrius Butkevicius
862eec34db cmd/syncthing: Rename weak to rolling (#4830) 2018-03-20 23:42:36 +01:00
Jakob Borg
7da829a86f authors: Add grahammiln 2018-03-20 19:59:55 +01:00
Simon Frei
81bd428b25 review & forgotten fixes from other PR 2018-03-18 01:10:13 +00:00
Simon Frei
ae74ac8329 gui: Tabify edit folder modal
Copied over from settings. Moves ignore patterns from its own modal into a tab
of the folder edit modal. Advanced settings are in a tab instead of a expandable
section and versioning gets it's own tab.
Also added modal hidden event handler to remove the anchor in the url.
2018-03-18 01:10:13 +00:00
Simon Frei
5520022766 lig/ignore, lib/logger: Fix race and potential dead-locks when logging (#4821) 2018-03-17 16:49:12 +01:00
Jakob Borg
7872bfa173 cmd/syncthing: Improve local host check (fixes #4815) (#4816)
This does a less restrictive (and more correct) check on the IP address,
and also allows the fully qualified "localhost.".
2018-03-15 11:29:52 +01:00
Jakob Borg
bea3c01772 vendor: github.com/Zillode/notify is now github.com/syncthing/notify (#4813)
Given that we've taken on the resposibility of maintaining this forked
package I've added it to the Syncthing organization. We still vendor it
like an external package, because it's convenient to keep it as a fork
of upstream to easier merge and file pull requests towards them.
2018-03-14 14:48:22 +01:00
Simon Frei
55a7830ff9 lib/fs, lib/model: Make tests caching compatible (fixes #4749) (#4804) 2018-03-13 14:03:10 +01:00
Jakob Borg
470ef87dd5 vendor: Don't panic in FS watcher on old FreeBSD (fixes #4806) (#4809)
This adds a recover step to the notify package to avoid the panic. We
should get something like this upstreamed.
2018-03-13 13:31:26 +01:00
Simon Frei
b31bad1c4d lib/model: Remove unused shouldIgnore function (#4805) 2018-03-12 20:16:40 +01:00
Simon Frei
8b4346c3ec lib/scanner, lib/fs: Don't create file infos with abs paths (fixes #4799) (#4800) 2018-03-12 13:18:59 +01:00
Simon Frei
2bdb37d412 cmd/syncthing: More information in help about -logfile option (#4796) 2018-03-12 13:17:12 +01:00
Jakob Borg
1471c15b29 cmd/syncthing, lib/db: Be nicer about dropping deltas on upgrade (#4798)
When dropping delta index IDs due to upgrade, only drop our local one.
Previously, when dropping all of them, we would trigger a full send in
both directions on first connect after upgrade. Then the other side
would upgrade, doing the same thing. Net effect is full index data gets
sent twice in both directions.

With this change we just drop our local ID, meaning we will send our
full index on first connect after upgrade. When the other side upgrades,
they will do the same. This is a bit less cruel.
2018-03-10 11:42:01 +01:00
Jakob Borg
4b1782cf6d gui, man: Update docs & translations 2018-03-07 07:45:16 +01:00
Jakob Borg
71fab4d250 cmd/stdiscosrv: Record time of failed lookup
So that we can eventually garbage collect keys that noone is asking
about any more.
2018-03-06 16:15:29 +01:00
Jakob Borg
22ebc80329 cmd/stdiscosrv: Expose process metrics
Like this:

    $ curl -s http://localhost:9098/metrics | egrep '^syncthing_discovery_process'
    syncthing_discovery_process_cpu_seconds_total 12.92
    syncthing_discovery_process_max_fds 10240
    syncthing_discovery_process_open_fds 51
    syncthing_discovery_process_resident_memory_bytes 1.3674496e+08
    syncthing_discovery_process_start_time_seconds 1.52034731837e+09
    syncthing_discovery_process_virtual_memory_bytes 1.40324864e+08
2018-03-06 15:43:36 +01:00
Jakob Borg
74b820f287 test: Update conflict tests 2018-03-04 14:52:09 +01:00
Simon Frei
3949b750f5 gui: In remote need use index and auto-expand if only one folder (fixes #4759) (#4781) 2018-03-01 16:21:35 +01:00
Jakob Borg
a26778fa9b gui, man: Update docs & translations 2018-02-28 07:45:16 +01:00
Jakob Borg
d4b7be009c cmd/syncthing: Reset delta indexes on upgrade 2018-02-26 22:22:19 +00:00
Jakob Borg
c126831108 Add version penetration level stats 2018-02-25 18:04:28 +01:00
Audrius Butkevicius
2751be57dc lib/connections: Fix relay connections when two devices use the same relay (fixes #4778) (#4779) 2018-02-25 16:12:46 +01:00
Audrius Butkevicius
8df90bb475 lib/scanner: Track modified by in symlinks (#4777) 2018-02-25 14:51:37 +01:00
Simon Frei
36251b86f7 lib/model: Mark deleted file as conflicting when un-ignoring (#4776)
This completes #4750 as a followup to #4765.
2018-02-25 13:03:55 +01:00
Jakob Borg
42cc64e2ed lib/config, lib/model: Auto adjust pullers based on desired amount of pending data (#4748)
This makes the number of pullers vary with the desired amount of outstanding requests instead of being a fixed number.
2018-02-25 10:14:02 +01:00
Simon Frei
158859a1e2 lib: Handle metadata changes for send-only folders (fixes #4616, fixes #4627) (#4750)
Unignored files are marked as conflicting while scanning, which is then resolved
in the subsequent pull. Automatically reconciles needed items on send-only
folders, if they do not actually differ except for internal metadata.
2018-02-25 09:39:00 +01:00
Simon Frei
5822222c74 lib/model: More precise deletion detection (fixes #2571, fixes #4573) (#4765) 2018-02-25 09:27:54 +01:00
Jakob Borg
e99be02055 lib/model: Don't panic if block size in index is larger than protocol block size
This doesn't happen today, but it might in the future if the block size
were increased or made variable and we were talking to a client from the
future.
2018-02-24 07:34:23 -05:00
Matic Potočnik
1901a5a9f4 all: Fix typos (#4772)
Skip-check: authors
2018-02-24 08:51:29 +01:00
Jakob Borg
a27032f09e cmd/strelaysrv: Don't patch the default HTTP client (fixes #4745) 2018-02-21 09:56:04 -05:00
Jakob Borg
5e041dca9f cmd/strelaypoolsrv: Return better error codes and messages (#4770)
The current 500 "test failed" looks and sounds like a problem in the
relay pool server, while it actually indicates a problem on the
announcing side. Instead use 400 "connection test failed" to indicate
that the request was bad and what was the test.
2018-02-21 12:53:49 +01:00
Jakob Borg
c9ec6159e8 lib/fs, vendor: s/zillode/Zillode/ 2018-02-21 08:27:33 +01:00
Jakob Borg
5b17aae1b2 cmd/syncthing: Fix help text for STRECHECKDBEVERY (fixes #4764) 2018-02-21 08:26:57 +01:00
Jakob Borg
5931231a32 gui, man: Update docs & translations 2018-02-21 07:45:16 +01:00
Jakob Borg
d7dc37b3c4 There are more numbers in the Go version now 2018-02-20 08:03:34 +01:00
Simon Frei
6802505dda lib/fs: Fix MkdirAll failure due to \\?\ (fixes #4762) (#4763) 2018-02-16 15:19:20 +01:00
Audrius Butkevicius
d62820acf9 Few tweaks 2017-11-22 20:52:07 +00:00
Audrius Butkevicius
725baf0971 Add heatmap and per country break down (#13) 2017-11-11 15:51:59 +00:00
Audrius Butkevicius
8d95c82d7c Add new usage reports (#12) 2017-11-09 22:22:47 +00:00
Audrius Butkevicius
ad0a205116 Add block stats aggregates (#11) 2017-11-08 14:41:42 +00:00
Jakob Borg
c3c6ff0093 Various haxery 2017-10-26 15:10:17 +02:00
Jakob Borg
34903c4201 Option to listen on just HTTP (behind reverse proxy) 2017-10-25 09:26:13 +02:00
Audrius Butkevicius
335b08e3a8 Support fs watcher stuff, add missing fields (#9) 2017-10-21 21:15:00 +02:00
Simon Frei
c9bb3ba2e4 Remove obsolete UseAPIKey value (#8) 2017-10-20 17:24:25 +01:00
Audrius Butkevicius
b312f81e21 Add more columns, convert some fields to JSONB (#7) 2017-10-15 09:49:30 +02:00
Audrius Butkevicius
f8eea13406 Add support for v3 collection (#6) 2017-10-14 19:58:13 +02:00
Jakob Borg
9c099f1337 Don't mangle release candidate version numbers 2017-03-04 15:24:24 +01:00
Jakob Borg
f1e3b979a7 Timeline tweaks 2016-12-07 15:22:25 +01:00
Jakob Borg
f8a84d4436 Default time zone for minTs 2016-09-07 11:19:38 +02:00
Jakob Borg
2b5a735091 Report historical performance 2016-09-06 20:15:18 +02:00
Jakob Borg
dd175c5431 In the versions graph, filter out versions with <50 devices 2016-07-08 10:00:07 +02:00
Jakob Borg
6e960f1972 Less rounding in feature list 2016-06-09 11:55:05 +02:00
Jakob Borg
f5b7e8addd Also group compiler stats 2016-06-07 08:12:32 +02:00
Jakob Borg
82621f250c More details version/platform stats 2016-05-30 09:52:38 +02:00
Jakob Borg
ed45c43033 Also aggregate v0.11 2016-05-19 11:51:58 +09:00
Jakob Borg
13f9702a41 Fix compiler recognition regexp for new format 2016-04-09 14:56:59 +02:00
Jakob Borg
d884768344 Accept versioning data in reports 2016-02-15 12:50:00 +01:00
Jakob Borg
1ded36fcff Feature list sorting 2015-09-30 08:29:41 +02:00
Jakob Borg
c8c4b090ef Show v0.12 feature stats 2015-09-11 10:56:32 +02:00
Jakob Borg
e55ff6cccc Less duplication around database fields 2015-09-10 16:11:36 +02:00
Jakob Borg
30d0e5d298 Select must be specific about what fields it wants 2015-09-10 15:55:25 +02:00
Jakob Borg
201ff39b80 Receive v2 reports 2015-09-10 14:03:34 +02:00
Jakob Borg
ca9d3b597e Add bounced classification 2015-07-15 13:45:33 +02:00
Jakob Borg
0890b49e78 Add user movement graph 2015-07-15 13:23:13 +02:00
Jakob Borg
ca96b13233 Aggregate v0.10 2015-06-15 10:49:48 +02:00
Jakob Borg
525aac74aa Don't draw zero datapoints in graph 2015-06-15 10:46:48 +02:00
Jakob Borg
cf227509bf Micro layout tweak 2015-06-01 13:11:53 +02:00
Jakob Borg
cc4f32f259 Only graph the last year 2015-05-29 09:51:56 +02:00
Jakob Borg
6d9ed8ad92 Show version chart 2015-05-21 10:58:03 +02:00
Jakob Borg
3e1a562466 Reorganize, add aggregation 2015-05-21 08:52:19 +02:00
Jakob Borg
f3bf4cf722 Also show compilers and builders 2015-05-20 22:27:14 +02:00
Jakob Borg
e38fcbb566 Cache template result, not report 2015-03-04 13:02:53 +01:00
Jakob Borg
ed240145c7 Store reports in PostgreSQL 2015-03-04 11:31:46 +01:00
Jakob Borg
13f82a06ee Merge pull request #3 from uok/wording
Match wording with Syncthing GUI
2015-02-15 17:11:15 +01:00
Ben Schulz
349223b3b8 Match wording with Syncthing GUI 2015-02-15 12:00:15 +01:00
Jakob Borg
e5bea35515 Merge pull request #2 from uok/fix-favicon
Add favicon
2015-01-22 16:41:32 -08:00
Ben Schulz
dfbb245b60 Add favicon 2015-01-22 13:54:09 +01:00
Jakob Borg
cd5d546078 Summarize old versions 2014-12-09 16:56:47 +01:00
Jakob Borg
97a9ca24f3 Truncate list of versions 2014-12-09 16:52:02 +01:00
Jakob Borg
9c51cf50ad Better error reporting 2014-12-07 15:48:48 +01:00
Jakob Borg
5dab0e50aa Handle new format ID strings 2014-12-02 14:26:32 +01:00
Jakob Borg
48c40c87bc Set read and write timeouts on HTTPS requests 2014-10-13 11:18:13 +02:00
Jakob Borg
b37b19f344 Version handling for v0.9+ 2014-08-07 14:55:13 +02:00
Jakob Borg
0080ee6e86 Cleanup 2014-07-30 22:40:08 +02:00
Jakob Borg
caad69a312 Number formatting 2014-06-28 12:34:57 +02:00
Jakob Borg
518a4eb6ee Render report server side instead 2014-06-28 09:46:03 +02:00
Jakob Borg
113b110ab4 Summarize OS 2014-06-20 23:24:27 +02:00
Jakob Borg
ec85c9fdfe Forgot 5% for floats 2014-06-16 16:55:23 +02:00
Jakob Borg
d71a782179 Filter out zero values 2014-06-16 16:48:05 +02:00
Jakob Borg
d0aef07a38 Report five-percentile rather than min 2014-06-16 16:47:54 +02:00
Jakob Borg
8ad0a10c61 Thousands separator 2014-06-16 11:17:44 +02:00
Jakob Borg
85bb725eb6 Show data as it comes in (cached for one hour) 2014-06-16 11:14:01 +02:00
Jakob Borg
d8f9c096f3 Report on system memory size 2014-06-12 20:52:49 +02:00
Jakob Borg
e3e9f89861 Licensing 2014-06-12 02:45:19 +02:00
Jakob Borg
2faecfa403 Formatted report 2014-06-12 02:13:30 +02:00
Jakob Borg
1589942fc2 Initial 2014-06-12 01:40:54 +02:00
2715 changed files with 54596 additions and 1110321 deletions

5
.codecov.yml Normal file
View File

@@ -0,0 +1,5 @@
coverage:
range: "40...100"
ignore:
- "**.pb.go"

View File

@@ -11,6 +11,14 @@ requests directly to the developers. Worst case you might get a short
"that's a bug, please report it on GitHub" response on the forum, in which
case we thank you for your patience and following our advice. :)
### Please use the correct issue tracker
If your problem relates to a Syncthing wrapper or [sub-project](https://github.com/syncthing) such as [Syncthing for Android](https://github.com/syncthing/syncthing-android/issues), [SyncTrayzor](https://github.com/canton7/synctrayzor) or the [documentation](https://github.com/syncthing/docs/issues), please use their respective issue trackers.
### Does your log mention database corruption?
If your Syncthing log reports panics because of database corruption it is most likely a fault with your system's storage or memory. Affected log entries will contain lines starting with `panic: leveldb`. You will need to delete the index database to clear this, by running `syncthing -reset-database`.
### Please do post actual bug reports and feature requests.
If your issue is a bug report, replace this boilerplate with a description

View File

@@ -20,13 +20,8 @@ If this is a user visible change (including API and protocol changes), add a lin
to the corresponding pull request on https://github.com/syncthing/docs or describe
the documentation changes necessary.
### Authorship
## Authorship
Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
Every author of a code contribution (Go, Javascript, HTML, CSS etc, with the
possible exception of minor typo corrections and similar) is recorded in the
AUTHORS and NICKS files and the in-GUI credits. If this is your first
contribution, a maintainer will add you properly before accepting the
contribution. You need not do so yourself or worry about the fact that the
"authors" automated test fails. However, if your name (such as you want it
presented in the credits) is not visible on your Github profile or in your
commit messages, please assist by providing it here.

48
.github/SECURITY.md vendored Normal file
View File

@@ -0,0 +1,48 @@
## Reporting a Vulnerability
If you believe that you've found a Syncthing-related security vulnerability, please report it by sending email to the address security@syncthing.net. The PGP key for security@syncthing.net (B683AD7B76CAB013) below can be used to send encrypted mail or to verify responses received from that address.
```
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1
mQENBFShFlIBCACsW346HYskhKhxrdZMjyU5Ntsjvg6/ogqINDPoL10/oaIP0G+t
7zzC0K5Cq29ix43kNNLKTJNyPkdTeaJEcqslMUt6tovjHwoKJ073GP0W3KsNvBRg
ffCZOAexGfOsBSL9KHaYGK67Py3TFgtN1H/EmboU1arrLfAMrmqOip++EGqOxjse
gH0qk7Mk1TqEC5Xh3NGE7r1UobAlqdUv5E3v7U11NhAdP1zu/XZ/zvP5mwVQJMLv
iZyeWGliNI8nEeRjYw+S85f4gq7H2mgoeNBN4WwwK1hhz9qpvCsgPW3XqlExTPI4
1vM4PxpiFIuF0zuy2OwsmjrpTCZeBscr4Tj5ABEBAAG0K1N5bmN0aGluZyBTZWN1
cml0eSA8c2VjdXJpdHlAc3luY3RoaW5nLm5ldD6JATgEEwECACIFAlShFlICGwMG
CwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJELaDrXt2yrATB1UH/jnnIa6DekCA
2V36N/2+pFSNLVWeoZQxZ9ne9S7WSubaoK4oCWiuChPSAy5hagnKnNA3a9wrz5iN
0hgCA88NnTU18biZW6HX8xWEd3dnY3fX1sG0XdHuKlFria8ByfcrbShf/CttXkEd
Y5qPHH9aLIMtksBS1MIsRYrOCHiLNYFCKlbjDDCGT3tuk/yaU7aBAFVPIag54afC
zZAIvTIgRruLWkNT/sSEJx9ekhJzO/+pXNbSSHwhoj5OJh7sQjZWwPzNazelk48R
+ku8VpB5Wxgk2MnJPf1RxF+7saBAaeAVZsJcPN+Jp7u9LCFelIRn1ISsHbhLhyqL
cPXqllltLCKJAhwEEAECAAYFAlShFlsACgkQSfWuwLzlJMcEpA//VvZYHGZgZMM0
bBkYnjManYnJkHSQ2pHsjquSFH+fbfq2FxxTk/nQ/IAvBzt8NDTT3ylPOmsl4A8q
r8BxsRNENhU7zDQrKC5NtYUrzXhPGo8qfDwkeLyqd2msUvHn7EH3PNgiN2QaFWxE
21FqoXIpERKJRgRtv1SYZoMyNGjmT2hta1ZbwEfrPJnzjYhneoUGsRApG7p+uPqe
4LRpbG3Fa2vBm/UWUrOe+6jPzvMokjIJbdK0IjXamFAzwYW5fSZaFa94mR6L5f5m
X8Dhp66mBRTx7qk6ldEqptvaYGqihaWP1xbDaApQsGHQujtcBYGp+edlkabuW5h7
stl/7QTFkEPqHT4ybxEX0rLoFUGq56WUlKp27z40keStaXMgfrsxJtkz66Xs8vU4
7qZcLAcPsin+y0toqavtwtM/L7uCMu1yhbFRGJ+JL6saJAqzwS8l7r6E2R7OnVj1
BdASgxx1TgzW6ZFW5p1Iy6mtpkBypsnp8s3UcP3GFRnQe9gi1EXHjzuZAUGafe4Q
juvJ8t8xIcQMFuAylNIVyXvIWJoehqsVY9EBgVtE4y1SRh8XTT3Tddn8ab5fl7uh
HWAY8cRlv6WIOhK8w8oroiYx0SID8jMeEwJBS4DL7qWtJMkDo8ZEJiB+Pkd963MX
05QXt33AvJJ9PmbGCHkcH7198tCmA+e5AQ0EVKEWUgEIAPGczHpa6NdxY0pm+tIp
btiA6gdPE70pJYgJTKX7siCQ2w770H3CBSKmqEXadr7WnfIgUYIDaSxadeGzB/Mn
3SHCYRCqbA7mwu2k4wNNvCEM0xZqFAvaDJ4avlZ5oiMT8IFHKsjC77nkhmfXaIGt
hn10H2MFADjvJYul7vR+Ghg1wGhTGWo2u7VVj9BI+AfvnWaouFI0cx2KNWEI/Ocj
z6jk8nmC3yOEFQECM/hF4lkAOv9CQUa8UcvAr31trzawmV1iSsKjmVZgqd0N4T8f
hUikqUPZGNCRcqEUffTzggIyGPbedFnZw9Di7o1xByxyTrZycemAVqaVGF+9nFLG
pccAEQEAAYkBHwQYAQIACQUCVKEWUgIbDAAKCRC2g617dsqwEzrjB/9q0T8A4XUE
p0g6xq86jhmh4jlEedxrfXUL3R6ejFtuKMThulxEP0xiQ7xLzBhOnvyxLCsVbjp8
0JtJTVCq44UzUiIBuVNRoYG29uXuTUL2UtI27VhjFxMzLDwZL97tbGR6lzdM/+U5
9PC+PIvS0yz13z2t3/x3KUOnBgxZnpy9h4AdKwjrNVtnQdGDXSKlJBLb57TFcS94
f3roZ5Gpw3AWYSmSSiZWbhks2UNzYSQ84LAKlV1NkktO+qRc/pUqr6IxMjOKc7XP
e8u1Nst6fN3GNqZOV+jUYs/fqrJgp1TUWjNTuf22Rl0Idz7XLPJKYFh9W7T/4MbU
M7Q8GZuww1rk
=No/v
-----END PGP PUBLIC KEY BLOCK-----
```

7
.gitignore vendored
View File

@@ -17,3 +17,10 @@ RELEASE
deb
lib/auto/gui.files.go
snapcraft.yaml
prime/
snap/
parts/
stage/
*.snap
*.bz2
/repos

20
.golangci.yml Normal file
View File

@@ -0,0 +1,20 @@
linters-settings:
maligned:
suggest-new: true
linters:
enable-all: true
disable:
- goimports
- depguard
- lll
- gochecknoinits
- gochecknoglobals
- gofmt
- scopelint
service:
golangci-lint-version: 1.16.x
prepare:
- GO111MODULE=on go mod vendor
- go run build.go assets

99
AUTHORS
View File

@@ -1,41 +1,61 @@
# This is the official list of Syncthing authors for copyright purposes.
# The format is:
#
# THIS FILE IS MOSTLY AUTO GENERATED. IF YOU'VE MADE A COMMIT TO THE
# REPOSITORY YOU WILL BE ADDED HERE AUTOMATICALLY WITHOUT THE NEED FOR
# ANY MANUAL ACTION.
#
# That said, you are welcome to correct your name or add a nickname / GitHub
# user name as appropriate. The format is:
#
# Name Name Name (nickname) <email1@example.com> <email2@example.com>
#
# After changing this list, run "go run script/authors.go" to sort and update
# the GUI HTML.
# The in-GUI authors list is periodically automatically updated from the
# contents of this file.
#
Aaron Bieber (qbit) <qbit@deftly.net>
Adam Piggott (ProactiveServices) <aD@simplypeachy.co.uk> <simplypeachy@users.noreply.github.com> <ProactiveServices@users.noreply.github.com>
Adam Piggott (ProactiveServices) <aD@simplypeachy.co.uk> <simplypeachy@users.noreply.github.com> <ProactiveServices@users.noreply.github.com> <adam@proactiveservices.co.uk>
Adel Qalieh (adelq) <aqalieh95@gmail.com> <adelq@users.noreply.github.com>
Alessandro G. (alessandro.g89) <alessandro.g89@gmail.com>
Alexander Graf (alex2108) <register-github@alex-graf.de>
Alexandre Viau (aviau) <alexandre@alexandreviau.net> <aviau@debian.org>
Anderson Mesquita (andersonvom) <andersonvom@gmail.com>
andresvia <andres.via@gmail.com>
Andrew Dunham (andrew-d) <andrew@du.nham.ca>
Andrey D (scienmind) <scintertech@cryptolab.net>
Andrew Rabert (nvllsvm) <ar@nullsum.net> <6550543+nvllsvm@users.noreply.github.com>
Andrey D (scienmind) <scintertech@cryptolab.net> <scienmind@users.noreply.github.com>
André Colomb (acolomb) <src@andre.colomb.de> <github.com@andre.colomb.de>
andyleap <andyleap@gmail.com>
Antoine Lamielle (0x010C) <antoine.lamielle@0x010c.fr> <gh@0x010c.fr>
Antony Male (canton7) <antony.male@gmail.com>
Aranjedeath <Aranjedeath@users.noreply.github.com>
Arthur Axel fREW Schmidt (frioux) <frew@afoolishmanifesto.com> <frioux@gmail.com>
Audrius Butkevicius (AudriusButkevicius) <audrius.butkevicius@gmail.com>
Audrius Butkevicius (AudriusButkevicius) <audrius.butkevicius@gmail.com> <github@audrius.rocks>
BAHADIR YILMAZ <bahadiryilmaz32@gmail.com>
Bart De Vries (mogwa1) <devriesb@gmail.com>
Ben Curthoys (bencurthoys) <ben@bencurthoys.com>
Ben Schulz (uok) <ueomkail@gmail.com> <uok@users.noreply.github.com>
Ben Shepherd (benshep) <bjashepherd@gmail.com>
Ben Sidhom (bsidhom) <bsidhom@gmail.com>
Benedikt Heine (bebehei) <bebe@bebehei.de>
Benedikt Morbach <benedikt.morbach@googlemail.com>
Benno Fünfstück <benno.fuenfstueck@gmail.com>
Benny Ng (tpng) <benny.tpng@gmail.com>
Boris Rybalkin <ribalkin@gmail.com>
Brandon Philips (philips) <brandon@ifup.org>
Brendan Long (brendanlong) <self@brendanlong.com>
Brian R. Becker (brbecker) <brbecker@gmail.com>
Caleb Callaway (cqcallaw) <enlightened.despot@gmail.com>
Carsten Hagemann (Moter8) <moter8@gmail.com>
Cathryne Linenweaver (Cathryne) <cathryne.linenweaver@gmail.com> <Cathryne@users.noreply.github.com>
Carsten Hagemann (carstenhag) <moter8@gmail.com> <carsten@chagemann.de>
Cathryne Linenweaver (Cathryne) <cathryne.linenweaver@gmail.com> <Cathryne@users.noreply.github.com> <katrinleinweber@MAC.local>
Cedric Staniewski (xduugu) <cedric@gmx.ca>
Chris Howie (cdhowie) <me@chrishowie.com>
Chris Joel (cdata) <chris@scriptolo.gy>
Chris Tonkinson <chris@masterbran.ch>
chucic <chucic@seznam.cz>
Colin Kennedy (moshen) <moshen.colin@gmail.com>
Cromefire_ <tim.l@nghorst.net>
Dale Visser <dale.visser@live.com>
Daniel Bergmann (brgmnn) <dan.arne.bergmann@gmail.com> <brgmnn@users.noreply.github.com>
Daniel Harte (norgeous) <daniel@harte.me> <daniel@danielharte.co.uk> <norgeous@users.noreply.github.com>
Daniel Martí (mvdan) <mvdan@mvdan.cc>
@@ -43,36 +63,62 @@ Darshil Chanpura (dtchanpura) <dtchanpura@gmail.com> <dcprime314@gmail.com>
David Rimmer (dinosore) <dinosore@dbrsoftware.co.uk>
Denis A. (dva) <denisva@gmail.com>
Dennis Wilson (snnd) <dw@risu.io>
dependabot-preview[bot] <dependabot-preview[bot]@users.noreply.github.com>
dependabot[bot] <dependabot[bot]@users.noreply.github.com>
derekriemer <derek.riemer@colorado.edu>
desbma <desbma@users.noreply.github.com>
Dmitry Saveliev (dsaveliev) <d.e.saveliev@gmail.com>
Dominik Heidler (asdil12) <dominik@heidler.eu>
Elias Jarlebring (jarlebring) <jarlebring@gmail.com>
Elliot Huffman <thelich2@gmail.com>
Emil Hessman (ceh) <emil@hessman.se>
Erik Meitner (WSGCSysadmin) <e.meitner@willystreet.coop>
Evgeny Kuznetsov <evgeny@kuznetsov.md>
Federico Castagnini (facastagnini) <federico.castagnini@gmail.com>
Felix Ableitner (Nutomic) <me@nutomic.com>
Felix Unterpaintner (bigbear2nd) <bigbear2nd@gmail.com>
Francois-Xavier Gsell (zukoo) <fxgsell@gmail.com>
Frank Isemann (fti7) <frank@isemann.name>
georgespatton <georgespatton@users.noreply.github.com>
Gilli Sigurdsson (gillisig) <gilli@vx.is>
Graham Miln (grahammiln) <graham.miln@dssw.co.uk> <graham.miln@miln.eu>
Han Boetes <han@boetes.org>
Harrison Jones (harrisonhjones) <harrisonhjones@users.noreply.github.com>
Heiko Zuerker (Smiley73) <heiko@zuerker.org>
Hugo Locurcio <hugo.locurcio@hugo.pro>
Iain Barnett <iainspeed@gmail.com>
Ian Johnson (anonymouse64) <ian.johnson@canonical.com> <person.uwsome@gmail.com>
Iskander Sharipov (Alex) <quasilyte@gmail.com>
Jaakko Hannikainen (jgke) <jgke@jgke.fi>
Jacek Szafarkiewicz (hadogenes) <szafar@linux.pl>
Jake Peterson (acogdev) <jake@acogdev.com>
Jakob Borg (calmh) <jakob@nym.se> <jakob@kastelo.net>
James Patterson (jpjp) <jamespatterson@operamail.com> <jpjp@users.noreply.github.com>
janost <janost@tuta.io>
Jaroslav Malec (dzarda) <dzardacz@gmail.com>
jaseg <githubaccount@jaseg.net>
Jaya Chithra (jayachithra) <s.k.jayachithra@gmail.com>
Jens Diemer (jedie) <github.com@jensdiemer.de> <git@jensdiemer.de>
Jerry Jacobs (xor-gate) <jerry.jacobs@xor-gate.org> <xor-gate@users.noreply.github.com>
Jochen Voss (seehuhn) <voss@seehuhn.de>
Johan Andersson <j@i19.se>
Johan Vromans (sciurius) <jvromans@squirrel.nl>
John Rinehart (fuzzybear3965) <johnrichardrinehart@gmail.com>
Jonas Thelemann <e-mail@jonas-thelemann.de>
Jonathan Cross <jcross@gmail.com>
Jose Manuel Delicado (jmdaweb) <jmdaweb@hotmail.com> <jmdaweb@users.noreply.github.com>
Jörg Thalheim <Mic92@users.noreply.github.com>
Kalle Laine <pahakalle@protonmail.com>
Karol Różycki (krozycki) <rozycki.karol@gmail.com>
Keith Turner <kturner@apache.org>
Kelong Cong (kc1212) <kc04bc@gmx.com> <kc1212@users.noreply.github.com>
Ken'ichi Kamada (kamadak) <kamada@nanohz.org>
Kevin Allen (ironmig) <kma1660@gmail.com>
Kevin White, Jr. (kwhite17) <kevinwhite1710@gmail.com>
klemens <ka7@github.com>
Kurt Fitzner (Kudalufi) <kurt@va1der.ca> <kurt.fitzner@gmail.com>
Lars K.W. Gohlke (lkwg82) <lkwg82@gmx.de>
Laurent Arnoud <laurent@spkdev.net>
Laurent Etiemble (letiemble) <laurent.etiemble@gmail.com> <laurent.etiemble@monobjc.net>
Leo Arias (elopio) <yo@elopio.net>
Liu Siyuan (liusy182) <liusy182@gmail.com> <liusy182@hotmail.com>
@@ -82,46 +128,81 @@ Majed Abdulaziz (majedev) <majed.alhajry@gmail.com>
Marc Laporte (marclaporte) <marc@marclaporte.com> <marc@laporte.name>
Marc Pujol (kilburn) <kilburn@la3.org>
Marcin Dziadus (marcindziadus) <dziadus.marcin@gmail.com>
marco-m <marco.molteni@laposte.net>
Mark Pulford (mpx) <mark@kyne.com.au>
Mateusz Naściszewski (mateon1) <matin1111@wp.pl>
Matic Potočnik <hairyfotr@gmail.com>
Matt Burke (burkemw3) <mburke@amplify.com> <burkemw3@gmail.com>
Matt Robenolt <matt@ydekproductions.com>
Matteo Ruina <matteo.ruina@gmail.com>
Maurizio Tomasi <ziotom78@gmail.com>
Max Schulze (kralo) <max.schulze@online.de> <kralo@users.noreply.github.com>
MaximAL <almaximal@ya.ru>
Maxime Thirouin <m@moox.io>
Michael Jephcote (Rewt0r) <rewt0r@gmx.com> <Rewt0r@users.noreply.github.com>
Michael Ploujnikov (plouj) <ploujj@gmail.com>
Michael Tilli (pyfisch) <pyfisch@gmail.com>
Mike Boone <mike@boonedocks.net>
MikeLund <MikeLund@users.noreply.github.com>
Mingxuan Lin <gdlmx@users.noreply.github.com>
Nate Morrison (nrm21) <natemorrison@gmail.com>
Nicholas Rishel (PrototypeNM1) <rishel.nick@gmail.com> <PrototypeNM1@users.noreply.github.com>
Nico Stapelbroek <3368018+nstapelbroek@users.noreply.github.com>
Nicolas Braud-Santoni <nicolas@braud-santoni.eu>
Niels Peter Roest (Niller303) <nielsproest@hotmail.com> <seje.niels@hotmail.com>
Nils Jakobi (thunderstorm99) <jakobi.nils@gmail.com>
Nitroretro <43112364+Nitroretro@users.noreply.github.com>
NoLooseEnds <jon.koslung@gmail.com>
otbutz <tbutz@optitool.de>
Oyebanji Jacob Mayowa <oyebanji05@gmail.com>
Pascal Jungblut (pascalj) <github@pascalj.com> <mail@pascal-jungblut.com>
Pawel Palenica (qepasa) <pawelpalenica11@gmail.com>
Paweł Rozlach <vespian@users.noreply.github.com>
perewa <cavalcante.ten@gmail.com>
Peter Badida <KeyWeeUsr@users.noreply.github.com>
Peter Dave Hello <hsu@peterdavehello.org>
Peter Hoeg (peterhoeg) <peter@speartail.com>
Peter Marquardt (wwwutz) <wwwutz@gmail.com> <wwwutz@googlemail.com>
Phil Davis <phil.davis@inf.org>
Philippe Schommers (filoozoom) <philippe@schommers.be>
Phill Luby (pluby) <phill.luby@newredo.com>
Pier Paolo Ramon <ramonpierre@gmail.com>
Piotr Bejda (piobpl) <piotrb10@gmail.com>
Pramodh KP (pramodhkp) <pramodh.p@directi.com> <1507241+pramodhkp@users.noreply.github.com>
Richard Hartmann <RichiH@users.noreply.github.com>
Robert Carosi (nov1n) <robert@carosi.nl>
Roman Zaynetdinov (zaynetro) <romanznet@gmail.com>
Ross Smith II (rasa) <ross@smithii.com>
rubenbe <github-com-00ff86@vandamme.email>
Ryan Sullivan (KayoticSully) <kayoticsully@gmail.com>
Sacheendra Talluri (sacheendra) <sacheendra.t@gmail.com>
Scott Klupfel (kluppy) <kluppy@going2blue.com>
Sergey Mishin (ralder) <ralder@yandex.ru>
Simon Frei (imsodin) <freisim93@gmail.com>
Sly_tom_cat <slytomcat@mail.ru>
Stefan Kuntz (Stefan-Code) <stefan.github@gmail.com> <Stefan.github@gmail.com>
Stefan Tatschner (rumpelsepp) <stefan@sevenbyte.org> <rumpelsepp@sevenbyte.org>
Stefan Tatschner (rumpelsepp) <stefan@sevenbyte.org> <rumpelsepp@sevenbyte.org> <stefan@rumpelsepp.org>
Suhas Gundimeda (snugghash) <suhas.gundimeda@gmail.com> <snugghash@gmail.com>
Taylor Khan (nelsonkhan) <nelsonkhan@gmail.com>
Thomas Hipp <thomashipp@gmail.com>
Tim Abell (timabell) <tim@timwise.co.uk>
Tim Howes (timhowes) <timhowes@berkeley.edu>
Tobias Nygren (tnn2) <tnn@nygren.pp.se>
Tobias Tom (tobiastom) <t.tom@succont.de>
Tom Jakubowski <tom@crystae.net>
Tomas Cerveny (kozec) <kozec@kozec.com>
Tommy Thorn <tommy-github-email@thorn.ws>
Tully Robinson (tojrobinson) <tully@tojr.org>
Tyler Brazier (tylerbrazier) <tyler@tylerbrazier.com>
Unrud (Unrud) <unrud@openaliasbox.org> <Unrud@users.noreply.github.com>
Veeti Paananen (veeti) <veeti.paananen@rojekti.fi>
Victor Buinsky (buinsky) <vix_booja@tut.by>
Vil Brekin (Vilbrekin) <vilbrekin@gmail.com>
Vladimir Rusinov <vrusinov@google.com>
wangguoliang <liangcszzu@163.com>
William A. Kennington III (wkennington) <william@wkennington.com>
Wulf Weich (wweich) <wweich@users.noreply.github.com> <wweich@gmx.de> <wulf@weich-kr.de>
Xavier O. (damajor) <damajor@gmail.com>
xjtdy888 (xjtdy888) <xjtdy888@163.com>
Yannic A. (eipiminus1) <eipiminusone+github@gmail.com> <eipiminus1@users.noreply.github.com>
佛跳墙 <daoquan@qq.com>

View File

@@ -1,92 +1,73 @@
## Conduct
# Contributor Covenant Code of Conduct
* We are committed to providing a friendly, safe and welcoming
environment for all, regardless of gender, sexual orientation,
disability, ethnicity, religion, or similar personal characteristic.
## Our Pledge
* On IRC, please avoid using overtly sexual nicknames or other nicknames
that might detract from a friendly, safe and welcoming environment for
all.
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, gender identity and expression, level of experience,
education, socio-economic status, nationality, personal appearance, race,
religion, or sexual identity and orientation.
* Please be kind and courteous. There's no need to be mean or rude.
## Our Standards
* Respect that people have differences of opinion and that every design
or implementation choice carries a trade-off and numerous costs. There
is seldom a right answer.
Examples of behavior that contributes to creating a positive environment
include:
* Please keep unstructured critique to a minimum. If you have solid
ideas you want to experiment with, make a fork and see how it works.
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
* We will exclude you from interaction if you insult, demean or harass
anyone. That is not welcome behaviour. We interpret the term
"harassment" as including the definition in the <a
href="http://citizencodeofconduct.org/">Citizen Code of Conduct</a>;
if you have any lack of clarity about what might be included in that
concept, please read their definition. In particular, we don't
tolerate behavior that excludes people in socially marginalized
groups.
Examples of unacceptable behavior by participants include:
* Private harassment is also unacceptable. No matter who you are, if you
feel you have been or are being harassed or made uncomfortable by a
community member, please contact one of the channel ops or any of the
Syncthing core team immediately. Whether you're a regular contributor
or a newcomer, we care about making this community a safe place for
you and we've got your back.
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
* Likewise any spamming, trolling, flaming, baiting or other
attention-stealing behaviour is not welcome.
## Our Responsibilities
## Moderation
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
These are the policies for upholding our community's standards of
conduct in our communication channels, most notably in Syncthing-related
IRC channels and on the web forum.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
1. Remarks that violate the Syncthing standards of conduct, including
hateful, hurtful, oppressive, or exclusionary remarks, are not
allowed. (Cursing is allowed, but never targeting another user, and
never in a hateful manner.)
## Scope
2. Remarks that moderators find inappropriate, whether listed in the
code of conduct or not, are also not allowed.
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
3. Moderators will first respond to such remarks with a warning.
## Enforcement
4. If the warning is unheeded, the user will be "kicked," i.e., kicked
out of the communication channel to cool off.
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at security@syncthing.net. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
5. If the user comes back and continues to make trouble, they will be
banned, i.e., indefinitely excluded.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
6. Moderators may choose at their discretion to un-ban the user if it
was a first offense and they offer the offended party a genuine
apology.
## Attribution
7. If a moderator bans someone and you think it was unjustified, please
take it up with that moderator, or with a different moderator, **in
private**. Complaints about bans in-channel are not allowed.
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
8. Moderators are held to a higher standard than other community
members. If a moderator creates an inappropriate situation, they
should expect less leeway than others.
In the Syncthing community we strive to go the extra step to look out
for each other. Don't just aim to be technically unimpeachable, try to
be your best self. In particular, avoid flirting with offensive or
sensitive issues, particularly if they're off-topic; this all too
often leads to unnecessary fights, hurt feelings, and damaged trust;
worse, it can drive people away from the community entirely.
And if someone takes issue with something you said or did, resist the
urge to be defensive. Just stop doing what it was they complained about
and apologize. Even if you feel you were misinterpreted or unfairly
accused, chances are good there was something you could've communicated
better — remember that it's your responsibility to make your fellow
community members comfortable. Everyone wants to get along and we are
all here first and foremost because we want to talk about cool
technology. You will find that people will be eager to assume good
intent and forgive as long as you earn their trust.
*Adapted from the [Rust Code of Conduct](https://github.com/rust-lang/rust/wiki/Note-development-policy#conduct)*
*Adapted from the [Node.js Policy on Trolling](http://blog.izs.me/post/30036893703/policy-on-trolling)*
[homepage]: https://www.contributor-covenant.org

View File

@@ -1,29 +1,27 @@
FROM golang:1.9 AS builder
FROM golang:1.12 AS builder
WORKDIR /go/src/github.com/syncthing/syncthing
WORKDIR /src
COPY . .
ENV CGO_ENABLED=0
ENV BUILD_HOST=syncthing.net
ENV BUILD_USER=docker
RUN rm -f syncthing && go run build.go build syncthing
RUN rm -f syncthing && go run build.go -no-upgrade build syncthing
FROM alpine
RUN apk add --no-cache ca-certificates
EXPOSE 8384 22000 21027/udp
COPY --from=builder /go/src/github.com/syncthing/syncthing/syncthing /bin/syncthing
VOLUME ["/var/syncthing"]
RUN echo 'syncthing:x:1000:1000::/var/syncthing:/sbin/nologin' >> /etc/passwd \
&& echo 'syncthing:!::0:::::' >> /etc/shadow \
&& mkdir /var/syncthing \
&& chown syncthing /var/syncthing
RUN apk add --no-cache ca-certificates su-exec
USER syncthing
ENV STNOUPGRADE=1
COPY --from=builder /src/syncthing /bin/syncthing
COPY --from=builder /src/script/docker-entrypoint.sh /bin/entrypoint.sh
ENV PUID=1000 PGID=1000
HEALTHCHECK --interval=1m --timeout=10s \
CMD nc -z localhost 8384 || exit 1
ENTRYPOINT ["/bin/syncthing", "-home", "/var/syncthing/config", "-gui-address", "0.0.0.0:8384"]
ENTRYPOINT ["/bin/entrypoint.sh", "-home", "/var/syncthing/config", "-gui-address", "0.0.0.0:8384"]

34
README-Docker.md Normal file
View File

@@ -0,0 +1,34 @@
# Docker Container for Syncthing
Use the Dockerfile in this repo, or pull the `syncthing/syncthing` image
from Docker Hub.
Use the `/var/syncthing` volume to have the synchronized files available on the
host. You can add more folders and map them as you prefer.
Note that Syncthing runs as UID 1000 and GID 1000 by default. These may be
altered with the ``PUID`` and ``PGID`` environment variables.
## Example Usage
```
$ docker pull syncthing/syncthing
$ docker run -p 8384:8384 -p 22000:22000 \
-v /wherever/st-sync:/var/syncthing \
syncthing/syncthing:latest
```
Note that local device discovery will not work with the above command, resulting in poor local transfer rates if local device addresses are not manually configured.
To allow local discovery, the docker host network can be used instead:
```
$ docker pull syncthing/syncthing
$ docker run --network=host \
-v /wherever/st-sync:/var/syncthing \
syncthing/syncthing:latest
```
Be aware that syncthing alone is now in control of what interfaces and ports it
listens on. You can edit the syncthing configuration to change the defaults if
there are conflicts.

View File

@@ -2,7 +2,6 @@
---
[![Latest Downloads](https://img.shields.io/badge/latest-downloads-brightgreen.svg?style=flat-square)](https://build.syncthing.net/latest/)
[![Latest Linux & Cross Build](https://img.shields.io/teamcity/https/build.syncthing.net/s/Syncthing_BuildLinuxCross.svg?style=flat-square&label=linux+%26+cross+build)](https://build.syncthing.net/viewType.html?buildTypeId=Syncthing_BuildLinuxCross&guest=1)
[![Latest Windows Build](https://img.shields.io/teamcity/https/build.syncthing.net/s/Syncthing_BuildWindows.svg?style=flat-square&label=windows+build)](https://build.syncthing.net/viewType.html?buildTypeId=Syncthing_BuildWindows&guest=1)
[![Latest Mac Build](https://img.shields.io/teamcity/https/build.syncthing.net/s/Syncthing_BuildMac.svg?style=flat-square&label=mac+build)](https://build.syncthing.net/viewType.html?buildTypeId=Syncthing_BuildMac&guest=1)

BIN
assets/logo.ico Normal file
View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 160 KiB

417
build.go
View File

@@ -34,30 +34,36 @@ import (
)
var (
versionRe = regexp.MustCompile(`-[0-9]{1,3}-g[0-9a-f]{5,10}`)
goarch string
goos string
noupgrade bool
version string
goVersion float64
race bool
debug = os.Getenv("BUILDDEBUG") != ""
noBuildGopath bool
extraTags string
installSuffix string
pkgdir string
debugBinary bool
versionRe = regexp.MustCompile(`-[0-9]{1,3}-g[0-9a-f]{5,10}`)
goarch string
goos string
noupgrade bool
version string
goCmd string
goVersion float64
race bool
debug = os.Getenv("BUILDDEBUG") != ""
extraTags string
installSuffix string
pkgdir string
cc string
debugBinary bool
coverage bool
timeout = "120s"
gogoProtoVersion = "v1.2.0"
)
type target struct {
name string
debname string
debdeps []string
debpre string
debpost string
description string
buildPkg string
binaryName string
archiveFiles []archiveFile
systemdServices []string
installationFiles []archiveFile
tags []string
}
@@ -110,12 +116,21 @@ var targets = map[string]target{
{src: "etc/linux-systemd/system/syncthing-resume.service", dst: "deb/lib/systemd/system/syncthing-resume.service", perm: 0644},
{src: "etc/linux-systemd/user/syncthing.service", dst: "deb/usr/lib/systemd/user/syncthing.service", perm: 0644},
{src: "etc/firewall-ufw/syncthing", dst: "deb/etc/ufw/applications.d/syncthing", perm: 0644},
{src: "etc/linux-desktop/syncthing-start.desktop", dst: "deb/usr/share/applications/syncthing-start.desktop", perm: 0644},
{src: "etc/linux-desktop/syncthing-ui.desktop", dst: "deb/usr/share/applications/syncthing-ui.desktop", perm: 0644},
{src: "assets/logo-32.png", dst: "deb/usr/share/icons/hicolor/32x32/apps/syncthing.png", perm: 0644},
{src: "assets/logo-64.png", dst: "deb/usr/share/icons/hicolor/64x64/apps/syncthing.png", perm: 0644},
{src: "assets/logo-128.png", dst: "deb/usr/share/icons/hicolor/128x128/apps/syncthing.png", perm: 0644},
{src: "assets/logo-256.png", dst: "deb/usr/share/icons/hicolor/256x256/apps/syncthing.png", perm: 0644},
{src: "assets/logo-512.png", dst: "deb/usr/share/icons/hicolor/512x512/apps/syncthing.png", perm: 0644},
{src: "assets/logo-only.svg", dst: "deb/usr/share/icons/hicolor/scalable/apps/syncthing.svg", perm: 0644},
},
},
"stdiscosrv": {
name: "stdiscosrv",
debname: "syncthing-discosrv",
debdeps: []string{"libc6"},
debpre: "cmd/stdiscosrv/scripts/preinst",
description: "Syncthing Discovery Server",
buildPkg: "github.com/syncthing/syncthing/cmd/stdiscosrv",
binaryName: "stdiscosrv", // .exe will be added automatically for Windows builds
@@ -125,12 +140,17 @@ var targets = map[string]target{
{src: "LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
},
systemdServices: []string{
"cmd/stdiscosrv/etc/linux-systemd/stdiscosrv.service",
},
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "cmd/stdiscosrv/README.md", dst: "deb/usr/share/doc/syncthing-discosrv/README.txt", perm: 0644},
{src: "cmd/stdiscosrv/LICENSE", dst: "deb/usr/share/doc/syncthing-discosrv/LICENSE.txt", perm: 0644},
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing-discosrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-discosrv/AUTHORS.txt", perm: 0644},
{src: "man/stdiscosrv.1", dst: "deb/usr/share/man/man1/stdiscosrv.1", perm: 0644},
{src: "cmd/stdiscosrv/etc/linux-systemd/default", dst: "deb/etc/default/syncthing-discosrv", perm: 0644},
{src: "cmd/stdiscosrv/etc/firewall-ufw/stdiscosrv", dst: "deb/etc/ufw/applications.d/stdiscosrv", perm: 0644},
},
tags: []string{"purego"},
},
@@ -138,6 +158,7 @@ var targets = map[string]target{
name: "strelaysrv",
debname: "syncthing-relaysrv",
debdeps: []string{"libc6"},
debpre: "cmd/strelaysrv/scripts/preinst",
description: "Syncthing Relay Server",
buildPkg: "github.com/syncthing/syncthing/cmd/strelaysrv",
binaryName: "strelaysrv", // .exe will be added automatically for Windows builds
@@ -145,14 +166,21 @@ var targets = map[string]target{
{src: "{{binary}}", dst: "{{binary}}", perm: 0755},
{src: "cmd/strelaysrv/README.md", dst: "README.txt", perm: 0644},
{src: "cmd/strelaysrv/LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
},
systemdServices: []string{
"cmd/strelaysrv/etc/linux-systemd/strelaysrv.service",
},
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "cmd/strelaysrv/README.md", dst: "deb/usr/share/doc/syncthing-relaysrv/README.txt", perm: 0644},
{src: "cmd/strelaysrv/LICENSE", dst: "deb/usr/share/doc/syncthing-relaysrv/LICENSE.txt", perm: 0644},
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing-relaysrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-relaysrv/AUTHORS.txt", perm: 0644},
{src: "man/strelaysrv.1", dst: "deb/usr/share/man/man1/strelaysrv.1", perm: 0644},
{src: "cmd/strelaysrv/etc/linux-systemd/default", dst: "deb/etc/default/syncthing-relaysrv", perm: 0644},
{src: "cmd/strelaysrv/etc/firewall-ufw/strelaysrv", dst: "deb/etc/ufw/applications.d/strelaysrv", perm: 0644},
},
},
"strelaypoolsrv": {
@@ -177,6 +205,19 @@ var targets = map[string]target{
},
}
// These are repos we need to clone to run "go generate"
type dependencyRepo struct {
path string
repo string
commit string
}
var dependencyRepos = []dependencyRepo{
{path: "protobuf", repo: "https://github.com/gogo/protobuf.git", commit: gogoProtoVersion},
{path: "xdr", repo: "https://github.com/calmh/xdr.git", commit: "08e072f9cb16"},
}
func init() {
// The "syncthing" target includes a few more files found in the "etc"
// and "extra" dirs.
@@ -205,41 +246,6 @@ func main() {
}()
}
if gopath := gopath(); gopath == "" {
gopath, err := temporaryBuildDir()
if err != nil {
log.Fatal(err)
}
if !noBuildGopath {
lazyRebuildAssets()
if err := buildGOPATH(gopath); err != nil {
log.Fatal(err)
}
}
os.Setenv("GOPATH", gopath)
log.Println("GOPATH is", gopath)
} else {
inside := false
wd, _ := os.Getwd()
wd, _ = filepath.EvalSymlinks(wd)
for _, p := range filepath.SplitList(gopath) {
p, _ = filepath.EvalSymlinks(p)
if filepath.Join(p, "src/github.com/syncthing/syncthing") == wd {
inside = true
break
}
}
if !inside {
fmt.Println("You seem to have GOPATH set but the Syncthing source not placed correctly within it, which may cause problems.")
}
}
// Set path to $GOPATH/bin:$PATH so that we can for sure find tools we
// might have installed during "build.go setup".
os.Setenv("PATH", fmt.Sprintf("%s%cbin%c%s", os.Getenv("GOPATH"), os.PathSeparator, os.PathListSeparator, os.Getenv("PATH")))
checkArchitecture()
// Invoking build.go with no parameters at all builds everything (incrementally),
// which is what you want for maximum error checking during development.
if flag.NArg() == 0 {
@@ -261,20 +267,8 @@ func main() {
}
}
func checkArchitecture() {
switch goarch {
case "386", "amd64", "arm", "arm64", "ppc64", "ppc64le", "mips", "mipsle":
break
default:
log.Printf("Unknown goarch %q; proceed with caution!", goarch)
}
}
func runCommand(cmd string, target target) {
switch cmd {
case "setup":
setup()
case "install":
var tags []string
if noupgrade {
@@ -322,9 +316,6 @@ func runCommand(cmd string, target target) {
case "snap":
buildSnap(target)
case "clean":
clean()
case "vet":
metalintShort()
@@ -337,13 +328,6 @@ func runCommand(cmd string, target target) {
case "version":
fmt.Println(getVersion())
case "gopath":
gopath, err := temporaryBuildDir()
if err != nil {
log.Fatal(err)
}
fmt.Println(gopath)
default:
log.Fatalf("Unknown command %q", cmd)
}
@@ -352,63 +336,41 @@ func runCommand(cmd string, target target) {
func parseFlags() {
flag.StringVar(&goarch, "goarch", runtime.GOARCH, "GOARCH")
flag.StringVar(&goos, "goos", runtime.GOOS, "GOOS")
flag.StringVar(&goCmd, "gocmd", "go", "Specify `go` command")
flag.BoolVar(&noupgrade, "no-upgrade", noupgrade, "Disable upgrade functionality")
flag.StringVar(&version, "version", getVersion(), "Set compiled in version string")
flag.BoolVar(&race, "race", race, "Use race detector")
flag.BoolVar(&noBuildGopath, "no-build-gopath", noBuildGopath, "Don't build GOPATH, assume it's OK")
flag.StringVar(&extraTags, "tags", extraTags, "Extra tags, space separated")
flag.StringVar(&installSuffix, "installsuffix", installSuffix, "Install suffix, optional")
flag.StringVar(&pkgdir, "pkgdir", "", "Set -pkgdir parameter for `go build`")
flag.StringVar(&cc, "cc", os.Getenv("CC"), "Set CC environment variable for `go build`")
flag.BoolVar(&debugBinary, "debug-binary", debugBinary, "Create unoptimized binary to use with delve, set -gcflags='-N -l' and omit -ldflags")
flag.BoolVar(&coverage, "coverage", coverage, "Write coverage profile of tests to coverage.txt")
flag.Parse()
}
func setup() {
packages := []string{
"github.com/alecthomas/gometalinter",
"github.com/AlekSi/gocov-xml",
"github.com/axw/gocov/gocov",
"github.com/FiloSottile/gvt",
"github.com/golang/lint/golint",
"github.com/gordonklaus/ineffassign",
"github.com/mdempsky/unconvert",
"github.com/mitchellh/go-wordwrap",
"github.com/opennota/check/cmd/...",
"github.com/tsenart/deadcode",
"golang.org/x/net/html",
"golang.org/x/tools/cmd/cover",
"honnef.co/go/tools/cmd/gosimple",
"honnef.co/go/tools/cmd/staticcheck",
"honnef.co/go/tools/cmd/unused",
}
for _, pkg := range packages {
fmt.Println(pkg)
runPrint("go", "get", "-u", pkg)
}
runPrint("go", "install", "-v", "github.com/syncthing/syncthing/vendor/github.com/gogo/protobuf/protoc-gen-gogofast")
}
func test(pkgs ...string) {
lazyRebuildAssets()
useRace := runtime.GOARCH == "amd64"
switch runtime.GOOS {
case "darwin", "linux", "freebsd", "windows":
default:
useRace = false
args := []string{"test", "-short", "-timeout", timeout, "-tags", "purego"}
if runtime.GOARCH == "amd64" {
switch runtime.GOOS {
case "darwin", "linux", "freebsd": // , "windows": # See https://github.com/golang/go/issues/27089
args = append(args, "-race")
}
}
if useRace {
runPrint("go", append([]string{"test", "-short", "-race", "-timeout", "60s", "-tags", "purego"}, pkgs...)...)
} else {
runPrint("go", append([]string{"test", "-short", "-timeout", "60s", "-tags", "purego"}, pkgs...)...)
if coverage {
args = append(args, "-covermode", "atomic", "-coverprofile", "coverage.txt", "-coverpkg", strings.Join(pkgs, ","))
}
runPrint(goCmd, append(args, pkgs...)...)
}
func bench(pkgs ...string) {
lazyRebuildAssets()
runPrint("go", append([]string{"test", "-run", "NONE", "-bench", "."}, pkgs...)...)
runPrint(goCmd, append([]string{"test", "-run", "NONE", "-bench", "."}, pkgs...)...)
}
func install(target target, tags []string) {
@@ -427,7 +389,20 @@ func install(target target, tags []string) {
os.Setenv("GOOS", goos)
os.Setenv("GOARCH", goarch)
runPrint("go", args...)
os.Setenv("CC", cc)
// On Windows generate a special file which the Go compiler will
// automatically use when generating Windows binaries to set things like
// the file icon, version, etc.
if goos == "windows" {
sysoPath, err := shouldBuildSyso(cwd)
if err != nil {
log.Printf("Warning: Windows binaries will not have file information encoded: %v", err)
}
defer shouldCleanupSyso(sysoPath)
}
runPrint(goCmd, args...)
}
func build(target target, tags []string) {
@@ -437,12 +412,29 @@ func build(target target, tags []string) {
rmr(target.BinaryName())
args := []string{"build", "-i", "-v"}
args := []string{"build", "-v"}
args = appendParameters(args, tags, target)
os.Setenv("GOOS", goos)
os.Setenv("GOARCH", goarch)
runPrint("go", args...)
os.Setenv("CC", cc)
// On Windows generate a special file which the Go compiler will
// automatically use when generating Windows binaries to set things like
// the file icon, version, etc.
if goos == "windows" {
cwd, err := os.Getwd()
if err != nil {
log.Fatal(err)
}
sysoPath, err := shouldBuildSyso(cwd)
if err != nil {
log.Printf("Warning: Windows binaries will not have file information encoded: %v", err)
}
defer shouldCleanupSyso(sysoPath)
}
runPrint(goCmd, args...)
}
func appendParameters(args []string, tags []string, target target) []string {
@@ -577,9 +569,15 @@ func buildDeb(target target) {
for _, dep := range target.debdeps {
args = append(args, "-d", dep)
}
for _, service := range target.systemdServices {
args = append(args, "--deb-systemd", service)
}
if target.debpost != "" {
args = append(args, "--after-upgrade", target.debpost)
}
if target.debpre != "" {
args = append(args, "--before-install", target.debpre)
}
runPrint("fpm", args...)
}
@@ -599,6 +597,8 @@ func buildSnap(target target) {
snaparch := goarch
if snaparch == "armhf" {
goarch = "arm"
} else if snaparch == "i386" {
goarch = "386"
}
snapver := version
if strings.HasPrefix(snapver, "v") {
@@ -609,9 +609,10 @@ func buildSnap(target target) {
snapgrade = "stable"
}
err = tmpl.Execute(f, map[string]string{
"Version": snapver,
"Architecture": snaparch,
"Grade": snapgrade,
"Version": snapver,
"HostArchitecture": runtime.GOARCH,
"TargetArchitecture": snaparch,
"Grade": snapgrade,
})
if err != nil {
log.Fatal(err)
@@ -621,6 +622,56 @@ func buildSnap(target target) {
runPrint("snapcraft")
}
func shouldBuildSyso(dir string) (string, error) {
jsonPath := filepath.Join(dir, "versioninfo.json")
file, err := os.Create(filepath.Join(dir, "versioninfo.json"))
if err != nil {
return "", errors.New("failed to create " + jsonPath + ": " + err.Error())
}
major, minor, patch, build := semanticVersion()
fmt.Fprintf(file, `{
"FixedFileInfo": {
"FileVersion": {
"Major": %s,
"Minor": %s,
"Patch": %s,
"Build": %s
}
},
"StringFileInfo": {
"FileDescription": "Open Source Continuous File Synchronization",
"LegalCopyright": "The Syncthing Authors",
"ProductVersion": "%s",
"ProductName": "Syncthing"
},
"IconPath": "assets/logo.ico"
}`, major, minor, patch, build, getVersion())
file.Close()
defer func() {
if err := os.Remove(jsonPath); err != nil {
log.Printf("Warning: unable to remove generated %s: %v. Please remove it manually.", jsonPath, err)
}
}()
sysoPath := filepath.Join(dir, "cmd", "syncthing", "resource.syso")
if _, err := runError("goversioninfo", "-o", sysoPath); err != nil {
return "", errors.New("failed to create " + sysoPath + ": " + err.Error())
}
return sysoPath, nil
}
func shouldCleanupSyso(sysoFilePath string) {
if sysoFilePath == "" {
return
}
if err := os.Remove(sysoFilePath); err != nil {
log.Printf("Warning: unable to remove generated %s: %v. Please remove it manually.", sysoFilePath, err)
}
}
// copyFile copies a file from src to dst, ensuring the containing directory
// exists. The permission bits are copied as well. If dst already exists and
// the contents are identical to src the modification time is not updated.
@@ -668,12 +719,12 @@ func listFiles(dir string) []string {
}
func rebuildAssets() {
runPipe("lib/auto/gui.files.go", "go", "run", "script/genassets.go", "gui")
runPipe("cmd/strelaypoolsrv/auto/gui.go", "go", "run", "script/genassets.go", "cmd/strelaypoolsrv/gui")
os.Setenv("SOURCE_DATE_EPOCH", fmt.Sprint(buildStamp()))
runPrint(goCmd, "generate", "github.com/syncthing/syncthing/lib/auto", "github.com/syncthing/syncthing/cmd/strelaypoolsrv/auto")
}
func lazyRebuildAssets() {
if shouldRebuildAssets("lib/auto/gui.files.go", "gui") || shouldRebuildAssets("cmd/strelaypoolsrv/auto/gui.go", "cmd/strelaypoolsrv/auto/gui") {
if shouldRebuildAssets("lib/auto/gui.files.go", "gui") || shouldRebuildAssets("cmd/strelaypoolsrv/auto/gui.files.go", "cmd/strelaypoolsrv/auto/gui") {
rebuildAssets()
}
}
@@ -705,12 +756,21 @@ func shouldRebuildAssets(target, srcdir string) bool {
}
func proto() {
runPrint("go", "generate", "github.com/syncthing/syncthing/lib/...", "github.com/syncthing/syncthing/cmd/stdiscosrv")
runPrint(goCmd, "get", fmt.Sprintf("github.com/gogo/protobuf/protoc-gen-gogofast@%v", gogoProtoVersion))
os.MkdirAll("repos", 0755)
for _, dep := range dependencyRepos {
path := filepath.Join("repos", dep.path)
if _, err := os.Stat(path); err != nil {
runPrintInDir("repos", "git", "clone", dep.repo, dep.path)
runPrintInDir(path, "git", "checkout", dep.commit)
}
}
runPrint(goCmd, "generate", "github.com/syncthing/syncthing/lib/...", "github.com/syncthing/syncthing/cmd/stdiscosrv")
}
func translate() {
os.Chdir("gui/default/assets/lang")
runPipe("lang-en-new.json", "go", "run", "../../../../script/translate.go", "lang-en.json", "../../../")
runPipe("lang-en-new.json", goCmd, "run", "../../../../script/translate.go", "lang-en.json", "../../../")
os.Remove("lang-en.json")
err := os.Rename("lang-en-new.json", "lang-en.json")
if err != nil {
@@ -721,12 +781,7 @@ func translate() {
func transifex() {
os.Chdir("gui/default/assets/lang")
runPrint("go", "run", "../../../../script/transifexdl.go")
}
func clean() {
rmr("bin")
rmr(filepath.Join(os.Getenv("GOPATH"), fmt.Sprintf("pkg/%s_%s/github.com/syncthing", goos, goarch)))
runPrint(goCmd, "run", "../../../../script/transifexdl.go")
}
func ldflags() string {
@@ -737,10 +792,10 @@ func ldflags() string {
b := new(bytes.Buffer)
b.WriteString("-w")
fmt.Fprintf(b, " -X main.Version%c%s", sep, version)
fmt.Fprintf(b, " -X main.BuildStamp%c%d", sep, buildStamp())
fmt.Fprintf(b, " -X main.BuildUser%c%s", sep, buildUser())
fmt.Fprintf(b, " -X main.BuildHost%c%s", sep, buildHost())
fmt.Fprintf(b, " -X github.com/syncthing/syncthing/lib/build.Version%c%s", sep, version)
fmt.Fprintf(b, " -X github.com/syncthing/syncthing/lib/build.Stamp%c%d", sep, buildStamp())
fmt.Fprintf(b, " -X github.com/syncthing/syncthing/lib/build.User%c%s", sep, buildUser())
fmt.Fprintf(b, " -X github.com/syncthing/syncthing/lib/build.Host%c%s", sep, buildHost())
return b.String()
}
@@ -797,6 +852,15 @@ func getVersion() string {
return "unknown-dev"
}
func semanticVersion() (major, minor, patch, build string) {
r := regexp.MustCompile(`v(?P<Major>\d+)\.(?P<Minor>\d+).(?P<Patch>\d+).*\+(?P<CommitsAhead>\d+)`)
matches := r.FindStringSubmatch(getVersion())
if len(matches) != 5 {
return "0", "0", "0", "0"
}
return matches[1], matches[2], matches[3], matches[4]
}
func getBranchSuffix() string {
bs, err := runError("git", "branch", "-a", "--contains")
if err != nil {
@@ -891,7 +955,7 @@ func buildHost() string {
func buildArch() string {
os := goos
if os == "darwin" {
os = "macosx"
os = "macos"
}
return fmt.Sprintf("%s-%s", os, goarch)
}
@@ -914,6 +978,10 @@ func runError(cmd string, args ...string) ([]byte, error) {
}
func runPrint(cmd string, args ...string) {
runPrintInDir(".", cmd, args...)
}
func runPrintInDir(dir string, cmd string, args ...string) {
if debug {
t0 := time.Now()
log.Println("runPrint:", cmd, strings.Join(args, " "))
@@ -924,6 +992,7 @@ func runPrint(cmd string, args ...string) {
ecmd := exec.Command(cmd, args...)
ecmd.Stdout = os.Stdout
ecmd.Stderr = os.Stderr
ecmd.Dir = dir
err := ecmd.Run()
if err != nil {
log.Fatal(err)
@@ -1143,12 +1212,12 @@ func windowsCodesign(file string) {
func metalint() {
lazyRebuildAssets()
runPrint("go", "test", "-run", "Metalint", "./meta")
runPrint(goCmd, "test", "-run", "Metalint", "./meta")
}
func metalintShort() {
lazyRebuildAssets()
runPrint("go", "test", "-short", "-run", "Metalint", "./meta")
runPrint(goCmd, "test", "-short", "-run", "Metalint", "./meta")
}
func temporaryBuildDir() (string, error) {
@@ -1177,90 +1246,6 @@ func temporaryBuildDir() (string, error) {
return filepath.Join(tmpDir, base), nil
}
func buildGOPATH(gopath string) error {
pkg := filepath.Join(gopath, "src/github.com/syncthing/syncthing")
dirs := []string{"cmd", "lib", "meta", "script", "test", "vendor"}
if debug {
t0 := time.Now()
log.Println("build temporary GOPATH in", gopath)
defer func() {
log.Println("... in", time.Since(t0))
}()
}
// Walk the sources and copy the files into the temporary GOPATH.
// Remember which files are supposed to be present so we can clean
// out everything else in the next step. The copyFile() step will
// only actually copy the file if it doesn't exist or the contents
// differ.
exists := map[string]struct{}{}
for _, dir := range dirs {
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
dst := filepath.Join(pkg, path)
exists[dst] = struct{}{}
if err := copyFile(path, dst, info.Mode()); err != nil {
return err
}
return nil
})
if err != nil {
return err
}
}
// Walk the temporary GOPATH and remove any files that we wouldn't
// have copied there in the previous step.
filepath.Walk(pkg, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
if _, ok := exists[path]; !ok {
os.Remove(path)
}
return nil
})
return nil
}
func gopath() string {
if gopath := os.Getenv("GOPATH"); gopath != "" {
// The env var is set, use that.
return gopath
}
// Ask Go what it thinks.
bs, err := runError("go", "env", "GOPATH")
if err != nil {
return ""
}
// We got something. Check if we are in fact available in that location.
gopath := string(bs)
if _, err := os.Stat(filepath.Join(gopath, "src/github.com/syncthing/syncthing/build.go")); err == nil {
// That seems to be the gopath.
return gopath
}
// The gopath is not valid.
return ""
}
func (t target) BinaryName() string {
if goos == "windows" {
return t.binaryName + ".exe"

View File

@@ -59,8 +59,8 @@ case "${1:-default}" in
go run script/authors.go
build transifex
pushd man ; ./refresh.sh ; popd
git add -A gui man
git commit -m 'gui, man: Update docs & translations'
git add -A gui man AUTHORS
git commit -m 'gui, man, authors: Update docs, translations, and contributors'
;;
noupgrade)

View File

@@ -1,19 +0,0 @@
Copyright (C) 2014 Audrius Butkevičius
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
- The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -1,115 +1,95 @@
// Copyright (C) 2014 Audrius Butkevičius
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"context"
"crypto/tls"
"fmt"
"net"
"net/http"
"strings"
"github.com/AudriusButkevicius/cli"
"github.com/syncthing/syncthing/lib/config"
)
type APIClient struct {
httpClient http.Client
endpoint string
apikey string
username string
password string
id string
csrf string
http.Client
cfg config.GUIConfiguration
apikey string
}
var instance *APIClient
func getClient(c *cli.Context) *APIClient {
if instance != nil {
return instance
}
endpoint := c.GlobalString("endpoint")
if !strings.HasPrefix(endpoint, "http") {
endpoint = "http://" + endpoint
}
func getClient(cfg config.GUIConfiguration) *APIClient {
httpClient := http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
InsecureSkipVerify: c.GlobalBool("insecure"),
InsecureSkipVerify: true,
},
DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
return net.Dial(cfg.Network(), cfg.Address())
},
},
}
client := APIClient{
httpClient: httpClient,
endpoint: endpoint,
apikey: c.GlobalString("apikey"),
username: c.GlobalString("username"),
password: c.GlobalString("password"),
return &APIClient{
Client: httpClient,
cfg: cfg,
apikey: cfg.APIKey,
}
if client.apikey == "" {
request, err := http.NewRequest("GET", client.endpoint, nil)
die(err)
response := client.handleRequest(request)
client.id = response.Header.Get("X-Syncthing-ID")
if client.id == "" {
die("Failed to get device ID")
}
for _, item := range response.Cookies() {
if item.Name == "CSRF-Token-"+client.id[:5] {
client.csrf = item.Value
goto csrffound
}
}
die("Failed to get CSRF token")
csrffound:
}
instance = &client
return &client
}
func (client *APIClient) handleRequest(request *http.Request) *http.Response {
if client.apikey != "" {
request.Header.Set("X-API-Key", client.apikey)
func (c *APIClient) Endpoint() string {
if c.cfg.Network() == "unix" {
return "http://unix/"
}
if client.username != "" || client.password != "" {
request.SetBasicAuth(client.username, client.password)
}
if client.csrf != "" {
request.Header.Set("X-CSRF-Token-"+client.id[:5], client.csrf)
url := c.cfg.URL()
if !strings.HasSuffix(url, "/") {
url += "/"
}
return url
}
response, err := client.httpClient.Do(request)
die(err)
func (c *APIClient) Do(req *http.Request) (*http.Response, error) {
req.Header.Set("X-API-Key", c.apikey)
resp, err := c.Client.Do(req)
if err != nil {
return nil, err
}
return resp, checkResponse(resp)
}
func (c *APIClient) Get(url string) (*http.Response, error) {
request, err := http.NewRequest("GET", c.Endpoint()+"rest/"+url, nil)
if err != nil {
return nil, err
}
return c.Do(request)
}
func (c *APIClient) Post(url, body string) (*http.Response, error) {
request, err := http.NewRequest("POST", c.Endpoint()+"rest/"+url, bytes.NewBufferString(body))
if err != nil {
return nil, err
}
return c.Do(request)
}
func checkResponse(response *http.Response) error {
if response.StatusCode == 404 {
die("Invalid endpoint or API call")
} else if response.StatusCode == 401 {
die("Invalid username or password")
return fmt.Errorf("Invalid endpoint or API call")
} else if response.StatusCode == 403 {
if client.apikey == "" {
die("Invalid CSRF token")
}
die("Invalid API key")
return fmt.Errorf("Invalid API key")
} else if response.StatusCode != 200 {
body := strings.TrimSpace(string(responseToBArray(response)))
if body != "" {
die(body)
data, err := responseToBArray(response)
if err != nil {
return err
}
die("Unknown HTTP status returned: " + response.Status)
body := strings.TrimSpace(string(data))
return fmt.Errorf("Unexpected HTTP status returned: %s\n%s", response.Status, body)
}
return response
}
func httpGet(c *cli.Context, url string) *http.Response {
client := getClient(c)
request, err := http.NewRequest("GET", client.endpoint+"/rest/"+url, nil)
die(err)
return client.handleRequest(request)
}
func httpPost(c *cli.Context, url string, body string) *http.Response {
client := getClient(c)
request, err := http.NewRequest("POST", client.endpoint+"/rest/"+url, bytes.NewBufferString(body))
die(err)
return client.handleRequest(request)
return nil
}

View File

@@ -1,188 +0,0 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"fmt"
"strings"
"github.com/AudriusButkevicius/cli"
"github.com/syncthing/syncthing/lib/config"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "devices",
HideHelp: true,
Usage: "Device command group",
Subcommands: []cli.Command{
{
Name: "list",
Usage: "List registered devices",
Requires: &cli.Requires{},
Action: devicesList,
},
{
Name: "add",
Usage: "Add a new device",
Requires: &cli.Requires{"device id", "device name?"},
Action: devicesAdd,
},
{
Name: "remove",
Usage: "Remove an existing device",
Requires: &cli.Requires{"device id"},
Action: devicesRemove,
},
{
Name: "get",
Usage: "Get a property of a device",
Requires: &cli.Requires{"device id", "property"},
Action: devicesGet,
},
{
Name: "set",
Usage: "Set a property of a device",
Requires: &cli.Requires{"device id", "property", "value..."},
Action: devicesSet,
},
},
})
}
func devicesList(c *cli.Context) {
cfg := getConfig(c)
first := true
writer := newTableWriter()
for _, device := range cfg.Devices {
if !first {
fmt.Fprintln(writer)
}
fmt.Fprintln(writer, "ID:\t", device.DeviceID, "\t")
fmt.Fprintln(writer, "Name:\t", device.Name, "\t(name)")
fmt.Fprintln(writer, "Address:\t", strings.Join(device.Addresses, " "), "\t(address)")
fmt.Fprintln(writer, "Compression:\t", device.Compression, "\t(compression)")
fmt.Fprintln(writer, "Certificate name:\t", device.CertName, "\t(certname)")
fmt.Fprintln(writer, "Introducer:\t", device.Introducer, "\t(introducer)")
first = false
}
writer.Flush()
}
func devicesAdd(c *cli.Context) {
nid := c.Args()[0]
id := parseDeviceID(nid)
newDevice := config.DeviceConfiguration{
DeviceID: id,
Name: nid,
Addresses: []string{"dynamic"},
}
if len(c.Args()) > 1 {
newDevice.Name = c.Args()[1]
}
if len(c.Args()) > 2 {
addresses := c.Args()[2:]
for _, item := range addresses {
if item == "dynamic" {
continue
}
validAddress(item)
}
newDevice.Addresses = addresses
}
cfg := getConfig(c)
for _, device := range cfg.Devices {
if device.DeviceID == id {
die("Device " + nid + " already exists")
}
}
cfg.Devices = append(cfg.Devices, newDevice)
setConfig(c, cfg)
}
func devicesRemove(c *cli.Context) {
nid := c.Args()[0]
id := parseDeviceID(nid)
if nid == getMyID(c) {
die("Cannot remove yourself")
}
cfg := getConfig(c)
for i, device := range cfg.Devices {
if device.DeviceID == id {
last := len(cfg.Devices) - 1
cfg.Devices[i] = cfg.Devices[last]
cfg.Devices = cfg.Devices[:last]
setConfig(c, cfg)
return
}
}
die("Device " + nid + " not found")
}
func devicesGet(c *cli.Context) {
nid := c.Args()[0]
id := parseDeviceID(nid)
arg := c.Args()[1]
cfg := getConfig(c)
for _, device := range cfg.Devices {
if device.DeviceID != id {
continue
}
switch strings.ToLower(arg) {
case "name":
fmt.Println(device.Name)
case "address":
fmt.Println(strings.Join(device.Addresses, "\n"))
case "compression":
fmt.Println(device.Compression.String())
case "certname":
fmt.Println(device.CertName)
case "introducer":
fmt.Println(device.Introducer)
default:
die("Invalid property: " + arg + "\nAvailable properties: name, address, compression, certname, introducer")
}
return
}
die("Device " + nid + " not found")
}
func devicesSet(c *cli.Context) {
nid := c.Args()[0]
id := parseDeviceID(nid)
arg := c.Args()[1]
config := getConfig(c)
for i, device := range config.Devices {
if device.DeviceID != id {
continue
}
switch strings.ToLower(arg) {
case "name":
config.Devices[i].Name = strings.Join(c.Args()[2:], " ")
case "address":
for _, item := range c.Args()[2:] {
if item == "dynamic" {
continue
}
validAddress(item)
}
config.Devices[i].Addresses = c.Args()[2:]
case "compression":
err := config.Devices[i].Compression.UnmarshalText([]byte(c.Args()[2]))
die(err)
case "certname":
config.Devices[i].CertName = strings.Join(c.Args()[2:], " ")
case "introducer":
config.Devices[i].Introducer = parseBool(c.Args()[2])
default:
die("Invalid property: " + arg + "\nAvailable properties: name, address, compression, certname, introducer")
}
setConfig(c, config)
return
}
die("Device " + nid + " not found")
}

View File

@@ -1,67 +0,0 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"encoding/json"
"fmt"
"strings"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "errors",
HideHelp: true,
Usage: "Error command group",
Subcommands: []cli.Command{
{
Name: "show",
Usage: "Show pending errors",
Requires: &cli.Requires{},
Action: errorsShow,
},
{
Name: "push",
Usage: "Push an error to active clients",
Requires: &cli.Requires{"error message..."},
Action: errorsPush,
},
{
Name: "clear",
Usage: "Clear pending errors",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/error/clear"),
},
},
})
}
func errorsShow(c *cli.Context) {
response := httpGet(c, "system/error")
var data map[string][]map[string]interface{}
json.Unmarshal(responseToBArray(response), &data)
writer := newTableWriter()
for _, item := range data["errors"] {
time := item["time"].(string)[:19]
time = strings.Replace(time, "T", " ", 1)
err := item["error"].(string)
err = strings.TrimSpace(err)
fmt.Fprintln(writer, time+":\t"+err)
}
writer.Flush()
}
func errorsPush(c *cli.Context) {
err := strings.Join(c.Args(), " ")
response := httpPost(c, "system/error", strings.TrimSpace(err))
if response.StatusCode != 200 {
err = fmt.Sprint("Failed to push error\nStatus code: ", response.StatusCode)
body := string(responseToBArray(response))
if body != "" {
err += "\nBody: " + body
}
die(err)
}
}

View File

@@ -1,361 +0,0 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"fmt"
"path/filepath"
"strings"
"github.com/AudriusButkevicius/cli"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/fs"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "folders",
HideHelp: true,
Usage: "Folder command group",
Subcommands: []cli.Command{
{
Name: "list",
Usage: "List available folders",
Requires: &cli.Requires{},
Action: foldersList,
},
{
Name: "add",
Usage: "Add a new folder",
Requires: &cli.Requires{"folder id", "directory"},
Action: foldersAdd,
},
{
Name: "remove",
Usage: "Remove an existing folder",
Requires: &cli.Requires{"folder id"},
Action: foldersRemove,
},
{
Name: "override",
Usage: "Override changes from other nodes for a master folder",
Requires: &cli.Requires{"folder id"},
Action: foldersOverride,
},
{
Name: "get",
Usage: "Get a property of a folder",
Requires: &cli.Requires{"folder id", "property"},
Action: foldersGet,
},
{
Name: "set",
Usage: "Set a property of a folder",
Requires: &cli.Requires{"folder id", "property", "value..."},
Action: foldersSet,
},
{
Name: "unset",
Usage: "Unset a property of a folder",
Requires: &cli.Requires{"folder id", "property"},
Action: foldersUnset,
},
{
Name: "devices",
Usage: "Folder devices command group",
HideHelp: true,
Subcommands: []cli.Command{
{
Name: "list",
Usage: "List of devices which the folder is shared with",
Requires: &cli.Requires{"folder id"},
Action: foldersDevicesList,
},
{
Name: "add",
Usage: "Share a folder with a device",
Requires: &cli.Requires{"folder id", "device id"},
Action: foldersDevicesAdd,
},
{
Name: "remove",
Usage: "Unshare a folder with a device",
Requires: &cli.Requires{"folder id", "device id"},
Action: foldersDevicesRemove,
},
{
Name: "clear",
Usage: "Unshare a folder with all devices",
Requires: &cli.Requires{"folder id"},
Action: foldersDevicesClear,
},
},
},
},
})
}
func foldersList(c *cli.Context) {
cfg := getConfig(c)
first := true
writer := newTableWriter()
for _, folder := range cfg.Folders {
if !first {
fmt.Fprintln(writer)
}
fs := folder.Filesystem()
fmt.Fprintln(writer, "ID:\t", folder.ID, "\t")
fmt.Fprintln(writer, "Path:\t", fs.URI(), "\t(directory)")
fmt.Fprintln(writer, "Path type:\t", fs.Type(), "\t(directory-type)")
fmt.Fprintln(writer, "Folder type:\t", folder.Type, "\t(type)")
fmt.Fprintln(writer, "Ignore permissions:\t", folder.IgnorePerms, "\t(permissions)")
fmt.Fprintln(writer, "Rescan interval in seconds:\t", folder.RescanIntervalS, "\t(rescan)")
if folder.Versioning.Type != "" {
fmt.Fprintln(writer, "Versioning:\t", folder.Versioning.Type, "\t(versioning)")
for key, value := range folder.Versioning.Params {
fmt.Fprintf(writer, "Versioning %s:\t %s \t(versioning-%s)\n", key, value, key)
}
}
first = false
}
writer.Flush()
}
func foldersAdd(c *cli.Context) {
cfg := getConfig(c)
abs, err := filepath.Abs(c.Args()[1])
die(err)
folder := config.FolderConfiguration{
ID: c.Args()[0],
Path: filepath.Clean(abs),
FilesystemType: fs.FilesystemTypeBasic,
}
cfg.Folders = append(cfg.Folders, folder)
setConfig(c, cfg)
}
func foldersRemove(c *cli.Context) {
cfg := getConfig(c)
rid := c.Args()[0]
for i, folder := range cfg.Folders {
if folder.ID == rid {
last := len(cfg.Folders) - 1
cfg.Folders[i] = cfg.Folders[last]
cfg.Folders = cfg.Folders[:last]
setConfig(c, cfg)
return
}
}
die("Folder " + rid + " not found")
}
func foldersOverride(c *cli.Context) {
cfg := getConfig(c)
rid := c.Args()[0]
for _, folder := range cfg.Folders {
if folder.ID == rid && folder.Type == config.FolderTypeSendOnly {
response := httpPost(c, "db/override", "")
if response.StatusCode != 200 {
err := fmt.Sprint("Failed to override changes\nStatus code: ", response.StatusCode)
body := string(responseToBArray(response))
if body != "" {
err += "\nBody: " + body
}
die(err)
}
return
}
}
die("Folder " + rid + " not found or folder not master")
}
func foldersGet(c *cli.Context) {
cfg := getConfig(c)
rid := c.Args()[0]
arg := strings.ToLower(c.Args()[1])
for _, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
if strings.HasPrefix(arg, "versioning-") {
arg = arg[11:]
value, ok := folder.Versioning.Params[arg]
if ok {
fmt.Println(value)
return
}
die("Versioning property " + c.Args()[1][11:] + " not found")
}
switch arg {
case "directory":
fmt.Println(folder.Filesystem().URI())
case "directory-type":
fmt.Println(folder.Filesystem().Type())
case "type":
fmt.Println(folder.Type)
case "permissions":
fmt.Println(folder.IgnorePerms)
case "rescan":
fmt.Println(folder.RescanIntervalS)
case "versioning":
if folder.Versioning.Type != "" {
fmt.Println(folder.Versioning.Type)
}
default:
die("Invalid property: " + c.Args()[1] + "\nAvailable properties: directory, directory-type, type, permissions, versioning, versioning-<key>")
}
return
}
die("Folder " + rid + " not found")
}
func foldersSet(c *cli.Context) {
rid := c.Args()[0]
arg := strings.ToLower(c.Args()[1])
val := strings.Join(c.Args()[2:], " ")
cfg := getConfig(c)
for i, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
if strings.HasPrefix(arg, "versioning-") {
cfg.Folders[i].Versioning.Params[arg[11:]] = val
setConfig(c, cfg)
return
}
switch arg {
case "directory":
cfg.Folders[i].Path = val
case "directory-type":
var fsType fs.FilesystemType
fsType.UnmarshalText([]byte(val))
cfg.Folders[i].FilesystemType = fsType
case "type":
var t config.FolderType
if err := t.UnmarshalText([]byte(val)); err != nil {
die("Invalid folder type: " + err.Error())
}
cfg.Folders[i].Type = t
case "permissions":
cfg.Folders[i].IgnorePerms = parseBool(val)
case "rescan":
cfg.Folders[i].RescanIntervalS = parseInt(val)
case "versioning":
cfg.Folders[i].Versioning.Type = val
default:
die("Invalid property: " + c.Args()[1] + "\nAvailable properties: directory, master, permissions, versioning, versioning-<key>")
}
setConfig(c, cfg)
return
}
die("Folder " + rid + " not found")
}
func foldersUnset(c *cli.Context) {
rid := c.Args()[0]
arg := strings.ToLower(c.Args()[1])
cfg := getConfig(c)
for i, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
if strings.HasPrefix(arg, "versioning-") {
arg = arg[11:]
if _, ok := folder.Versioning.Params[arg]; ok {
delete(cfg.Folders[i].Versioning.Params, arg)
setConfig(c, cfg)
return
}
die("Versioning property " + c.Args()[1][11:] + " not found")
}
switch arg {
case "versioning":
cfg.Folders[i].Versioning.Type = ""
cfg.Folders[i].Versioning.Params = make(map[string]string)
default:
die("Invalid property: " + c.Args()[1] + "\nAvailable properties: versioning, versioning-<key>")
}
setConfig(c, cfg)
return
}
die("Folder " + rid + " not found")
}
func foldersDevicesList(c *cli.Context) {
rid := c.Args()[0]
cfg := getConfig(c)
for _, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
for _, device := range folder.Devices {
fmt.Println(device.DeviceID)
}
return
}
die("Folder " + rid + " not found")
}
func foldersDevicesAdd(c *cli.Context) {
rid := c.Args()[0]
nid := parseDeviceID(c.Args()[1])
cfg := getConfig(c)
for i, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
for _, device := range folder.Devices {
if device.DeviceID == nid {
die("Device " + c.Args()[1] + " is already part of this folder")
}
}
for _, device := range cfg.Devices {
if device.DeviceID == nid {
cfg.Folders[i].Devices = append(folder.Devices, config.FolderDeviceConfiguration{
DeviceID: device.DeviceID,
})
setConfig(c, cfg)
return
}
}
die("Device " + c.Args()[1] + " not found in device list")
}
die("Folder " + rid + " not found")
}
func foldersDevicesRemove(c *cli.Context) {
rid := c.Args()[0]
nid := parseDeviceID(c.Args()[1])
cfg := getConfig(c)
for ri, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
for ni, device := range folder.Devices {
if device.DeviceID == nid {
last := len(folder.Devices) - 1
cfg.Folders[ri].Devices[ni] = folder.Devices[last]
cfg.Folders[ri].Devices = cfg.Folders[ri].Devices[:last]
setConfig(c, cfg)
return
}
}
die("Device " + c.Args()[1] + " not found")
}
die("Folder " + rid + " not found")
}
func foldersDevicesClear(c *cli.Context) {
rid := c.Args()[0]
cfg := getConfig(c)
for i, folder := range cfg.Folders {
if folder.ID != rid {
continue
}
cfg.Folders[i].Devices = []config.FolderDeviceConfiguration{}
setConfig(c, cfg)
return
}
die("Folder " + rid + " not found")
}

View File

@@ -1,78 +0,0 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"encoding/json"
"fmt"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, []cli.Command{
{
Name: "id",
Usage: "Get ID of the Syncthing client",
Requires: &cli.Requires{},
Action: generalID,
},
{
Name: "status",
Usage: "Configuration status, whether or not a restart is required for changes to take effect",
Requires: &cli.Requires{},
Action: generalStatus,
},
{
Name: "restart",
Usage: "Restart syncthing",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/restart"),
},
{
Name: "shutdown",
Usage: "Shutdown syncthing",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/shutdown"),
},
{
Name: "reset",
Usage: "Reset syncthing deleting all folders and devices",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/reset"),
},
{
Name: "upgrade",
Usage: "Upgrade syncthing (if a newer version is available)",
Requires: &cli.Requires{},
Action: wrappedHTTPPost("system/upgrade"),
},
{
Name: "version",
Usage: "Syncthing client version",
Requires: &cli.Requires{},
Action: generalVersion,
},
}...)
}
func generalID(c *cli.Context) {
fmt.Println(getMyID(c))
}
func generalStatus(c *cli.Context) {
response := httpGet(c, "system/config/insync")
var status struct{ ConfigInSync bool }
json.Unmarshal(responseToBArray(response), &status)
if !status.ConfigInSync {
die("Config out of sync")
}
fmt.Println("Config in sync")
}
func generalVersion(c *cli.Context) {
response := httpGet(c, "system/version")
version := make(map[string]interface{})
json.Unmarshal(responseToBArray(response), &version)
prettyPrintJSON(version)
}

View File

@@ -1,127 +0,0 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"fmt"
"strings"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "gui",
HideHelp: true,
Usage: "GUI command group",
Subcommands: []cli.Command{
{
Name: "dump",
Usage: "Show all GUI configuration settings",
Requires: &cli.Requires{},
Action: guiDump,
},
{
Name: "get",
Usage: "Get a GUI configuration setting",
Requires: &cli.Requires{"setting"},
Action: guiGet,
},
{
Name: "set",
Usage: "Set a GUI configuration setting",
Requires: &cli.Requires{"setting", "value"},
Action: guiSet,
},
{
Name: "unset",
Usage: "Unset a GUI configuration setting",
Requires: &cli.Requires{"setting"},
Action: guiUnset,
},
},
})
}
func guiDump(c *cli.Context) {
cfg := getConfig(c).GUI
writer := newTableWriter()
fmt.Fprintln(writer, "Enabled:\t", cfg.Enabled, "\t(enabled)")
fmt.Fprintln(writer, "Use HTTPS:\t", cfg.UseTLS(), "\t(tls)")
fmt.Fprintln(writer, "Listen Addresses:\t", cfg.Address(), "\t(address)")
if cfg.User != "" {
fmt.Fprintln(writer, "Authentication User:\t", cfg.User, "\t(username)")
fmt.Fprintln(writer, "Authentication Password:\t", cfg.Password, "\t(password)")
}
if cfg.APIKey != "" {
fmt.Fprintln(writer, "API Key:\t", cfg.APIKey, "\t(apikey)")
}
writer.Flush()
}
func guiGet(c *cli.Context) {
cfg := getConfig(c).GUI
arg := c.Args()[0]
switch strings.ToLower(arg) {
case "enabled":
fmt.Println(cfg.Enabled)
case "tls":
fmt.Println(cfg.UseTLS())
case "address":
fmt.Println(cfg.Address())
case "user":
if cfg.User != "" {
fmt.Println(cfg.User)
}
case "password":
if cfg.User != "" {
fmt.Println(cfg.Password)
}
case "apikey":
if cfg.APIKey != "" {
fmt.Println(cfg.APIKey)
}
default:
die("Invalid setting: " + arg + "\nAvailable settings: enabled, tls, address, user, password, apikey")
}
}
func guiSet(c *cli.Context) {
cfg := getConfig(c)
arg := c.Args()[0]
val := c.Args()[1]
switch strings.ToLower(arg) {
case "enabled":
cfg.GUI.Enabled = parseBool(val)
case "tls":
cfg.GUI.RawUseTLS = parseBool(val)
case "address":
validAddress(val)
cfg.GUI.RawAddress = val
case "user":
cfg.GUI.User = val
case "password":
cfg.GUI.Password = val
case "apikey":
cfg.GUI.APIKey = val
default:
die("Invalid setting: " + arg + "\nAvailable settings: enabled, tls, address, user, password, apikey")
}
setConfig(c, cfg)
}
func guiUnset(c *cli.Context) {
cfg := getConfig(c)
arg := c.Args()[0]
switch strings.ToLower(arg) {
case "user":
cfg.GUI.User = ""
case "password":
cfg.GUI.Password = ""
case "apikey":
cfg.GUI.APIKey = ""
default:
die("Invalid setting: " + arg + "\nAvailable settings: user, password, apikey")
}
setConfig(c, cfg)
}

View File

@@ -1,173 +0,0 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"fmt"
"strings"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "options",
HideHelp: true,
Usage: "Options command group",
Subcommands: []cli.Command{
{
Name: "dump",
Usage: "Show all Syncthing option settings",
Requires: &cli.Requires{},
Action: optionsDump,
},
{
Name: "get",
Usage: "Get a Syncthing option setting",
Requires: &cli.Requires{"setting"},
Action: optionsGet,
},
{
Name: "set",
Usage: "Set a Syncthing option setting",
Requires: &cli.Requires{"setting", "value..."},
Action: optionsSet,
},
},
})
}
func optionsDump(c *cli.Context) {
cfg := getConfig(c).Options
writer := newTableWriter()
fmt.Fprintln(writer, "Sync protocol listen addresses:\t", strings.Join(cfg.ListenAddresses, " "), "\t(addresses)")
fmt.Fprintln(writer, "Global discovery enabled:\t", cfg.GlobalAnnEnabled, "\t(globalannenabled)")
fmt.Fprintln(writer, "Global discovery servers:\t", strings.Join(cfg.GlobalAnnServers, " "), "\t(globalannserver)")
fmt.Fprintln(writer, "Local discovery enabled:\t", cfg.LocalAnnEnabled, "\t(localannenabled)")
fmt.Fprintln(writer, "Local discovery port:\t", cfg.LocalAnnPort, "\t(localannport)")
fmt.Fprintln(writer, "Outgoing rate limit in KiB/s:\t", cfg.MaxSendKbps, "\t(maxsend)")
fmt.Fprintln(writer, "Incoming rate limit in KiB/s:\t", cfg.MaxRecvKbps, "\t(maxrecv)")
fmt.Fprintln(writer, "Reconnect interval in seconds:\t", cfg.ReconnectIntervalS, "\t(reconnect)")
fmt.Fprintln(writer, "Start browser:\t", cfg.StartBrowser, "\t(browser)")
fmt.Fprintln(writer, "Enable UPnP:\t", cfg.NATEnabled, "\t(nat)")
fmt.Fprintln(writer, "UPnP Lease in minutes:\t", cfg.NATLeaseM, "\t(natlease)")
fmt.Fprintln(writer, "UPnP Renewal period in minutes:\t", cfg.NATRenewalM, "\t(natrenew)")
fmt.Fprintln(writer, "Restart on Wake Up:\t", cfg.RestartOnWakeup, "\t(wake)")
reporting := "unrecognized value"
switch cfg.URAccepted {
case -1:
reporting = "false"
case 0:
reporting = "undecided/false"
case 1:
reporting = "true"
}
fmt.Fprintln(writer, "Anonymous usage reporting:\t", reporting, "\t(reporting)")
writer.Flush()
}
func optionsGet(c *cli.Context) {
cfg := getConfig(c).Options
arg := c.Args()[0]
switch strings.ToLower(arg) {
case "address":
fmt.Println(strings.Join(cfg.ListenAddresses, "\n"))
case "globalannenabled":
fmt.Println(cfg.GlobalAnnEnabled)
case "globalannservers":
fmt.Println(strings.Join(cfg.GlobalAnnServers, "\n"))
case "localannenabled":
fmt.Println(cfg.LocalAnnEnabled)
case "localannport":
fmt.Println(cfg.LocalAnnPort)
case "maxsend":
fmt.Println(cfg.MaxSendKbps)
case "maxrecv":
fmt.Println(cfg.MaxRecvKbps)
case "reconnect":
fmt.Println(cfg.ReconnectIntervalS)
case "browser":
fmt.Println(cfg.StartBrowser)
case "nat":
fmt.Println(cfg.NATEnabled)
case "natlease":
fmt.Println(cfg.NATLeaseM)
case "natrenew":
fmt.Println(cfg.NATRenewalM)
case "reporting":
switch cfg.URAccepted {
case -1:
fmt.Println("false")
case 0:
fmt.Println("undecided/false")
case 1:
fmt.Println("true")
default:
fmt.Println("unknown")
}
case "wake":
fmt.Println(cfg.RestartOnWakeup)
default:
die("Invalid setting: " + arg + "\nAvailable settings: address, globalannenabled, globalannserver, localannenabled, localannport, maxsend, maxrecv, reconnect, browser, upnp, upnplease, upnprenew, reporting, wake")
}
}
func optionsSet(c *cli.Context) {
config := getConfig(c)
arg := c.Args()[0]
val := c.Args()[1]
switch strings.ToLower(arg) {
case "address":
for _, item := range c.Args().Tail() {
validAddress(item)
}
config.Options.ListenAddresses = c.Args().Tail()
case "globalannenabled":
config.Options.GlobalAnnEnabled = parseBool(val)
case "globalannserver":
for _, item := range c.Args().Tail() {
validAddress(item)
}
config.Options.GlobalAnnServers = c.Args().Tail()
case "localannenabled":
config.Options.LocalAnnEnabled = parseBool(val)
case "localannport":
config.Options.LocalAnnPort = parsePort(val)
case "maxsend":
config.Options.MaxSendKbps = parseUint(val)
case "maxrecv":
config.Options.MaxRecvKbps = parseUint(val)
case "reconnect":
config.Options.ReconnectIntervalS = parseUint(val)
case "browser":
config.Options.StartBrowser = parseBool(val)
case "nat":
config.Options.NATEnabled = parseBool(val)
case "natlease":
config.Options.NATLeaseM = parseUint(val)
case "natrenew":
config.Options.NATRenewalM = parseUint(val)
case "reporting":
switch strings.ToLower(val) {
case "u", "undecided", "unset":
config.Options.URAccepted = 0
default:
boolvalue := parseBool(val)
if boolvalue {
config.Options.URAccepted = 1
} else {
config.Options.URAccepted = -1
}
}
case "wake":
config.Options.RestartOnWakeup = parseBool(val)
default:
die("Invalid setting: " + arg + "\nAvailable settings: address, globalannenabled, globalannserver, localannenabled, localannport, maxsend, maxrecv, reconnect, browser, upnp, upnplease, upnprenew, reporting, wake")
}
setConfig(c, config)
}

View File

@@ -1,72 +0,0 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
import (
"encoding/json"
"fmt"
"github.com/AudriusButkevicius/cli"
)
func init() {
cliCommands = append(cliCommands, cli.Command{
Name: "report",
HideHelp: true,
Usage: "Reporting command group",
Subcommands: []cli.Command{
{
Name: "system",
Usage: "Report system state",
Requires: &cli.Requires{},
Action: reportSystem,
},
{
Name: "connections",
Usage: "Report about connections to other devices",
Requires: &cli.Requires{},
Action: reportConnections,
},
{
Name: "usage",
Usage: "Usage report",
Requires: &cli.Requires{},
Action: reportUsage,
},
},
})
}
func reportSystem(c *cli.Context) {
response := httpGet(c, "system/status")
data := make(map[string]interface{})
json.Unmarshal(responseToBArray(response), &data)
prettyPrintJSON(data)
}
func reportConnections(c *cli.Context) {
response := httpGet(c, "system/connections")
data := make(map[string]map[string]interface{})
json.Unmarshal(responseToBArray(response), &data)
var overall map[string]interface{}
for key, value := range data {
if key == "total" {
overall = value
continue
}
value["Device ID"] = key
prettyPrintJSON(value)
fmt.Println()
}
if overall != nil {
fmt.Println("=== Overall statistics ===")
prettyPrintJSON(overall)
}
}
func reportUsage(c *cli.Context) {
response := httpGet(c, "svc/report")
report := make(map[string]interface{})
json.Unmarshal(responseToBArray(response), &report)
prettyPrintJSON(report)
}

60
cmd/stcli/errors.go Normal file
View File

@@ -0,0 +1,60 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"fmt"
"strings"
"github.com/urfave/cli"
)
var errorsCommand = cli.Command{
Name: "errors",
HideHelp: true,
Usage: "Error command group",
Subcommands: []cli.Command{
{
Name: "show",
Usage: "Show pending errors",
Action: expects(0, dumpOutput("system/error")),
},
{
Name: "push",
Usage: "Push an error to active clients",
ArgsUsage: "[error message]",
Action: expects(1, errorsPush),
},
{
Name: "clear",
Usage: "Clear pending errors",
Action: expects(0, emptyPost("system/error/clear")),
},
},
}
func errorsPush(c *cli.Context) error {
client := c.App.Metadata["client"].(*APIClient)
errStr := strings.Join(c.Args(), " ")
response, err := client.Post("system/error", strings.TrimSpace(errStr))
if err != nil {
return err
}
if response.StatusCode != 200 {
errStr = fmt.Sprint("Failed to push error\nStatus code: ", response.StatusCode)
bytes, err := responseToBArray(response)
if err != nil {
return err
}
body := string(bytes)
if body != "" {
errStr += "\nBody: " + body
}
return fmt.Errorf(errStr)
}
return nil
}

View File

@@ -1,31 +0,0 @@
// Copyright (C) 2014 Audrius Butkevičius
package main
var jsonAttributeLabels = map[string]string{
"folderMaxMiB": "Largest folder size in MiB",
"folderMaxFiles": "Largest folder file count",
"longVersion": "Long version",
"totMiB": "Total size in MiB",
"totFiles": "Total files",
"uniqueID": "Unique ID",
"numFolders": "Folder count",
"numDevices": "Device count",
"memoryUsageMiB": "Memory usage in MiB",
"memorySize": "Total memory in MiB",
"sha256Perf": "SHA256 Benchmark",
"At": "Last contacted",
"Completion": "Percent complete",
"InBytesTotal": "Total bytes received",
"OutBytesTotal": "Total bytes sent",
"ClientVersion": "Client version",
"alloc": "Memory allocated in bytes",
"sys": "Memory using in bytes",
"cpuPercent": "CPU load in percent",
"extAnnounceOK": "External announcments working",
"goroutines": "Number of Go routines",
"myID": "Client ID",
"tilde": "Tilde expands to",
"arch": "Architecture",
"os": "OS",
}

View File

@@ -1,63 +1,192 @@
// Copyright (C) 2014 Audrius Butkevičius
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"sort"
"bufio"
"crypto/tls"
"encoding/json"
"flag"
"log"
"os"
"reflect"
"strings"
"github.com/AudriusButkevicius/cli"
"github.com/AudriusButkevicius/recli"
"github.com/flynn-archive/go-shlex"
"github.com/mattn/go-isatty"
"github.com/pkg/errors"
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/locations"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/urfave/cli"
)
type ByAlphabet []cli.Command
func (a ByAlphabet) Len() int { return len(a) }
func (a ByAlphabet) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (a ByAlphabet) Less(i, j int) bool { return a[i].Name < a[j].Name }
var cliCommands []cli.Command
func main() {
app := cli.NewApp()
app.Name = "syncthing-cli"
app.Author = "Audrius Butkevičius"
app.Email = "audrius.butkevicius@gmail.com"
app.Usage = "Syncthing command line interface"
app.Version = "0.1"
app.HideHelp = true
// This is somewhat a hack around a chicken and egg problem.
// We need to set the home directory and potentially other flags to know where the syncthing instance is running
// in order to get it's config ... which we then use to construct the actual CLI ... at which point it's too late
// to add flags there...
homeBaseDir := locations.GetBaseDir(locations.ConfigBaseDir)
guiCfg := config.GUIConfiguration{}
app.Flags = []cli.Flag{
flags := flag.NewFlagSet("", flag.ContinueOnError)
flags.StringVar(&guiCfg.RawAddress, "gui-address", guiCfg.RawAddress, "Override GUI address (e.g. \"http://192.0.2.42:8443\")")
flags.StringVar(&guiCfg.APIKey, "gui-apikey", guiCfg.APIKey, "Override GUI API key")
flags.StringVar(&homeBaseDir, "home", homeBaseDir, "Set configuration directory")
// Implement the same flags at the lower CLI, with the same default values (pre-parse), but do nothing with them.
// This is so that we could reuse os.Args
fakeFlags := []cli.Flag{
cli.StringFlag{
Name: "endpoint, e",
Value: "http://127.0.0.1:8384",
Usage: "End point to connect to",
EnvVar: "STENDPOINT",
Name: "gui-address",
Value: guiCfg.RawAddress,
Usage: "Override GUI address (e.g. \"http://192.0.2.42:8443\")",
},
cli.StringFlag{
Name: "apikey, k",
Value: "",
Usage: "API Key",
EnvVar: "STAPIKEY",
Name: "gui-apikey",
Value: guiCfg.APIKey,
Usage: "Override GUI API key",
},
cli.StringFlag{
Name: "username, u",
Value: "",
Usage: "Username",
EnvVar: "STUSERNAME",
},
cli.StringFlag{
Name: "password, p",
Value: "",
Usage: "Password",
EnvVar: "STPASSWORD",
},
cli.BoolFlag{
Name: "insecure, i",
Usage: "Do not verify SSL certificate",
EnvVar: "STINSECURE",
Name: "home",
Value: homeBaseDir,
Usage: "Set configuration directory",
},
}
sort.Sort(ByAlphabet(cliCommands))
app.Commands = cliCommands
app.RunAndExitOnError()
// Do not print usage of these flags, and ignore errors as this can't understand plenty of things
flags.Usage = func() {}
_ = flags.Parse(os.Args[1:])
// Now if the API key and address is not provided (we are not connecting to a remote instance),
// try to rip it out of the config.
if guiCfg.RawAddress == "" && guiCfg.APIKey == "" {
// Update the base directory
err := locations.SetBaseDir(locations.ConfigBaseDir, homeBaseDir)
if err != nil {
log.Fatal(errors.Wrap(err, "setting home"))
}
// Load the certs and get the ID
cert, err := tls.LoadX509KeyPair(
locations.Get(locations.CertFile),
locations.Get(locations.KeyFile),
)
if err != nil {
log.Fatal(errors.Wrap(err, "reading device ID"))
}
myID := protocol.NewDeviceID(cert.Certificate[0])
// Load the config
cfg, err := config.Load(locations.Get(locations.ConfigFile), myID)
if err != nil {
log.Fatalln(errors.Wrap(err, "loading config"))
}
guiCfg = cfg.GUI()
} else if guiCfg.Address() == "" || guiCfg.APIKey == "" {
log.Fatalln("Both -gui-address and -gui-apikey should be specified")
}
if guiCfg.Address() == "" {
log.Fatalln("Could not find GUI Address")
}
if guiCfg.APIKey == "" {
log.Fatalln("Could not find GUI API key")
}
client := getClient(guiCfg)
cfg, err := getConfig(client)
original := cfg.Copy()
if err != nil {
log.Fatalln(errors.Wrap(err, "getting config"))
}
// Copy the config and set the default flags
recliCfg := recli.DefaultConfig
recliCfg.IDTag.Name = "xml"
recliCfg.SkipTag.Name = "json"
commands, err := recli.New(recliCfg).Construct(&cfg)
if err != nil {
log.Fatalln(errors.Wrap(err, "config reflect"))
}
// Construct the actual CLI
app := cli.NewApp()
app.Name = "stcli"
app.HelpName = app.Name
app.Author = "The Syncthing Authors"
app.Usage = "Syncthing command line interface"
app.Version = strings.Replace(build.LongVersion, "syncthing", app.Name, 1)
app.Flags = fakeFlags
app.Metadata = map[string]interface{}{
"client": client,
}
app.Commands = []cli.Command{
{
Name: "config",
HideHelp: true,
Usage: "Configuration modification command group",
Subcommands: commands,
},
showCommand,
operationCommand,
errorsCommand,
}
tty := isatty.IsTerminal(os.Stdin.Fd()) || isatty.IsCygwinTerminal(os.Stdin.Fd())
if !tty {
// Not a TTY, consume from stdin
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
input, err := shlex.Split(scanner.Text())
if err != nil {
log.Fatalln(errors.Wrap(err, "parsing input"))
}
if len(input) == 0 {
continue
}
err = app.Run(append(os.Args, input...))
if err != nil {
log.Fatalln(err)
}
}
err = scanner.Err()
if err != nil {
log.Fatalln(err)
}
} else {
err = app.Run(os.Args)
if err != nil {
log.Fatalln(err)
}
}
if !reflect.DeepEqual(cfg, original) {
body, err := json.MarshalIndent(cfg, "", " ")
if err != nil {
log.Fatalln(err)
}
resp, err := client.Post("system/config", string(body))
if err != nil {
log.Fatalln(err)
}
if resp.StatusCode != 200 {
body, err := responseToBArray(resp)
if err != nil {
log.Fatalln(err)
}
log.Fatalln(string(body))
}
}
}

78
cmd/stcli/operations.go Normal file
View File

@@ -0,0 +1,78 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"fmt"
"github.com/urfave/cli"
)
var operationCommand = cli.Command{
Name: "operations",
HideHelp: true,
Usage: "Operation command group",
Subcommands: []cli.Command{
{
Name: "restart",
Usage: "Restart syncthing",
Action: expects(0, emptyPost("system/restart")),
},
{
Name: "shutdown",
Usage: "Shutdown syncthing",
Action: expects(0, emptyPost("system/shutdown")),
},
{
Name: "reset",
Usage: "Reset syncthing deleting all folders and devices",
Action: expects(0, emptyPost("system/reset")),
},
{
Name: "upgrade",
Usage: "Upgrade syncthing (if a newer version is available)",
Action: expects(0, emptyPost("system/upgrade")),
},
{
Name: "folder-override",
Usage: "Override changes on folder (remote for sendonly, local for receiveonly)",
ArgsUsage: "[folder id]",
Action: expects(1, foldersOverride),
},
},
}
func foldersOverride(c *cli.Context) error {
client := c.App.Metadata["client"].(*APIClient)
cfg, err := getConfig(client)
if err != nil {
return err
}
rid := c.Args()[0]
for _, folder := range cfg.Folders {
if folder.ID == rid {
response, err := client.Post("db/override", "")
if err != nil {
return err
}
if response.StatusCode != 200 {
errStr := fmt.Sprint("Failed to override changes\nStatus code: ", response.StatusCode)
bytes, err := responseToBArray(response)
if err != nil {
return err
}
body := string(bytes)
if body != "" {
errStr += "\nBody: " + body
}
return fmt.Errorf(errStr)
}
return nil
}
}
return fmt.Errorf("Folder " + rid + " not found")
}

44
cmd/stcli/show.go Normal file
View File

@@ -0,0 +1,44 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"github.com/urfave/cli"
)
var showCommand = cli.Command{
Name: "show",
HideHelp: true,
Usage: "Show command group",
Subcommands: []cli.Command{
{
Name: "version",
Usage: "Show syncthing client version",
Action: expects(0, dumpOutput("system/version")),
},
{
Name: "config-status",
Usage: "Show configuration status, whether or not a restart is required for changes to take effect",
Action: expects(0, dumpOutput("system/config/insync")),
},
{
Name: "system",
Usage: "Show system status",
Action: expects(0, dumpOutput("system/status")),
},
{
Name: "connections",
Usage: "Report about connections to other devices",
Action: expects(0, dumpOutput("system/connections")),
},
{
Name: "usage",
Usage: "Show usage report",
Action: expects(0, dumpOutput("svc/report")),
},
},
}

View File

@@ -1,4 +1,8 @@
// Copyright (C) 2014 Audrius Butkevičius
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
@@ -8,78 +12,37 @@ import (
"io/ioutil"
"net/http"
"os"
"regexp"
"sort"
"strconv"
"strings"
"text/tabwriter"
"unicode"
"github.com/AudriusButkevicius/cli"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/urfave/cli"
)
func responseToBArray(response *http.Response) []byte {
defer response.Body.Close()
func responseToBArray(response *http.Response) ([]byte, error) {
bytes, err := ioutil.ReadAll(response.Body)
if err != nil {
die(err)
return nil, err
}
return bytes
return bytes, response.Body.Close()
}
func die(vals ...interface{}) {
if len(vals) > 1 || vals[0] != nil {
os.Stderr.WriteString(fmt.Sprintln(vals...))
os.Exit(1)
func emptyPost(url string) cli.ActionFunc {
return func(c *cli.Context) error {
client := c.App.Metadata["client"].(*APIClient)
_, err := client.Post(url, "")
return err
}
}
func wrappedHTTPPost(url string) func(c *cli.Context) {
return func(c *cli.Context) {
httpPost(c, url, "")
}
}
func prettyPrintJSON(json map[string]interface{}) {
writer := newTableWriter()
remap := make(map[string]interface{})
for k, v := range json {
key, ok := jsonAttributeLabels[k]
if !ok {
key = firstUpper(k)
func dumpOutput(url string) cli.ActionFunc {
return func(c *cli.Context) error {
client := c.App.Metadata["client"].(*APIClient)
response, err := client.Get(url)
if err != nil {
return err
}
remap[key] = v
return prettyPrintResponse(c, response)
}
jsonKeys := make([]string, 0, len(remap))
for key := range remap {
jsonKeys = append(jsonKeys, key)
}
sort.Strings(jsonKeys)
for _, k := range jsonKeys {
var value string
rvalue := remap[k]
switch rvalue.(type) {
case int, int16, int32, int64, uint, uint16, uint32, uint64, float32, float64:
value = fmt.Sprintf("%.0f", rvalue)
default:
value = fmt.Sprint(rvalue)
}
if value == "" {
continue
}
fmt.Fprintln(writer, k+":\t"+value)
}
writer.Flush()
}
func firstUpper(str string) string {
for i, v := range str {
return string(unicode.ToUpper(v)) + str[i+1:]
}
return ""
}
func newTableWriter() *tabwriter.Writer {
@@ -88,78 +51,51 @@ func newTableWriter() *tabwriter.Writer {
return writer
}
func getMyID(c *cli.Context) string {
response := httpGet(c, "system/status")
data := make(map[string]interface{})
json.Unmarshal(responseToBArray(response), &data)
return data["myID"].(string)
}
func getConfig(c *cli.Context) config.Configuration {
response := httpGet(c, "system/config")
config := config.Configuration{}
json.Unmarshal(responseToBArray(response), &config)
return config
}
func setConfig(c *cli.Context, cfg config.Configuration) {
body, err := json.Marshal(cfg)
die(err)
response := httpPost(c, "system/config", string(body))
if response.StatusCode != 200 {
die("Unexpected status code", response.StatusCode)
}
}
func parseBool(input string) bool {
val, err := strconv.ParseBool(input)
func getConfig(c *APIClient) (config.Configuration, error) {
cfg := config.Configuration{}
response, err := c.Get("system/config")
if err != nil {
die(input + " is not a valid value for a boolean")
return cfg, err
}
return val
}
func parseInt(input string) int {
val, err := strconv.ParseInt(input, 0, 64)
bytes, err := responseToBArray(response)
if err != nil {
die(input + " is not a valid value for an integer")
return cfg, err
}
return int(val)
err = json.Unmarshal(bytes, &cfg)
if err == nil {
return cfg, err
}
return cfg, nil
}
func parseUint(input string) int {
val, err := strconv.ParseUint(input, 0, 64)
func expects(n int, actionFunc cli.ActionFunc) cli.ActionFunc {
return func(ctx *cli.Context) error {
if ctx.NArg() != n {
plural := ""
if n != 1 {
plural = "s"
}
return fmt.Errorf("expected %d argument%s, got %d", n, plural, ctx.NArg())
}
return actionFunc(ctx)
}
}
func prettyPrintJSON(data interface{}) error {
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
return enc.Encode(data)
}
func prettyPrintResponse(c *cli.Context, response *http.Response) error {
bytes, err := responseToBArray(response)
if err != nil {
die(input + " is not a valid value for an unsigned integer")
return err
}
return int(val)
}
func parsePort(input string) int {
port := parseUint(input)
if port < 1 || port > 65535 {
die(input + " is not a valid port\nExpected value between 1 and 65535")
}
return port
}
func validAddress(input string) {
tokens := strings.Split(input, ":")
if len(tokens) != 2 {
die(input + " is not a valid value for an address\nExpected format <ip or hostname>:<port>")
}
matched, err := regexp.MatchString("^[a-zA-Z0-9]+([-a-zA-Z0-9.]+[-a-zA-Z0-9]+)?$", tokens[0])
die(err)
if !matched {
die(input + " is not a valid value for an address\nExpected format <ip or hostname>:<port>")
}
parsePort(tokens[1])
}
func parseDeviceID(input string) protocol.DeviceID {
device, err := protocol.DeviceIDFromString(input)
if err != nil {
die(input + " is not a valid device id")
}
return device
var data interface{}
if err := json.Unmarshal(bytes, &data); err != nil {
return err
}
// TODO: Check flag for pretty print format
return prettyPrintJSON(data)
}

View File

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,204 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"errors"
"io/ioutil"
"regexp"
"strings"
"sync"
raven "github.com/getsentry/raven-go"
"github.com/maruel/panicparse/stack"
)
const reportServer = "https://crash.syncthing.net/report/"
var loader = newGithubSourceCodeLoader()
func init() {
raven.SetSourceCodeLoader(loader)
}
var (
clients = make(map[string]*raven.Client)
clientsMut sync.Mutex
)
func sendReport(dsn, path string, report []byte) error {
pkt, err := parseReport(path, report)
if err != nil {
return err
}
clientsMut.Lock()
defer clientsMut.Unlock()
cli, ok := clients[dsn]
if !ok {
cli, err = raven.New(dsn)
if err != nil {
return err
}
clients[dsn] = cli
}
// The client sets release and such on the packet before sending, in the
// misguided idea that it knows this better than than the packet we give
// it. So we copy the values from the packet to the client first...
cli.SetRelease(pkt.Release)
cli.SetEnvironment(pkt.Environment)
defer cli.Wait()
_, errC := cli.Capture(pkt, nil)
return <-errC
}
func parseReport(path string, report []byte) (*raven.Packet, error) {
parts := bytes.SplitN(report, []byte("\n"), 2)
if len(parts) != 2 {
return nil, errors.New("no first line")
}
version, err := parseVersion(string(parts[0]))
if err != nil {
return nil, err
}
report = parts[1]
foundPanic := false
var subjectLine []byte
for {
parts = bytes.SplitN(report, []byte("\n"), 2)
if len(parts) != 2 {
return nil, errors.New("no panic line found")
}
line := parts[0]
report = parts[1]
if foundPanic {
// The previous line was our "Panic at ..." header. We are now
// at the beginning of the real panic trace and this is our
// subject line.
subjectLine = line
break
} else if bytes.HasPrefix(line, []byte("Panic at")) {
foundPanic = true
}
}
r := bytes.NewReader(report)
ctx, err := stack.ParseDump(r, ioutil.Discard, false)
if err != nil {
return nil, err
}
// Lock the source code loader to the version we are processing here.
if version.commit != "" {
// We have a commit hash, so we know exactly which source to use
loader.LockWithVersion(version.commit)
} else if strings.HasPrefix(version.tag, "v") {
// Lets hope the tag is close enough
loader.LockWithVersion(version.tag)
} else {
// Last resort
loader.LockWithVersion("master")
}
defer loader.Unlock()
var trace raven.Stacktrace
for _, gr := range ctx.Goroutines {
if gr.First {
trace.Frames = make([]*raven.StacktraceFrame, len(gr.Stack.Calls))
for i, sc := range gr.Stack.Calls {
trace.Frames[len(trace.Frames)-1-i] = raven.NewStacktraceFrame(0, sc.Func.Name(), sc.SrcPath, sc.Line, 3, nil)
}
break
}
}
pkt := &raven.Packet{
Message: string(subjectLine),
Platform: "go",
Release: version.tag,
Environment: version.environment(),
Tags: raven.Tags{
raven.Tag{Key: "version", Value: version.version},
raven.Tag{Key: "tag", Value: version.tag},
raven.Tag{Key: "codename", Value: version.codename},
raven.Tag{Key: "runtime", Value: version.runtime},
raven.Tag{Key: "goos", Value: version.goos},
raven.Tag{Key: "goarch", Value: version.goarch},
raven.Tag{Key: "builder", Value: version.builder},
},
Extra: raven.Extra{
"url": reportServer + path,
},
Interfaces: []raven.Interface{&trace},
}
if version.commit != "" {
pkt.Tags = append(pkt.Tags, raven.Tag{Key: "commit", Value: version.commit})
}
return pkt, nil
}
// syncthing v1.1.4-rc.1+30-g6aaae618-dirty-crashrep "Erbium Earthworm" (go1.12.5 darwin-amd64) jb@kvin.kastelo.net 2019-05-23 16:08:14 UTC
var longVersionRE = regexp.MustCompile(`syncthing\s+(v[^\s]+)\s+"([^"]+)"\s\(([^\s]+)\s+([^-]+)-([^)]+)\)\s+([^\s]+)`)
type version struct {
version string // "v1.1.4-rc.1+30-g6aaae618-dirty-crashrep"
tag string // "v1.1.4-rc.1"
commit string // "6aaae618", blank when absent
codename string // "Erbium Earthworm"
runtime string // "go1.12.5"
goos string // "darwin"
goarch string // "amd64"
builder string // "jb@kvin.kastelo.net"
}
func (v version) environment() string {
if v.commit != "" {
return "Development"
}
if strings.Contains(v.tag, "-rc.") {
return "Candidate"
}
if strings.Contains(v.tag, "-") {
return "Beta"
}
return "Stable"
}
func parseVersion(line string) (version, error) {
m := longVersionRE.FindStringSubmatch(line)
if len(m) == 0 {
return version{}, errors.New("unintelligeble version string")
}
v := version{
version: m[1],
codename: m[2],
runtime: m[3],
goos: m[4],
goarch: m[5],
builder: m[6],
}
parts := strings.Split(v.version, "+")
v.tag = parts[0]
if len(parts) > 1 {
fields := strings.Split(parts[1], "-")
if len(fields) >= 2 && strings.HasPrefix(fields[1], "g") {
v.commit = fields[1][1:]
}
}
return v, nil
}

View File

@@ -0,0 +1,64 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"fmt"
"io/ioutil"
"testing"
)
func TestParseVersion(t *testing.T) {
cases := []struct {
longVersion string
parsed version
}{
{
longVersion: `syncthing v1.1.4-rc.1+30-g6aaae618-dirty-crashrep "Erbium Earthworm" (go1.12.5 darwin-amd64) jb@kvin.kastelo.net 2019-05-23 16:08:14 UTC`,
parsed: version{
version: "v1.1.4-rc.1+30-g6aaae618-dirty-crashrep",
tag: "v1.1.4-rc.1",
commit: "6aaae618",
codename: "Erbium Earthworm",
runtime: "go1.12.5",
goos: "darwin",
goarch: "amd64",
builder: "jb@kvin.kastelo.net",
},
},
}
for _, tc := range cases {
v, err := parseVersion(tc.longVersion)
if err != nil {
t.Error(err)
continue
}
if v != tc.parsed {
t.Error(v)
}
}
}
func TestParseReport(t *testing.T) {
bs, err := ioutil.ReadFile("_testdata/panic.log")
if err != nil {
t.Fatal(err)
}
pkt, err := parseReport("1/2/345", bs)
if err != nil {
t.Fatal(err)
}
bs, err = pkt.JSON()
if err != nil {
t.Fatal(err)
}
fmt.Printf("%s\n", bs)
}

View File

@@ -0,0 +1,114 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"fmt"
"io/ioutil"
"net/http"
"path/filepath"
"strings"
"sync"
"time"
)
const (
urlPrefix = "https://raw.githubusercontent.com/syncthing/syncthing/"
httpTimeout = 10 * time.Second
)
type githubSourceCodeLoader struct {
mut sync.Mutex
version string
cache map[string]map[string][][]byte // version -> file -> lines
client *http.Client
}
func newGithubSourceCodeLoader() *githubSourceCodeLoader {
return &githubSourceCodeLoader{
cache: make(map[string]map[string][][]byte),
client: &http.Client{Timeout: httpTimeout},
}
}
func (l *githubSourceCodeLoader) LockWithVersion(version string) {
l.mut.Lock()
l.version = version
if _, ok := l.cache[version]; !ok {
l.cache[version] = make(map[string][][]byte)
}
}
func (l *githubSourceCodeLoader) Unlock() {
l.mut.Unlock()
}
func (l *githubSourceCodeLoader) Load(filename string, line, context int) ([][]byte, int) {
filename = filepath.ToSlash(filename)
lines, ok := l.cache[l.version][filename]
if !ok {
// Cache whatever we managed to find (or nil if nothing, so we don't try again)
defer func() {
l.cache[l.version][filename] = lines
}()
knownPrefixes := []string{"/lib/", "/cmd/"}
var idx int
for _, pref := range knownPrefixes {
idx = strings.Index(filename, pref)
if idx >= 0 {
break
}
}
if idx == -1 {
return nil, 0
}
url := urlPrefix + l.version + filename[idx:]
resp, err := l.client.Get(url)
if err != nil || resp.StatusCode != http.StatusOK {
fmt.Println("Loading source:", err.Error())
return nil, 0
}
data, err := ioutil.ReadAll(resp.Body)
_ = resp.Body.Close()
if err != nil {
fmt.Println("Loading source:", err.Error())
return nil, 0
}
lines = bytes.Split(data, []byte{'\n'})
}
return getLineFromLines(lines, line, context)
}
func getLineFromLines(lines [][]byte, line, context int) ([][]byte, int) {
if lines == nil {
// cached error from ReadFile: return no lines
return nil, 0
}
line-- // stack trace lines are 1-indexed
start := line - context
var idx int
if start < 0 {
start = 0
idx = line
} else {
idx = context
}
end := line + context + 1
if line >= len(lines) {
return nil, 0
}
if end > len(lines) {
end = len(lines)
}
return lines[start:end], idx
}

View File

@@ -0,0 +1,160 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
// Command stcrashreceiver is a trivial HTTP server that allows two things:
//
// - uploading files (crash reports) named like a SHA256 hash using a PUT request
// - checking whether such file exists using a HEAD request
//
// Typically this should be deployed behind something that manages HTTPS.
package main
import (
"bytes"
"compress/gzip"
"flag"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"path"
"path/filepath"
"strings"
)
const maxRequestSize = 1 << 20 // 1 MiB
func main() {
dir := flag.String("dir", ".", "Directory to store reports in")
dsn := flag.String("dsn", "", "Sentry DSN")
listen := flag.String("listen", ":22039", "HTTP listen address")
flag.Parse()
cr := &crashReceiver{
dir: *dir,
dsn: *dsn,
}
log.SetOutput(os.Stdout)
if err := http.ListenAndServe(*listen, cr); err != nil {
log.Fatalln("HTTP serve:", err)
}
}
type crashReceiver struct {
dir string
dsn string
}
func (r *crashReceiver) ServeHTTP(w http.ResponseWriter, req *http.Request) {
// The final path component should be a SHA256 hash in hex, so 64 hex
// characters. We don't care about case on the request but use lower
// case internally.
reportID := strings.ToLower(path.Base(req.URL.Path))
if len(reportID) != 64 {
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
for _, c := range reportID {
if c >= 'a' && c <= 'f' {
continue
}
if c >= '0' && c <= '9' {
continue
}
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
// The location of the report on disk, compressed
fullPath := filepath.Join(r.dir, r.dirFor(reportID), reportID) + ".gz"
switch req.Method {
case http.MethodGet:
r.serveGet(fullPath, w, req)
case http.MethodHead:
r.serveHead(fullPath, w, req)
case http.MethodPut:
r.servePut(reportID, fullPath, w, req)
default:
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
}
}
// serveGet responds to GET requests by serving the uncompressed report.
func (r *crashReceiver) serveGet(fullPath string, w http.ResponseWriter, _ *http.Request) {
fd, err := os.Open(fullPath)
if err != nil {
http.Error(w, "Not found", http.StatusNotFound)
return
}
defer fd.Close()
gr, err := gzip.NewReader(fd)
if err != nil {
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
_, _ = io.Copy(w, gr) // best effort
}
// serveHead responds to HEAD requests by checking if the named report
// already exists in the system.
func (r *crashReceiver) serveHead(fullPath string, w http.ResponseWriter, _ *http.Request) {
if _, err := os.Lstat(fullPath); err != nil {
http.Error(w, "Not found", http.StatusNotFound)
}
}
// servePut accepts and stores the given report.
func (r *crashReceiver) servePut(reportID, fullPath string, w http.ResponseWriter, req *http.Request) {
// Ensure the destination directory exists
if err := os.MkdirAll(filepath.Dir(fullPath), 0755); err != nil {
log.Println("Creating directory:", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
// Read at most maxRequestSize of report data.
log.Println("Receiving report", reportID)
lr := io.LimitReader(req.Body, maxRequestSize)
bs, err := ioutil.ReadAll(lr)
if err != nil {
log.Println("Reading report:", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
// Compress the report for storage
buf := new(bytes.Buffer)
gw := gzip.NewWriter(buf)
_, _ = gw.Write(bs) // can't fail
gw.Close()
// Create an output file with the compressed report
err = ioutil.WriteFile(fullPath, buf.Bytes(), 0644)
if err != nil {
log.Println("Saving report:", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
// Send the report to Sentry
if r.dsn != "" {
go func() {
// There's no need for the client to have to wait for this part.
if err := sendReport(r.dsn, reportID, bs); err != nil {
log.Println("Failed to send report:", err)
}
}()
}
}
// 01234567890abcdef... => 01/23
func (r *crashReceiver) dirFor(base string) string {
return filepath.Join(base[0:2], base[2:4])
}

View File

@@ -8,6 +8,7 @@ package main
import (
"bytes"
"context"
"crypto/tls"
"encoding/json"
"encoding/pem"
@@ -23,7 +24,6 @@ import (
"time"
"github.com/syncthing/syncthing/lib/protocol"
"golang.org/x/net/context"
)
// announcement is the format received from and sent to clients
@@ -195,6 +195,7 @@ func (s *apiSrv) handleGET(ctx context.Context, w http.ResponseWriter, req *http
if misses%notFoundMissesWriteInterval == 0 {
rec.Misses = misses
rec.Missed = time.Now().UnixNano()
rec.Addresses = nil
// rec.Seen retained from get
s.db.put(key, rec)
@@ -334,7 +335,21 @@ func fixupAddresses(remote net.IP, addresses []string) []string {
}
ip := net.ParseIP(host)
// Some classes of IP are no-go.
if ip.IsLoopback() || ip.IsMulticast() {
continue
}
if host == "" || ip.IsUnspecified() {
// Replace the unspecified IP with the request source.
// ... unless the request source is the loopback address or
// multicast/unspecified (can't happen, really).
if remote.IsLoopback() || remote.IsMulticast() || remote.IsUnspecified() {
continue
}
// Do not use IPv6 remote address if requested scheme is ...4
// (i.e., tcp4, etc.)
if strings.HasSuffix(uri.Scheme, "4") && remote.To4() == nil {

View File

@@ -0,0 +1,65 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"fmt"
"net"
"testing"
)
func TestFixupAddresses(t *testing.T) {
cases := []struct {
remote net.IP
in []string
out []string
}{
{ // verbatim passthrough
in: []string{"tcp://1.2.3.4:22000"},
out: []string{"tcp://1.2.3.4:22000"},
}, { // unspecified replaced by remote
remote: net.ParseIP("1.2.3.4"),
in: []string{"tcp://:22000", "tcp://192.0.2.42:22000"},
out: []string{"tcp://1.2.3.4:22000", "tcp://192.0.2.42:22000"},
}, { // unspecified not used as replacement
remote: net.ParseIP("0.0.0.0"),
in: []string{"tcp://:22000", "tcp://192.0.2.42:22000"},
out: []string{"tcp://192.0.2.42:22000"},
}, { // unspecified not used as replacement
remote: net.ParseIP("::"),
in: []string{"tcp://:22000", "tcp://192.0.2.42:22000"},
out: []string{"tcp://192.0.2.42:22000"},
}, { // localhost not used as replacement
remote: net.ParseIP("127.0.0.1"),
in: []string{"tcp://:22000", "tcp://192.0.2.42:22000"},
out: []string{"tcp://192.0.2.42:22000"},
}, { // localhost not used as replacement
remote: net.ParseIP("::1"),
in: []string{"tcp://:22000", "tcp://192.0.2.42:22000"},
out: []string{"tcp://192.0.2.42:22000"},
}, { // multicast not used as replacement
remote: net.ParseIP("224.0.0.1"),
in: []string{"tcp://:22000", "tcp://192.0.2.42:22000"},
out: []string{"tcp://192.0.2.42:22000"},
}, { // multicast not used as replacement
remote: net.ParseIP("ff80::42"),
in: []string{"tcp://:22000", "tcp://192.0.2.42:22000"},
out: []string{"tcp://192.0.2.42:22000"},
}, { // explicitly announced weirdness is also filtered
remote: net.ParseIP("192.0.2.42"),
in: []string{"tcp://:22000", "tcp://127.1.2.3:22000", "tcp://[::1]:22000", "tcp://[ff80::42]:22000"},
out: []string{"tcp://192.0.2.42:22000"},
},
}
for _, tc := range cases {
out := fixupAddresses(tc.remote, tc.in)
if fmt.Sprint(out) != fmt.Sprint(tc.out) {
t.Errorf("fixupAddresses(%v, %v) => %v, expected %v", tc.remote, tc.in, out, tc.out)
}
}
}

View File

@@ -5,7 +5,7 @@
// You can obtain one at https://mozilla.org/MPL/2.0/.
//go:generate go run ../../script/protofmt.go database.proto
//go:generate protoc -I ../../../../../ -I ../../vendor/ -I ../../vendor/github.com/gogo/protobuf/protobuf -I . --gogofast_out=. database.proto
//go:generate protoc -I ../../ -I . --gogofast_out=. database.proto
package main
@@ -162,10 +162,10 @@ func (s *levelDBStore) Serve() {
// Start the statistics serve routine. It will exit with us when
// statisticsTrigger is closed.
statisticsTrigger := make(chan struct{})
defer close(statisticsTrigger)
statisticsDone := make(chan struct{})
go s.statisticsServe(statisticsTrigger, statisticsDone)
loop:
for {
select {
case fn := <-s.inbox:
@@ -184,17 +184,24 @@ func (s *levelDBStore) Serve() {
case <-s.stop:
// We're done.
return
close(statisticsTrigger)
break loop
}
}
// Also wait for statisticsServe to return
<-statisticsDone
}
func (s *levelDBStore) statisticsServe(trigger <-chan struct{}, done chan<- struct{}) {
defer close(done)
for range trigger {
t0 := time.Now()
nowNanos := t0.UnixNano()
cutoff24h := t0.Add(-24 * time.Hour).UnixNano()
cutoff1w := t0.Add(-7 * 24 * time.Hour).UnixNano()
cutoff2Mon := t0.Add(-60 * 24 * time.Hour).UnixNano()
current, last24h, last1w, inactive, errors := 0, 0, 0, 0, 0
iter := s.db.NewIterator(&util.Range{}, nil)
@@ -217,6 +224,17 @@ func (s *levelDBStore) statisticsServe(trigger <-chan struct{}, done chan<- stru
last24h++
case rec.Seen > cutoff1w:
last1w++
case rec.Seen > cutoff2Mon:
inactive++
case rec.Missed < cutoff2Mon:
// It hasn't been seen lately and we haven't recorded
// someone asking for this device in a long time either;
// delete the record.
if err := s.db.Delete(iter.Key(), nil); err != nil {
databaseOperations.WithLabelValues(dbOpDelete, dbResError).Inc()
} else {
databaseOperations.WithLabelValues(dbOpDelete, dbResSuccess).Inc()
}
default:
inactive++
}

View File

@@ -1,17 +1,6 @@
// Code generated by protoc-gen-gogo. DO NOT EDIT.
// source: database.proto
/*
Package main is a generated protocol buffer package.
It is generated from these files:
database.proto
It has these top-level messages:
DatabaseRecord
ReplicationRecord
DatabaseAddress
*/
package main
import proto "github.com/gogo/protobuf/proto"
@@ -33,36 +22,121 @@ var _ = math.Inf
const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package
type DatabaseRecord struct {
Addresses []DatabaseAddress `protobuf:"bytes,1,rep,name=addresses" json:"addresses"`
Addresses []DatabaseAddress `protobuf:"bytes,1,rep,name=addresses,proto3" json:"addresses"`
Misses int32 `protobuf:"varint,2,opt,name=misses,proto3" json:"misses,omitempty"`
Seen int64 `protobuf:"varint,3,opt,name=seen,proto3" json:"seen,omitempty"`
Missed int64 `protobuf:"varint,4,opt,name=missed,proto3" json:"missed,omitempty"`
}
func (m *DatabaseRecord) Reset() { *m = DatabaseRecord{} }
func (m *DatabaseRecord) String() string { return proto.CompactTextString(m) }
func (*DatabaseRecord) ProtoMessage() {}
func (*DatabaseRecord) Descriptor() ([]byte, []int) { return fileDescriptorDatabase, []int{0} }
func (m *DatabaseRecord) Reset() { *m = DatabaseRecord{} }
func (m *DatabaseRecord) String() string { return proto.CompactTextString(m) }
func (*DatabaseRecord) ProtoMessage() {}
func (*DatabaseRecord) Descriptor() ([]byte, []int) {
return fileDescriptor_database_0f49e029703a04f5, []int{0}
}
func (m *DatabaseRecord) XXX_Unmarshal(b []byte) error {
return m.Unmarshal(b)
}
func (m *DatabaseRecord) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
if deterministic {
return xxx_messageInfo_DatabaseRecord.Marshal(b, m, deterministic)
} else {
b = b[:cap(b)]
n, err := m.MarshalTo(b)
if err != nil {
return nil, err
}
return b[:n], nil
}
}
func (dst *DatabaseRecord) XXX_Merge(src proto.Message) {
xxx_messageInfo_DatabaseRecord.Merge(dst, src)
}
func (m *DatabaseRecord) XXX_Size() int {
return m.Size()
}
func (m *DatabaseRecord) XXX_DiscardUnknown() {
xxx_messageInfo_DatabaseRecord.DiscardUnknown(m)
}
var xxx_messageInfo_DatabaseRecord proto.InternalMessageInfo
type ReplicationRecord struct {
Key string `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
Addresses []DatabaseAddress `protobuf:"bytes,2,rep,name=addresses" json:"addresses"`
Addresses []DatabaseAddress `protobuf:"bytes,2,rep,name=addresses,proto3" json:"addresses"`
Seen int64 `protobuf:"varint,3,opt,name=seen,proto3" json:"seen,omitempty"`
}
func (m *ReplicationRecord) Reset() { *m = ReplicationRecord{} }
func (m *ReplicationRecord) String() string { return proto.CompactTextString(m) }
func (*ReplicationRecord) ProtoMessage() {}
func (*ReplicationRecord) Descriptor() ([]byte, []int) { return fileDescriptorDatabase, []int{1} }
func (m *ReplicationRecord) Reset() { *m = ReplicationRecord{} }
func (m *ReplicationRecord) String() string { return proto.CompactTextString(m) }
func (*ReplicationRecord) ProtoMessage() {}
func (*ReplicationRecord) Descriptor() ([]byte, []int) {
return fileDescriptor_database_0f49e029703a04f5, []int{1}
}
func (m *ReplicationRecord) XXX_Unmarshal(b []byte) error {
return m.Unmarshal(b)
}
func (m *ReplicationRecord) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
if deterministic {
return xxx_messageInfo_ReplicationRecord.Marshal(b, m, deterministic)
} else {
b = b[:cap(b)]
n, err := m.MarshalTo(b)
if err != nil {
return nil, err
}
return b[:n], nil
}
}
func (dst *ReplicationRecord) XXX_Merge(src proto.Message) {
xxx_messageInfo_ReplicationRecord.Merge(dst, src)
}
func (m *ReplicationRecord) XXX_Size() int {
return m.Size()
}
func (m *ReplicationRecord) XXX_DiscardUnknown() {
xxx_messageInfo_ReplicationRecord.DiscardUnknown(m)
}
var xxx_messageInfo_ReplicationRecord proto.InternalMessageInfo
type DatabaseAddress struct {
Address string `protobuf:"bytes,1,opt,name=address,proto3" json:"address,omitempty"`
Expires int64 `protobuf:"varint,2,opt,name=expires,proto3" json:"expires,omitempty"`
}
func (m *DatabaseAddress) Reset() { *m = DatabaseAddress{} }
func (m *DatabaseAddress) String() string { return proto.CompactTextString(m) }
func (*DatabaseAddress) ProtoMessage() {}
func (*DatabaseAddress) Descriptor() ([]byte, []int) { return fileDescriptorDatabase, []int{2} }
func (m *DatabaseAddress) Reset() { *m = DatabaseAddress{} }
func (m *DatabaseAddress) String() string { return proto.CompactTextString(m) }
func (*DatabaseAddress) ProtoMessage() {}
func (*DatabaseAddress) Descriptor() ([]byte, []int) {
return fileDescriptor_database_0f49e029703a04f5, []int{2}
}
func (m *DatabaseAddress) XXX_Unmarshal(b []byte) error {
return m.Unmarshal(b)
}
func (m *DatabaseAddress) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
if deterministic {
return xxx_messageInfo_DatabaseAddress.Marshal(b, m, deterministic)
} else {
b = b[:cap(b)]
n, err := m.MarshalTo(b)
if err != nil {
return nil, err
}
return b[:n], nil
}
}
func (dst *DatabaseAddress) XXX_Merge(src proto.Message) {
xxx_messageInfo_DatabaseAddress.Merge(dst, src)
}
func (m *DatabaseAddress) XXX_Size() int {
return m.Size()
}
func (m *DatabaseAddress) XXX_DiscardUnknown() {
xxx_messageInfo_DatabaseAddress.DiscardUnknown(m)
}
var xxx_messageInfo_DatabaseAddress proto.InternalMessageInfo
func init() {
proto.RegisterType((*DatabaseRecord)(nil), "main.DatabaseRecord")
@@ -106,6 +180,11 @@ func (m *DatabaseRecord) MarshalTo(dAtA []byte) (int, error) {
i++
i = encodeVarintDatabase(dAtA, i, uint64(m.Seen))
}
if m.Missed != 0 {
dAtA[i] = 0x20
i++
i = encodeVarintDatabase(dAtA, i, uint64(m.Missed))
}
return i, nil
}
@@ -179,24 +258,6 @@ func (m *DatabaseAddress) MarshalTo(dAtA []byte) (int, error) {
return i, nil
}
func encodeFixed64Database(dAtA []byte, offset int, v uint64) int {
dAtA[offset] = uint8(v)
dAtA[offset+1] = uint8(v >> 8)
dAtA[offset+2] = uint8(v >> 16)
dAtA[offset+3] = uint8(v >> 24)
dAtA[offset+4] = uint8(v >> 32)
dAtA[offset+5] = uint8(v >> 40)
dAtA[offset+6] = uint8(v >> 48)
dAtA[offset+7] = uint8(v >> 56)
return offset + 8
}
func encodeFixed32Database(dAtA []byte, offset int, v uint32) int {
dAtA[offset] = uint8(v)
dAtA[offset+1] = uint8(v >> 8)
dAtA[offset+2] = uint8(v >> 16)
dAtA[offset+3] = uint8(v >> 24)
return offset + 4
}
func encodeVarintDatabase(dAtA []byte, offset int, v uint64) int {
for v >= 1<<7 {
dAtA[offset] = uint8(v&0x7f | 0x80)
@@ -207,6 +268,9 @@ func encodeVarintDatabase(dAtA []byte, offset int, v uint64) int {
return offset + 1
}
func (m *DatabaseRecord) Size() (n int) {
if m == nil {
return 0
}
var l int
_ = l
if len(m.Addresses) > 0 {
@@ -221,10 +285,16 @@ func (m *DatabaseRecord) Size() (n int) {
if m.Seen != 0 {
n += 1 + sovDatabase(uint64(m.Seen))
}
if m.Missed != 0 {
n += 1 + sovDatabase(uint64(m.Missed))
}
return n
}
func (m *ReplicationRecord) Size() (n int) {
if m == nil {
return 0
}
var l int
_ = l
l = len(m.Key)
@@ -244,6 +314,9 @@ func (m *ReplicationRecord) Size() (n int) {
}
func (m *DatabaseAddress) Size() (n int) {
if m == nil {
return 0
}
var l int
_ = l
l = len(m.Address)
@@ -367,6 +440,25 @@ func (m *DatabaseRecord) Unmarshal(dAtA []byte) error {
break
}
}
case 4:
if wireType != 0 {
return fmt.Errorf("proto: wrong wireType = %d for field Missed", wireType)
}
m.Missed = 0
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
m.Missed |= (int64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
default:
iNdEx = preIndex
skippy, err := skipDatabase(dAtA[iNdEx:])
@@ -720,24 +812,25 @@ var (
ErrIntOverflowDatabase = fmt.Errorf("proto: integer overflow")
)
func init() { proto.RegisterFile("database.proto", fileDescriptorDatabase) }
func init() { proto.RegisterFile("database.proto", fileDescriptor_database_0f49e029703a04f5) }
var fileDescriptorDatabase = []byte{
// 254 bytes of a gzipped FileDescriptorProto
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x4b, 0x49, 0x2c, 0x49,
0x4c, 0x4a, 0x2c, 0x4e, 0xd5, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0xc9, 0x4d, 0xcc, 0xcc,
0x93, 0xd2, 0x4d, 0xcf, 0x2c, 0xc9, 0x28, 0x4d, 0xd2, 0x4b, 0xce, 0xcf, 0xd5, 0x4f, 0xcf, 0x4f,
0xcf, 0xd7, 0x07, 0x4b, 0x26, 0x95, 0xa6, 0x81, 0x79, 0x60, 0x0e, 0x98, 0x05, 0xd1, 0xa4, 0x54,
0xce, 0xc5, 0xe7, 0x02, 0x35, 0x26, 0x28, 0x35, 0x39, 0xbf, 0x28, 0x45, 0xc8, 0x92, 0x8b, 0x33,
0x31, 0x25, 0xa5, 0x28, 0xb5, 0xb8, 0x38, 0xb5, 0x58, 0x82, 0x51, 0x81, 0x59, 0x83, 0xdb, 0x48,
0x54, 0x0f, 0x64, 0xb4, 0x1e, 0x4c, 0xa1, 0x23, 0x44, 0xda, 0x89, 0xe5, 0xc4, 0x3d, 0x79, 0x86,
0x20, 0x84, 0x6a, 0x21, 0x31, 0x2e, 0xb6, 0xdc, 0x4c, 0xb0, 0x3e, 0x26, 0x05, 0x46, 0x0d, 0xd6,
0x20, 0x28, 0x4f, 0x48, 0x88, 0x8b, 0xa5, 0x38, 0x35, 0x35, 0x4f, 0x82, 0x59, 0x81, 0x51, 0x83,
0x39, 0x08, 0xcc, 0x56, 0x2a, 0xe1, 0x12, 0x0c, 0x4a, 0x2d, 0xc8, 0xc9, 0x4c, 0x4e, 0x2c, 0xc9,
0xcc, 0xcf, 0x83, 0xda, 0x2d, 0xc0, 0xc5, 0x9c, 0x9d, 0x5a, 0x29, 0xc1, 0xa8, 0xc0, 0xa8, 0xc1,
0x19, 0x04, 0x62, 0xa2, 0xba, 0x86, 0x89, 0x24, 0xd7, 0x60, 0xb3, 0xd5, 0x95, 0x8b, 0x1f, 0x4d,
0x9f, 0x90, 0x04, 0x17, 0x3b, 0x54, 0x0f, 0xd4, 0x5e, 0x18, 0x17, 0x24, 0x93, 0x5a, 0x51, 0x90,
0x59, 0x04, 0xf5, 0x0f, 0x73, 0x10, 0x8c, 0xeb, 0x24, 0x70, 0xe2, 0xa1, 0x1c, 0xc3, 0x89, 0x47,
0x72, 0x8c, 0x17, 0x1e, 0xc9, 0x31, 0x3e, 0x78, 0x24, 0xc7, 0x98, 0xc4, 0x06, 0x0e, 0x4e, 0x63,
0x40, 0x00, 0x00, 0x00, 0xff, 0xff, 0x9e, 0x45, 0x60, 0x7e, 0x95, 0x01, 0x00, 0x00,
var fileDescriptor_database_0f49e029703a04f5 = []byte{
// 270 bytes of a gzipped FileDescriptorProto
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x94, 0x90, 0x41, 0x4a, 0xc4, 0x30,
0x18, 0x85, 0x9b, 0x49, 0x1d, 0x99, 0x08, 0xa3, 0x06, 0x94, 0x20, 0x12, 0x4b, 0xdd, 0x74, 0xd5,
0x01, 0x5d, 0xb9, 0x74, 0xd0, 0x0b, 0xe4, 0x06, 0xe9, 0xe4, 0x77, 0x08, 0x3a, 0x4d, 0x49, 0x2a,
0xe8, 0x29, 0xf4, 0x58, 0x5d, 0xce, 0xd2, 0x95, 0x68, 0x7b, 0x11, 0x69, 0x26, 0x55, 0x14, 0x37,
0xb3, 0x7b, 0xdf, 0xff, 0xbf, 0x97, 0xbc, 0x84, 0x4c, 0x95, 0xac, 0x65, 0x21, 0x1d, 0xe4, 0x95,
0x35, 0xb5, 0xa1, 0xf1, 0x4a, 0xea, 0xf2, 0xe4, 0xdc, 0x42, 0x65, 0xdc, 0xcc, 0x8f, 0x8a, 0xc7,
0xbb, 0xd9, 0xd2, 0x2c, 0x8d, 0x07, 0xaf, 0x36, 0xd6, 0xf4, 0x05, 0x91, 0xe9, 0x4d, 0x48, 0x0b,
0x58, 0x18, 0xab, 0xe8, 0x15, 0x99, 0x48, 0xa5, 0x2c, 0x38, 0x07, 0x8e, 0xa1, 0x04, 0x67, 0x7b,
0x17, 0x47, 0x79, 0x7f, 0x62, 0x3e, 0x18, 0xaf, 0x37, 0xeb, 0x79, 0xdc, 0xbc, 0x9f, 0x45, 0xe2,
0xc7, 0x4d, 0x8f, 0xc9, 0x78, 0xa5, 0x7d, 0x6e, 0x94, 0xa0, 0x6c, 0x47, 0x04, 0xa2, 0x94, 0xc4,
0x0e, 0xa0, 0x64, 0x38, 0x41, 0x19, 0x16, 0x5e, 0x7f, 0x7b, 0x15, 0x8b, 0xfd, 0x34, 0x50, 0x5a,
0x93, 0x43, 0x01, 0xd5, 0x83, 0x5e, 0xc8, 0x5a, 0x9b, 0x32, 0x74, 0x3a, 0x20, 0xf8, 0x1e, 0x9e,
0x19, 0x4a, 0x50, 0x36, 0x11, 0xbd, 0xfc, 0xdd, 0x72, 0xb4, 0x55, 0xcb, 0x7f, 0xda, 0xa4, 0xb7,
0x64, 0xff, 0x4f, 0x8e, 0x32, 0xb2, 0x1b, 0x32, 0xe1, 0xde, 0x01, 0xfb, 0x0d, 0x3c, 0x55, 0xda,
0x86, 0x77, 0x62, 0x31, 0xe0, 0xfc, 0xb4, 0xf9, 0xe4, 0x51, 0xd3, 0x72, 0xb4, 0x6e, 0x39, 0xfa,
0x68, 0x39, 0x7a, 0xed, 0x78, 0xb4, 0xee, 0x78, 0xf4, 0xd6, 0xf1, 0xa8, 0x18, 0xfb, 0x3f, 0xbf,
0xfc, 0x0a, 0x00, 0x00, 0xff, 0xff, 0x7a, 0xa2, 0xf6, 0x1e, 0xb0, 0x01, 0x00, 0x00,
}

View File

@@ -8,16 +8,22 @@ syntax = "proto3";
package main;
import "github.com/gogo/protobuf/gogoproto/gogo.proto";
import "repos/protobuf/gogoproto/gogo.proto";
option (gogoproto.goproto_getters_all) = false;
option (gogoproto.goproto_unkeyed_all) = false;
option (gogoproto.goproto_unrecognized_all) = false;
option (gogoproto.goproto_sizecache_all) = false;
message DatabaseRecord {
repeated DatabaseAddress addresses = 1 [(gogoproto.nullable) = false];
int32 misses = 2; // Number of lookups without hits
int32 misses = 2; // Number of lookups* without hits
int64 seen = 3; // Unix nanos, last device announce
int64 missed = 4; // Unix nanos, last* failed lookup
}
// *) Not every lookup results in a write, so may not be completely accurate
message ReplicationRecord {
string key = 1;
repeated DatabaseAddress addresses = 2 [(gogoproto.nullable) = false];

View File

@@ -154,7 +154,7 @@ func TestDatabaseGetSet(t *testing.T) {
}
if len(rec.Addresses) != 1 {
t.Log(rec.Addresses)
t.Fatal("should have one addres")
t.Fatal("should have one address")
}
if rec.Misses != 0 {
t.Log(rec.Misses)

View File

@@ -0,0 +1,4 @@
[stdiscosrv]
title=Syncthing discovery server
description=Lets syncthing clients discover each other
ports=8443/tcp

View File

@@ -0,0 +1,3 @@
# Default settings for syncthing-relaysrv (strelaysrv).
## Add Options here:
DISCOSRV_OPTS=

View File

@@ -0,0 +1,25 @@
[Unit]
Description=Syncthing Discovery Server
After=network.target
Documentation=man:stdiscosrv(1)
[Service]
WorkingDirectory=/var/lib/syncthing-discosrv
EnvironmentFile=/etc/default/syncthing-discosrv
ExecStart=/usr/bin/stdiscosrv $DISCOSRV_OPTS
# Hardening
User=syncthing-discosrv
Group=syncthing
ProtectSystem=strict
ReadWritePaths=/var/lib/syncthing-discosrv
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
[Install]
WantedBy=multi-user.target
Alias=syncthing-discosrv.service

View File

@@ -88,11 +88,6 @@ var (
)
func main() {
const (
cleanIntv = 1 * time.Hour
statsIntv = 5 * time.Minute
)
var listen string
var dir string
var metricsListen string
@@ -121,7 +116,7 @@ func main() {
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
log.Println("Failed to load keypair. Generating one, this might take a while...")
cert, err = tlsutil.NewCertificate(certFile, keyFile, "stdiscosrv", 0)
cert, err = tlsutil.NewCertificate(certFile, keyFile, "stdiscosrv")
if err != nil {
log.Fatalln("Failed to generate X509 key pair:", err)
}
@@ -164,7 +159,9 @@ func main() {
}
// Root of the service tree.
main := suture.NewSimple("main")
main := suture.New("main", suture.Spec{
PassThroughPanics: true,
})
// Start the database.
db, err := newLevelDBStore(dir)

View File

@@ -18,6 +18,9 @@ import (
"github.com/syncthing/syncthing/lib/protocol"
)
const replicationReadTimeout = time.Minute
const replicationHeartbeatInterval = time.Second * 30
type replicator interface {
send(key string, addrs []DatabaseAddress, seen int64)
}
@@ -79,10 +82,22 @@ func (s *replicationSender) Serve() {
return
}
heartBeatTicker := time.NewTicker(replicationHeartbeatInterval)
defer heartBeatTicker.Stop()
// Send records.
buf := make([]byte, 1024)
for {
select {
case <-heartBeatTicker.C:
if len(s.outbox) > 0 {
// No need to send heartbeats if there are events/prevrious
// heartbeats to send, they will keep the connection alive.
continue
}
// Empty replication message is the heartbeat:
s.outbox <- ReplicationRecord{}
case rec := <-s.outbox:
// Buffer must hold record plus four bytes for size
size := rec.Size()
@@ -106,6 +121,7 @@ func (s *replicationSender) Serve() {
if _, err := conn.Write(buf[:4+n]); err != nil {
replicationSendsTotal.WithLabelValues("error").Inc()
log.Println("Replication write:", err)
// Yes, we are loosing the replication event here.
return
}
replicationSendsTotal.WithLabelValues("success").Inc()
@@ -149,7 +165,7 @@ func (m replicationMultiplexer) send(key string, ps []DatabaseAddress, seen int6
}
}
// replicationListener acceptes incoming connections and reads replication
// replicationListener accepts incoming connections and reads replication
// items from them. Incoming items are applied to the KV store.
type replicationListener struct {
addr string
@@ -242,7 +258,7 @@ func (l *replicationListener) handle(conn net.Conn) {
default:
}
conn.SetReadDeadline(time.Now().Add(time.Minute))
conn.SetReadDeadline(time.Now().Add(replicationReadTimeout))
// First four bytes are the size
if _, err := io.ReadFull(conn, buf[:4]); err != nil {
@@ -256,6 +272,12 @@ func (l *replicationListener) handle(conn net.Conn) {
if len(buf) < size {
buf = make([]byte, size)
}
if size == 0 {
// Heartbeat, ignore
continue
}
if _, err := io.ReadFull(conn, buf[:size]); err != nil {
log.Println("Replication read record:", err)
replicationRecvsTotal.WithLabelValues("error").Inc()

View File

@@ -0,0 +1,4 @@
#!/bin/bash
addgroup --system syncthing
adduser --system --home /var/lib/syncthing-discosrv --ingroup syncthing syncthing-discosrv

View File

@@ -7,6 +7,8 @@
package main
import (
"os"
"github.com/prometheus/client_golang/prometheus"
)
@@ -93,6 +95,7 @@ const (
dbOpGet = "get"
dbOpPut = "put"
dbOpMerge = "merge"
dbOpDelete = "delete"
dbResSuccess = "success"
dbResNotFound = "not_found"
dbResError = "error"
@@ -105,4 +108,16 @@ func init() {
replicationSendsTotal, replicationRecvsTotal,
databaseKeys, databaseStatisticsSeconds,
databaseOperations, databaseOperationSeconds)
processCollectorOpts := prometheus.ProcessCollectorOpts{
Namespace: "syncthing_discovery",
PidFn: func() (int, error) {
return os.Getpid(), nil
},
}
prometheus.MustRegister(
prometheus.NewProcessCollector(processCollectorOpts),
)
}

View File

@@ -68,8 +68,8 @@ func main() {
}
blockSize := int(fi.Size())
if *standardBlocks || blockSize < protocol.BlockSize {
blockSize = protocol.BlockSize
if *standardBlocks || blockSize < protocol.MinBlockSize {
blockSize = protocol.BlockSize(fi.Size())
}
bs, err := scanner.Blocks(context.TODO(), fd, blockSize, fi.Size(), nil, true)
if err != nil {

44
cmd/stfindignored/main.go Normal file
View File

@@ -0,0 +1,44 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
// Commmand stfindignored lists ignored files under a given folder root.
package main
import (
"flag"
"fmt"
"os"
"github.com/syncthing/syncthing/lib/fs"
"github.com/syncthing/syncthing/lib/ignore"
)
func main() {
flag.Parse()
root := flag.Arg(0)
if root == "" {
root = "."
}
vfs := fs.NewWalkFilesystem(fs.NewFilesystem(fs.FilesystemTypeBasic, root))
ign := ignore.New(vfs)
if err := ign.Load(".stignore"); err != nil {
fmt.Fprintf(os.Stderr, "Fatal: loading ignores: %v\n", err)
os.Exit(1)
}
vfs.Walk(".", func(path string, info fs.FileInfo, err error) error {
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: %s: %v\n", path, err)
return fs.SkipDir
}
if ign.Match(path).IsIgnored() {
fmt.Println(path)
}
return nil
})
}

View File

@@ -82,7 +82,7 @@ func generateOneFile(fd io.ReadSeeker, p1 string, s int64) error {
return err
}
_ = os.Chmod(p1, os.FileMode(rand.Intn(0777)|0400))
os.Chmod(p1, os.FileMode(rand.Intn(0777)|0400))
t := time.Now().Add(-time.Duration(rand.Intn(30*86400)) * time.Second)
return os.Chtimes(p1, t, t)

View File

@@ -16,7 +16,7 @@ import (
"github.com/syncthing/syncthing/lib/protocol"
)
func dump(ldb *db.Instance) {
func dump(ldb *db.Lowlevel) {
it := ldb.NewIterator(nil, nil)
for it.Next() {
key := it.Key()

View File

@@ -37,7 +37,7 @@ func (h *ElementHeap) Pop() interface{} {
return x
}
func dumpsize(ldb *db.Instance) {
func dumpsize(ldb *db.Lowlevel) {
h := &ElementHeap{}
heap.Init(h)

242
cmd/stindex/idxck.go Normal file
View File

@@ -0,0 +1,242 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"encoding/binary"
"fmt"
"github.com/syncthing/syncthing/lib/db"
"github.com/syncthing/syncthing/lib/protocol"
)
type fileInfoKey struct {
folder uint32
device uint32
name string
}
type globalKey struct {
folder uint32
name string
}
type sequenceKey struct {
folder uint32
sequence uint64
}
func idxck(ldb *db.Lowlevel) (success bool) {
folders := make(map[uint32]string)
devices := make(map[uint32]string)
deviceToIDs := make(map[string]uint32)
fileInfos := make(map[fileInfoKey]protocol.FileInfo)
globals := make(map[globalKey]db.VersionList)
sequences := make(map[sequenceKey]string)
needs := make(map[globalKey]struct{})
var localDeviceKey uint32
success = true
it := ldb.NewIterator(nil, nil)
for it.Next() {
key := it.Key()
switch key[0] {
case db.KeyTypeDevice:
folder := binary.BigEndian.Uint32(key[1:])
device := binary.BigEndian.Uint32(key[1+4:])
name := nulString(key[1+4+4:])
var f protocol.FileInfo
err := f.Unmarshal(it.Value())
if err != nil {
fmt.Println("Unable to unmarshal FileInfo:", err)
success = false
continue
}
fileInfos[fileInfoKey{folder, device, name}] = f
case db.KeyTypeGlobal:
folder := binary.BigEndian.Uint32(key[1:])
name := nulString(key[1+4:])
var flv db.VersionList
if err := flv.Unmarshal(it.Value()); err != nil {
fmt.Println("Unable to unmarshal VersionList:", err)
success = false
continue
}
globals[globalKey{folder, name}] = flv
case db.KeyTypeFolderIdx:
key := binary.BigEndian.Uint32(it.Key()[1:])
folders[key] = string(it.Value())
case db.KeyTypeDeviceIdx:
key := binary.BigEndian.Uint32(it.Key()[1:])
devices[key] = string(it.Value())
deviceToIDs[string(it.Value())] = key
if bytes.Equal(it.Value(), protocol.LocalDeviceID[:]) {
localDeviceKey = key
}
case db.KeyTypeSequence:
folder := binary.BigEndian.Uint32(key[1:])
seq := binary.BigEndian.Uint64(key[5:])
val := it.Value()
sequences[sequenceKey{folder, seq}] = string(val[9:])
case db.KeyTypeNeed:
folder := binary.BigEndian.Uint32(key[1:])
name := nulString(key[1+4:])
needs[globalKey{folder, name}] = struct{}{}
}
}
if localDeviceKey == 0 {
fmt.Println("Missing key for local device in device index (bailing out)")
success = false
return
}
for fk, fi := range fileInfos {
if fk.name != fi.Name {
fmt.Printf("Mismatching FileInfo name, %q (key) != %q (actual)\n", fk.name, fi.Name)
success = false
}
folder := folders[fk.folder]
if folder == "" {
fmt.Printf("Unknown folder ID %d for FileInfo %q\n", fk.folder, fk.name)
success = false
continue
}
if devices[fk.device] == "" {
fmt.Printf("Unknown device ID %d for FileInfo %q, folder %q\n", fk.folder, fk.name, folder)
success = false
}
if fk.device == localDeviceKey {
name, ok := sequences[sequenceKey{fk.folder, uint64(fi.Sequence)}]
if !ok {
fmt.Printf("Sequence entry missing for FileInfo %q, folder %q, seq %d\n", fi.Name, folder, fi.Sequence)
success = false
continue
}
if name != fi.Name {
fmt.Printf("Sequence entry refers to wrong name, %q (seq) != %q (FileInfo), folder %q, seq %d\n", name, fi.Name, folder, fi.Sequence)
success = false
}
}
}
for gk, vl := range globals {
folder := folders[gk.folder]
if folder == "" {
fmt.Printf("Unknown folder ID %d for VersionList %q\n", gk.folder, gk.name)
success = false
}
for i, fv := range vl.Versions {
dev, ok := deviceToIDs[string(fv.Device)]
if !ok {
fmt.Printf("VersionList %q, folder %q refers to unknown device %q\n", gk.name, folder, fv.Device)
success = false
}
fi, ok := fileInfos[fileInfoKey{gk.folder, dev, gk.name}]
if !ok {
fmt.Printf("VersionList %q, folder %q, entry %d refers to unknown FileInfo\n", gk.name, folder, i)
success = false
}
if !fi.Version.Equal(fv.Version) {
fmt.Printf("VersionList %q, folder %q, entry %d, FileInfo version mismatch, %v (VersionList) != %v (FileInfo)\n", gk.name, folder, i, fv.Version, fi.Version)
success = false
}
if fi.IsInvalid() != fv.Invalid {
fmt.Printf("VersionList %q, folder %q, entry %d, FileInfo invalid mismatch, %v (VersionList) != %v (FileInfo)\n", gk.name, folder, i, fv.Invalid, fi.IsInvalid())
success = false
}
}
// If we need this file we should have a need entry for it. False
// positives from needsLocally for deleted files, where we might
// legitimately lack an entry if we never had it, and ignored files.
if needsLocally(vl) {
_, ok := needs[gk]
if !ok {
dev := deviceToIDs[string(vl.Versions[0].Device)]
fi := fileInfos[fileInfoKey{gk.folder, dev, gk.name}]
if !fi.IsDeleted() && !fi.IsIgnored() {
fmt.Printf("Missing need entry for needed file %q, folder %q\n", gk.name, folder)
}
}
}
}
seenSeq := make(map[fileInfoKey]uint64)
for sk, name := range sequences {
folder := folders[sk.folder]
if folder == "" {
fmt.Printf("Unknown folder ID %d for sequence entry %d, %q\n", sk.folder, sk.sequence, name)
success = false
continue
}
if prev, ok := seenSeq[fileInfoKey{folder: sk.folder, name: name}]; ok {
fmt.Printf("Duplicate sequence entry for %q, folder %q, seq %d (prev %d)\n", name, folder, sk.sequence, prev)
success = false
}
seenSeq[fileInfoKey{folder: sk.folder, name: name}] = sk.sequence
fi, ok := fileInfos[fileInfoKey{sk.folder, localDeviceKey, name}]
if !ok {
fmt.Printf("Missing FileInfo for sequence entry %d, folder %q, %q\n", sk.sequence, folder, name)
success = false
continue
}
if fi.Sequence != int64(sk.sequence) {
fmt.Printf("Sequence mismatch for %q, folder %q, %d (key) != %d (FileInfo)\n", name, folder, sk.sequence, fi.Sequence)
success = false
}
}
for nk := range needs {
folder := folders[nk.folder]
if folder == "" {
fmt.Printf("Unknown folder ID %d for need entry %q\n", nk.folder, nk.name)
success = false
continue
}
vl, ok := globals[nk]
if !ok {
fmt.Printf("Missing global for need entry %q, folder %q\n", nk.name, folder)
success = false
continue
}
if !needsLocally(vl) {
fmt.Printf("Need entry for file we don't need, %q, folder %q\n", nk.name, folder)
success = false
}
}
return
}
func needsLocally(vl db.VersionList) bool {
var lv *protocol.Vector
for _, fv := range vl.Versions {
if bytes.Equal(fv.Device, protocol.LocalDeviceID[:]) {
lv = &fv.Version
break
}
}
if lv == nil {
return true // proviosinally, it looks like we need the file
}
return !lv.GreaterEqual(vl.Versions[0].Version)
}

View File

@@ -21,7 +21,7 @@ func main() {
log.SetFlags(0)
log.SetOutput(os.Stdout)
flag.StringVar(&mode, "mode", "dump", "Mode of operation: dump, dumpsize")
flag.StringVar(&mode, "mode", "dump", "Mode of operation: dump, dumpsize, idxck")
flag.Parse()
@@ -30,9 +30,7 @@ func main() {
path = filepath.Join(defaultConfigDir(), "index-v0.14.0.db")
}
fmt.Println("Path:", path)
ldb, err := db.Open(path)
ldb, err := db.OpenRO(path)
if err != nil {
log.Fatal(err)
}
@@ -41,6 +39,10 @@ func main() {
dump(ldb)
} else if mode == "dumpsize" {
dumpsize(ldb)
} else if mode == "idxck" {
if !idxck(ldb) {
os.Exit(1)
}
} else {
fmt.Println("Unknown mode")
}

View File

@@ -1 +1 @@
gui.go
gui.files.go

View File

@@ -1,13 +1,10 @@
// Copyright (C) 2017 The Syncthing Authors.
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
// +build !windows
//go:generate go run ../../../script/genassets.go -o gui.files.go ../gui
package fs
func (e fsFileInfo) Mode() FileMode {
return FileMode(e.FileInfo.Mode())
}
// Package auto contains auto generated files for web assets.
package auto

View File

@@ -10,7 +10,7 @@
<title>Relay stats</title>
<link href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" rel="stylesheet"/>
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css"/>
<link rel="stylesheet" href="//use.fontawesome.com/releases/v5.0.13/css/all.css"/>
<style>
#map {
@@ -56,83 +56,83 @@
<tr>
<th rowspan="2">Address</td>
<th rowspan="2">
<a ng-click="sortType = 'status.numActiveSessions'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.numActiveSessions'; sortReverse = !sortReverse">
Sessions
<span ng-show="sortType == 'status.numActiveSessions' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.numActiveSessions' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.numActiveSessions' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.numActiveSessions' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
<th rowspan="2">
<a ng-click="sortType = 'status.numConnections'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.numConnections'; sortReverse = !sortReverse">
Connections
<span ng-show="sortType == 'status.numConnections' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.numConnections' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.numConnections' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.numConnections' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
<th rowspan="2">
<a ng-click="sortType = 'status.bytesProxied'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.bytesProxied'; sortReverse = !sortReverse">
Data relayed
<span ng-show="sortType == 'status.bytesProxied' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.bytesProxied' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.bytesProxied' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.bytesProxied' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
<th colspan="6" class="text-center">Transfer rate in the last period</th>
<th rowspan="2">
<a ng-click="sortType = 'status.uptimeSeconds'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.uptimeSeconds'; sortReverse = !sortReverse">
Uptime hours
<span ng-show="sortType == 'status.uptimeSeconds' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.uptimeSeconds' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.uptimeSeconds' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'status.uptimeSeconds' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
<th rowspan="2">
<a ng-click="sortType = 'status.options[\'provided-by\'] || \'\''; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.options[\'provided-by\'] || \'\''; sortReverse = !sortReverse">
Provided by
<span ng-show="sortType == 'status.options[\'provided-by\'] || \'\'' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.options[\'provided-by\'] || \'\'' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.options[\'provided-by\'] || \'\'' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.options[\'provided-by\'] || \'\'' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
</tr>
<tr>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[0]'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.kbps10s1m5m15m30m60m[0]'; sortReverse = !sortReverse">
10s
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[0]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[0]' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[0]' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[0]' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[1]'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.kbps10s1m5m15m30m60m[1]'; sortReverse = !sortReverse">
1m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[1]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[1]' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[1]' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[1]' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[2]'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.kbps10s1m5m15m30m60m[2]'; sortReverse = !sortReverse">
5m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[2]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[2]' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[2]' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[2]' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[3]'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.kbps10s1m5m15m30m60m[3]'; sortReverse = !sortReverse">
15m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[3]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[3]' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[3]' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[3]' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[4]'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.kbps10s1m5m15m30m60m[4]'; sortReverse = !sortReverse">
30m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[4]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[4]' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[4]' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[4]' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
<th>
<a ng-click="sortType = 'status.kbps10s1m5m15m30m60m[5]'; sortReverse = !sortReverse">
<a ng-click="sortType = 'stats.kbps10s1m5m15m30m60m[5]'; sortReverse = !sortReverse">
60m
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[5]' && !sortReverse" class="fa fa-caret-down"></span>
<span ng-show="sortType == 'status.kbps10s1m5m15m30m60m[5]' && sortReverse" class="fa fa-caret-up"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[5]' && !sortReverse" class="fas fa-caret-down"></span>
<span ng-show="sortType == 'stats.kbps10s1m5m15m30m60m[5]' && sortReverse" class="fas fa-caret-up"></span>
</a>
</th>
</tr>
@@ -140,21 +140,21 @@
<tbody>
<tr ng-repeat="relay in relays | orderBy:sortType:sortReverse:sortCompare" ng-mouseover="relay.showMarker()" ng-mouseleave="relay.hideMarker()">
<td>{{ relay.address }}</td>
<td ng-if="relay.status === undefined" colspan="11" class="text-center">Looking up...</td>
<td ng-if-start="relay.status !== undefined">{{ relay.status.numActiveSessions }}</td>
<td>{{ relay.status.numConnections }}</td>
<td>{{ relay.status.bytesProxied | bytes }}</td>
<td>{{ relay.status.kbps10s1m5m15m30m60m[0] * 128 | bytes }}/s</td>
<td>{{ relay.status.kbps10s1m5m15m30m60m[1] * 128 | bytes }}/s</td>
<td>{{ relay.status.kbps10s1m5m15m30m60m[2] * 128 | bytes }}/s</td>
<td>{{ relay.status.kbps10s1m5m15m30m60m[3] * 128 | bytes }}/s</td>
<td>{{ relay.status.kbps10s1m5m15m30m60m[4] * 128 | bytes }}/s</td>
<td>{{ relay.status.kbps10s1m5m15m30m60m[5] * 128 | bytes }}/s</td>
<td ng-if="relay.status.uptimeSeconds != undefined">{{ relay.status.uptimeSeconds/60/60 | number:0 }}</td>
<td ng-if="relay.status.uptimeSeconds == undefined"></td>
<td title="{{ relay.status.options['provided-by'] || '' }}" ng-if-end>
{{ relay.status.options['provided-by'] || '' | limitTo:50 }}
<span ng-if="(relay.status.options['provided-by'] || '').length > 50">&hellip;
<td ng-if="!relay.stats" colspan="11"></td>
<td ng-if-start="relay.stats">{{ relay.stats.numActiveSessions }}</td>
<td>{{ relay.stats.numConnections }}</td>
<td>{{ relay.stats.bytesProxied | bytes }}</td>
<td>{{ relay.stats.kbps10s1m5m15m30m60m[0] * 128 | bytes }}/s</td>
<td>{{ relay.stats.kbps10s1m5m15m30m60m[1] * 128 | bytes }}/s</td>
<td>{{ relay.stats.kbps10s1m5m15m30m60m[2] * 128 | bytes }}/s</td>
<td>{{ relay.stats.kbps10s1m5m15m30m60m[3] * 128 | bytes }}/s</td>
<td>{{ relay.stats.kbps10s1m5m15m30m60m[4] * 128 | bytes }}/s</td>
<td>{{ relay.stats.kbps10s1m5m15m30m60m[5] * 128 | bytes }}/s</td>
<td ng-if="relay.stats.uptimeSeconds != undefined">{{ relay.stats.uptimeSeconds/60/60 | number:0 }}</td>
<td ng-if="relay.stats.uptimeSeconds == undefined"></td>
<td title="{{ relay.stats.options['provided-by'] || '' }}" ng-if-end>
{{ relay.stats.options['provided-by'] || '' | limitTo:50 }}
<span ng-if="(relay.stats.options['provided-by'] || '').length > 50">&hellip;
</td>
</tr>
</tbody>
@@ -187,15 +187,15 @@
<script type="text/javascript" src="//code.jquery.com/jquery-2.1.4.min.js"></script>
<script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/angular.js/1.5.8/angular.min.js"></script>
<script type="text/javascript" src="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
<script type="text/javascript" src="//maps.googleapis.com/maps/api/js"></script>
<script type="text/javascript" src="//maps.googleapis.com/maps/api/js?key=AIzaSyDk5WJ8s7ueLKb99X5DbQ-vkWtPDAKqYs0"></script>
</body>
<script>
angular.module('syncthing', [
])
.config(function($httpProvider) {
.config(['$httpProvider', function($httpProvider) {
$httpProvider.defaults.timeout = 5000;
})
}])
.filter('bytes', function() {
return function(bytes, precision) {
if (isNaN(parseFloat(bytes)) || !isFinite(bytes)) return '-';
@@ -235,16 +235,16 @@
$scope.mapBounds = new google.maps.LatLngBounds();
$scope.tooltipTemplate = $('#infoTemplate').html();
$scope.usedLocations = {};
$scope.sortType = 'status.numActiveSessions';
$scope.sortType = 'stats.numActiveSessions';
$scope.sortReverse = true;
$scope.sortCompare = function(a, b) {
if (a.value == b.value) {
return 0;
}
if (a.type == "undefined") {
if (a.type == "undefined" || a.type == "null") {
return -1;
}
if (b.type == "undefined") {
if (b.type == "undefined" || b.type == "null") {
return 1;
}
return a.value > b.value ? 1 : -1;
@@ -252,25 +252,31 @@
$http.get("/endpoint").then(function(response) {
$scope.relays = response.data.relays;
var promises = [];
angular.forEach($scope.relays, function(relay) {
angular.forEach($scope.relays, function(relay) {
relay.uri = constructURI(relay.url);
relay.address = relay.url.split('/')[2];
addMarkerToMap(relay);
promises.push(getRelayStatus(relay));
if (relay.stats) {
angular.forEach($scope.totals, function(value, key) {
if (typeof $scope.totals[key] == 'number') {
$scope.totals[key] += relay.stats[key];
} else if (typeof $scope.totals[key] == 'object' && $scope.totals[key] instanceof Array) {
angular.forEach($scope.totals[key], function(value, index) {
$scope.totals[key][index] += relay.stats[key][index];
});
}
});
}
});
// Can only add circles once we know the totals for transfers, which means
// we need to resolve all statuses.
$q.all(promises).then(function() {
angular.forEach($scope.relays, function(relay) {
if (relay.status) {
addCircleToMap(relay);
}
});
// After the totals were calculated, add circles.
angular.forEach($scope.relays, function(relay) {
if (relay.stats) {
addCircleToMap(relay);
}
});
$scope.map.fitBounds($scope.mapBounds);
@@ -330,41 +336,10 @@
fillOpacity: 0.35,
map: $scope.map,
center: relay.marker.position,
radius: ((relay.status.bytesProxied * 100) / $scope.totals.bytesProxied) * 10000
radius: ((relay.stats.bytesProxied * 100) / $scope.totals.bytesProxied) * 10000
});
}
function getRelayStatus(relay) {
// Normal timeout doesn't deal with relays which accept the TCP connection
// but don't respond (some firewalls do that), so deal with it this way.
var timeoutRequest = $q.defer();
var resolveStatus = $q.defer();
$http.get("http://" + relay.uri.hostname + ':' + ((relay.uri.args.statusAddr && relay.uri.args.statusAddr.split(':')[1]) || "22070") + "/status", { timeout: timeoutRequest.promise }).then(function (response) {
relay.status = response.data;
resolveStatus.resolve();
angular.forEach($scope.totals, function(value, key) {
if (typeof $scope.totals[key] == 'number') {
$scope.totals[key] += response.data[key];
} else if (typeof $scope.totals[key] == 'object' && $scope.totals[key] instanceof Array) {
angular.forEach($scope.totals[key], function(value, index) {
$scope.totals[key][index] += response.data[key][index];
});
}
});
}, function() {
relay.status = null;
resolveStatus.resolve();
});
$timeout(function() {
timeoutRequest.resolve();
}, 5000);
return resolveStatus.promise;
}
function constructURI(url) {
var uri = document.createElement('a');
@@ -385,25 +360,25 @@
<script type="text/template" id="infoTemplate">
<div>
<p><b>{{ relay.uri.hostname }}</b> <span ng-if="relay.status.options['provided-by']">provided by <u>{{ relay.status.options['provided-by'] }}</u></span></p>
<div ng-if="relay.status">
<span ng-if="relay.status.startTime">Start time: {{ relay.status.startTime | date:"medium" }}</br></span>
<span ng-if="relay.status.bytesProxied != undefined">Proxied: {{ relay.status.bytesProxied | bytes }}</br></span>
<span ng-if="relay.status.numActiveSessions != undefined">Sessions: {{ relay.status.numActiveSessions }}</br></span>
<span ng-if="relay.status.numConnections != undefined">Clients: {{ relay.status.numConnections }}</br></span>
<span ng-if="relay.status.options.pools">Pools: {{ relay.status.options.pools.join(', ') }}</br></span>
<span ng-if="relay.status.options['global-rate'] != undefined">
<span ng-if="relay.status.options['global-rate'] > 0">Global rate limit: {{ relay.status.options['global-rate'] | bytes }}/s</span>
<span ng-if="relay.status.options['global-rate'] == 0">Global rate limit: unlimited</span>
<p><b>{{ relay.uri.hostname }}</b> <span ng-if="relay.stats.options['provided-by']">provided by <u>{{ relay.stats.options['provided-by'] }}</u></span></p>
<div ng-if="relay.stats">
<span ng-if="relay.stats.startTime">Start time: {{ relay.stats.startTime | date:"medium" }}</br></span>
<span ng-if="relay.stats.bytesProxied != undefined">Proxied: {{ relay.stats.bytesProxied | bytes }}</br></span>
<span ng-if="relay.stats.numActiveSessions != undefined">Sessions: {{ relay.stats.numActiveSessions }}</br></span>
<span ng-if="relay.stats.numConnections != undefined">Clients: {{ relay.stats.numConnections }}</br></span>
<span ng-if="relay.stats.options.pools">Pools: {{ relay.stats.options.pools.join(', ') }}</br></span>
<span ng-if="relay.stats.options['global-rate'] != undefined">
<span ng-if="relay.stats.options['global-rate'] > 0">Global rate limit: {{ relay.stats.options['global-rate'] | bytes }}/s</span>
<span ng-if="relay.stats.options['global-rate'] == 0">Global rate limit: unlimited</span>
<br/>
</span>
<span ng-if="relay.status.options['per-session-rate'] != undefined">
<span ng-if="relay.status.options['per-session-rate'] > 0">Session rate limit: {{ relay.status.options['per-session-rate'] | bytes }}/s</span>
<span ng-if="relay.status.options['per-session-rate'] == 0">Session rate limit: unlimited</span>
<span ng-if="relay.stats.options['per-session-rate'] != undefined">
<span ng-if="relay.stats.options['per-session-rate'] > 0">Session rate limit: {{ relay.stats.options['per-session-rate'] | bytes }}/s</span>
<span ng-if="relay.stats.options['per-session-rate'] == 0">Session rate limit: unlimited</span>
<br/>
</span>
</div>
<div ng-if="!relay.status">
<div ng-if="!relay.stats">
Data unavailable.
<div>
</div>

View File

@@ -13,18 +13,22 @@ import (
"fmt"
"io/ioutil"
"log"
"math/rand"
"mime"
"net"
"net/http"
"net/url"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/golang/groupcache/lru"
"github.com/oschwald/geoip2-golang"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/syncthing/syncthing/cmd/strelaypoolsrv/auto"
"github.com/syncthing/syncthing/lib/rand"
"github.com/syncthing/syncthing/lib/relay/client"
"github.com/syncthing/syncthing/lib/sync"
"github.com/syncthing/syncthing/lib/tlsutil"
@@ -34,12 +38,42 @@ import (
type location struct {
Latitude float64 `json:"latitude"`
Longitude float64 `json:"longitude"`
City string `json:"city"`
Country string `json:"country"`
Continent string `json:"continent"`
}
type relay struct {
URL string `json:"url"`
Location location `json:"location"`
uri *url.URL
URL string `json:"url"`
Location location `json:"location"`
uri *url.URL
Stats *stats `json:"stats"`
StatsRetrieved time.Time `json:"statsRetrieved"`
}
type stats struct {
StartTime time.Time `json:"startTime"`
UptimeSeconds int `json:"uptimeSeconds"`
PendingSessionKeys int `json:"numPendingSessionKeys"`
ActiveSessions int `json:"numActiveSessions"`
Connections int `json:"numConnections"`
Proxies int `json:"numProxies"`
BytesProxied int `json:"bytesProxied"`
GoVersion string `json:"goVersion"`
GoOS string `json:"goOS"`
GoArch string `json:"goArch"`
GoMaxProcs int `json:"goMaxProcs"`
GoRoutines int `json:"goNumRoutine"`
Rates []int64 `json:"kbps10s1m5m15m30m60m"`
Options struct {
NetworkTimeout int `json:"network-timeout"`
PintInterval int `json:"ping-interval"`
MessageTimeout int `json:"message-timeout"`
SessionRate int `json:"per-session-rate"`
GlobalRate int `json:"global-rate"`
Pools []string `json:"pools"`
ProvidedBy string `json:"provided-by"`
} `json:"options"`
}
func (r relay) String() string {
@@ -47,9 +81,9 @@ func (r relay) String() string {
}
type request struct {
relay relay
uri *url.URL
result chan result
relay *relay
result chan result
queueTimer *prometheus.Timer
}
type result struct {
@@ -58,23 +92,25 @@ type result struct {
}
var (
testCert tls.Certificate
listen = ":80"
dir string
evictionTime = time.Hour
debug bool
getLRUSize = 10 << 10
getLimitBurst = 10
getLimitAvg = 1
postLRUSize = 1 << 10
postLimitBurst = 2
postLimitAvg = 1
getLimit time.Duration
postLimit time.Duration
permRelaysFile string
ipHeader string
geoipPath string
proto string
testCert tls.Certificate
knownRelaysFile = filepath.Join(os.TempDir(), "strelaypoolsrv_known_relays")
listen = ":80"
dir string
evictionTime = time.Hour
debug bool
getLRUSize = 10 << 10
getLimitBurst = 10
getLimitAvg = 2
postLRUSize = 1 << 10
postLimitBurst = 2
postLimitAvg = 2
getLimit time.Duration
postLimit time.Duration
permRelaysFile string
ipHeader string
geoipPath string
proto string
statsRefresh = time.Minute / 2
getMut = sync.NewRWMutex()
getLRUCache *lru.Cache
@@ -85,26 +121,31 @@ var (
requests = make(chan request, 10)
mut = sync.NewRWMutex()
knownRelays = make([]relay, 0)
permanentRelays = make([]relay, 0)
knownRelays = make([]*relay, 0)
permanentRelays = make([]*relay, 0)
evictionTimers = make(map[string]*time.Timer)
)
const (
httpStatusEnhanceYourCalm = 429
)
func main() {
flag.StringVar(&listen, "listen", listen, "Listen address")
flag.StringVar(&dir, "keys", dir, "Directory where http-cert.pem and http-key.pem is stored for TLS listening")
flag.BoolVar(&debug, "debug", debug, "Enable debug output")
flag.DurationVar(&evictionTime, "eviction", evictionTime, "After how long the relay is evicted")
flag.IntVar(&getLRUSize, "get-limit-cache", getLRUSize, "Get request limiter cache size")
flag.IntVar(&getLimitAvg, "get-limit-avg", 2, "Allowed average get request rate, per 10 s")
flag.IntVar(&getLimitAvg, "get-limit-avg", getLimitAvg, "Allowed average get request rate, per 10 s")
flag.IntVar(&getLimitBurst, "get-limit-burst", getLimitBurst, "Allowed burst get requests")
flag.IntVar(&postLRUSize, "post-limit-cache", postLRUSize, "Post request limiter cache size")
flag.IntVar(&postLimitAvg, "post-limit-avg", 2, "Allowed average post request rate, per minute")
flag.IntVar(&postLimitAvg, "post-limit-avg", postLimitAvg, "Allowed average post request rate, per minute")
flag.IntVar(&postLimitBurst, "post-limit-burst", postLimitBurst, "Allowed burst post requests")
flag.StringVar(&permRelaysFile, "perm-relays", "", "Path to list of permanent relays")
flag.StringVar(&ipHeader, "ip-header", "", "Name of header which holds clients ip:port. Only meaningful when running behind a reverse proxy.")
flag.StringVar(&geoipPath, "geoip", "GeoLite2-City.mmdb", "Path to GeoLite2-City database")
flag.StringVar(&proto, "protocol", "tcp", "Protocol used for listening. 'tcp' for IPv4 and IPv6, 'tcp4' for IPv4, 'tcp6' for IPv6")
flag.DurationVar(&statsRefresh, "stats-refresh", statsRefresh, "Interval at which to refresh relay stats")
flag.Parse()
@@ -118,13 +159,31 @@ func main() {
var err error
if permRelaysFile != "" {
loadPermanentRelays(permRelaysFile)
permanentRelays = loadRelays(permRelaysFile)
}
testCert = createTestCertificate()
go requestProcessor()
// Load relays from cache in the background.
// Load them in a serial fashion to make sure any genuine requests
// are not dropped.
go func() {
for _, relay := range loadRelays(knownRelaysFile) {
resultChan := make(chan result)
requests <- request{relay, resultChan, nil}
result := <-resultChan
if result.err != nil {
relayTestsTotal.WithLabelValues("failed").Inc()
} else {
relayTestsTotal.WithLabelValues("success").Inc()
}
}
// Run the the stats refresher once the relays are loaded.
statsRefresher(statsRefresh)
}()
if dir != "" {
if debug {
log.Println("Starting TLS listener on", listen)
@@ -169,6 +228,7 @@ func main() {
handler := http.NewServeMux()
handler.HandleFunc("/", handleAssets)
handler.HandleFunc("/endpoint", handleRequest)
handler.HandleFunc("/metrics", handleMetrics)
srv := http.Server{
Handler: handler,
@@ -181,7 +241,18 @@ func main() {
}
}
func handleMetrics(w http.ResponseWriter, r *http.Request) {
timer := prometheus.NewTimer(metricsRequestsSeconds)
// Acquire the mutex just to make sure we're not caught mid-way stats collection
mut.RLock()
promhttp.Handler().ServeHTTP(w, r)
mut.RUnlock()
timer.ObserveDuration()
}
func handleAssets(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "no-cache, must-revalidate")
assets := auto.Assets()
path := r.URL.Path[1:]
if path == "" {
@@ -194,11 +265,28 @@ func handleAssets(w http.ResponseWriter, r *http.Request) {
return
}
etag := fmt.Sprintf("%d", auto.Generated)
modified := time.Unix(auto.Generated, 0).UTC()
w.Header().Set("Last-Modified", modified.Format(http.TimeFormat))
w.Header().Set("Etag", etag)
mtype := mimeTypeForFile(path)
if len(mtype) != 0 {
w.Header().Set("Content-Type", mtype)
}
if t, err := time.Parse(http.TimeFormat, r.Header.Get("If-Modified-Since")); err == nil && modified.Add(time.Second).After(t) {
w.WriteHeader(http.StatusNotModified)
return
}
if match := r.Header.Get("If-None-Match"); match != "" {
if strings.Contains(match, etag) {
w.WriteHeader(http.StatusNotModified)
return
}
}
if strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
w.Header().Set("Content-Encoding", "gzip")
} else {
@@ -241,6 +329,15 @@ func mimeTypeForFile(file string) string {
}
func handleRequest(w http.ResponseWriter, r *http.Request) {
timer := prometheus.NewTimer(apiRequestsSeconds.WithLabelValues(r.Method))
lw := NewLoggingResponseWriter(w)
defer func() {
timer.ObserveDuration()
apiRequestsTotal.WithLabelValues(r.Method, strconv.Itoa(lw.statusCode)).Inc()
}()
if ipHeader != "" {
r.RemoteAddr = r.Header.Get(ipHeader)
}
@@ -248,13 +345,13 @@ func handleRequest(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case "GET":
if limit(r.RemoteAddr, getLRUCache, getMut, getLimit, getLimitBurst) {
w.WriteHeader(429)
w.WriteHeader(httpStatusEnhanceYourCalm)
return
}
handleGetRequest(w, r)
case "POST":
if limit(r.RemoteAddr, postLRUCache, postMut, postLimit, postLimitBurst) {
w.WriteHeader(429)
w.WriteHeader(httpStatusEnhanceYourCalm)
return
}
handlePostRequest(w, r)
@@ -273,12 +370,9 @@ func handleGetRequest(w http.ResponseWriter, r *http.Request) {
mut.RUnlock()
// Shuffle
for i := range relays {
j := rand.Intn(i + 1)
relays[i], relays[j] = relays[j], relays[i]
}
rand.Shuffle(relays)
json.NewEncoder(w).Encode(map[string][]relay{
json.NewEncoder(w).Encode(map[string][]*relay{
"relays": relays,
})
}
@@ -315,13 +409,9 @@ func handlePostRequest(w http.ResponseWriter, r *http.Request) {
}
// Get the IP address of the client
rhost, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
if debug {
log.Println("Failed to split remote address", r.RemoteAddr)
}
http.Error(w, err.Error(), 500)
return
rhost := r.RemoteAddr
if host, _, err := net.SplitHostPort(rhost); err == nil {
rhost = host
}
ip := net.ParseIP(host)
@@ -333,18 +423,18 @@ func handlePostRequest(w http.ResponseWriter, r *http.Request) {
if debug {
log.Println("IP address advertised does not match client IP address", r.RemoteAddr, uri)
}
http.Error(w, "IP address does not match client IP", http.StatusUnauthorized)
http.Error(w, fmt.Sprintf("IP advertised %s does not match client IP %s", host, rhost), http.StatusUnauthorized)
return
}
newRelay.uri = uri
newRelay.Location = getLocation(uri.Host)
for _, current := range permanentRelays {
if current.uri.Host == newRelay.uri.Host {
if debug {
log.Println("Asked to add a relay", newRelay, "which exists in permanent list")
}
http.Error(w, "Invalid request", 500)
http.Error(w, "Invalid request", http.StatusBadRequest)
return
}
}
@@ -352,78 +442,105 @@ func handlePostRequest(w http.ResponseWriter, r *http.Request) {
reschan := make(chan result)
select {
case requests <- request{newRelay, uri, reschan}:
case requests <- request{&newRelay, reschan, prometheus.NewTimer(relayTestActionsSeconds.WithLabelValues("queue"))}:
result := <-reschan
if result.err != nil {
http.Error(w, result.err.Error(), 500)
relayTestsTotal.WithLabelValues("failed").Inc()
http.Error(w, result.err.Error(), http.StatusBadRequest)
return
}
relayTestsTotal.WithLabelValues("success").Inc()
w.Header().Set("Content-Type", "application/json; charset=utf-8")
json.NewEncoder(w).Encode(map[string]time.Duration{
"evictionIn": result.eviction,
})
default:
relayTestsTotal.WithLabelValues("dropped").Inc()
if debug {
log.Println("Dropping request")
}
w.WriteHeader(429)
w.WriteHeader(httpStatusEnhanceYourCalm)
}
}
func requestProcessor() {
for request := range requests {
if debug {
log.Println("Request for", request.relay)
}
if !client.TestRelay(request.uri, []tls.Certificate{testCert}, time.Second, 2*time.Second, 3) {
if debug {
log.Println("Test for relay", request.relay, "failed")
}
request.result <- result{fmt.Errorf("test failed"), 0}
continue
if request.queueTimer != nil {
request.queueTimer.ObserveDuration()
}
mut.Lock()
timer, ok := evictionTimers[request.relay.uri.Host]
if ok {
if debug {
log.Println("Stopping existing timer for", request.relay)
}
timer.Stop()
}
for i, current := range knownRelays {
if current.uri.Host == request.relay.uri.Host {
if debug {
log.Println("Relay", request.relay, "already exists")
}
// Evict the old entry anyway, as configuration might have changed.
last := len(knownRelays) - 1
knownRelays[i] = knownRelays[last]
knownRelays = knownRelays[:last]
goto found
}
}
if debug {
log.Println("Adding new relay", request.relay)
}
found:
knownRelays = append(knownRelays, request.relay)
evictionTimers[request.relay.uri.Host] = time.AfterFunc(evictionTime, evict(request.relay))
mut.Unlock()
request.result <- result{nil, evictionTime}
timer := prometheus.NewTimer(relayTestActionsSeconds.WithLabelValues("test"))
handleRelayTest(request)
timer.ObserveDuration()
}
}
func evict(relay relay) func() {
func handleRelayTest(request request) {
if debug {
log.Println("Request for", request.relay)
}
if !client.TestRelay(request.relay.uri, []tls.Certificate{testCert}, time.Second, 2*time.Second, 3) {
if debug {
log.Println("Test for relay", request.relay, "failed")
}
request.result <- result{fmt.Errorf("connection test failed"), 0}
return
}
stats := fetchStats(request.relay)
location := getLocation(request.relay.uri.Host)
mut.Lock()
if stats != nil {
updateMetrics(request.relay.uri.Host, *stats, location)
}
request.relay.Stats = stats
request.relay.StatsRetrieved = time.Now()
request.relay.Location = location
timer, ok := evictionTimers[request.relay.uri.Host]
if ok {
if debug {
log.Println("Stopping existing timer for", request.relay)
}
timer.Stop()
}
for i, current := range knownRelays {
if current.uri.Host == request.relay.uri.Host {
if debug {
log.Println("Relay", request.relay, "already exists")
}
// Evict the old entry anyway, as configuration might have changed.
last := len(knownRelays) - 1
knownRelays[i] = knownRelays[last]
knownRelays = knownRelays[:last]
goto found
}
}
if debug {
log.Println("Adding new relay", request.relay)
}
found:
knownRelays = append(knownRelays, request.relay)
evictionTimers[request.relay.uri.Host] = time.AfterFunc(evictionTime, evict(request.relay))
mut.Unlock()
if err := saveRelays(knownRelaysFile, knownRelays); err != nil {
log.Println("Failed to write known relays: " + err.Error())
}
request.result <- result{nil, evictionTime}
}
func evict(relay *relay) func() {
return func() {
mut.Lock()
defer mut.Unlock()
@@ -438,6 +555,7 @@ func evict(relay relay) func() {
last := len(knownRelays) - 1
knownRelays[i] = knownRelays[last]
knownRelays = knownRelays[:last]
deleteMetrics(current.uri.Host)
}
}
delete(evictionTimers, relay.uri.Host)
@@ -445,13 +563,12 @@ func evict(relay relay) func() {
}
func limit(addr string, cache *lru.Cache, lock sync.RWMutex, intv time.Duration, burst int) bool {
host, _, err := net.SplitHostPort(addr)
if err != nil {
return false
if host, _, err := net.SplitHostPort(addr); err == nil {
addr = host
}
lock.RLock()
bkt, ok := cache.Get(host)
bkt, ok := cache.Get(addr)
lock.RUnlock()
if ok {
bkt := bkt.(*rate.Limiter)
@@ -461,18 +578,20 @@ func limit(addr string, cache *lru.Cache, lock sync.RWMutex, intv time.Duration,
}
} else {
lock.Lock()
cache.Add(host, rate.NewLimiter(rate.Every(intv), burst))
cache.Add(addr, rate.NewLimiter(rate.Every(intv), burst))
lock.Unlock()
}
return false
}
func loadPermanentRelays(file string) {
func loadRelays(file string) []*relay {
content, err := ioutil.ReadFile(file)
if err != nil {
log.Fatal(err)
log.Println("Failed to load relays: " + err.Error())
return nil
}
var relays []*relay
for _, line := range strings.Split(string(content), "\n") {
if len(line) == 0 {
continue
@@ -481,21 +600,30 @@ func loadPermanentRelays(file string) {
uri, err := url.Parse(line)
if err != nil {
if debug {
log.Println("Skipping permanent relay", line, "due to parse error", err)
log.Println("Skipping relay", line, "due to parse error", err)
}
continue
}
permanentRelays = append(permanentRelays, relay{
relays = append(relays, &relay{
URL: line,
Location: getLocation(uri.Host),
uri: uri,
})
if debug {
log.Println("Adding permanent relay", line)
log.Println("Adding relay", line)
}
}
return relays
}
func saveRelays(file string, relays []*relay) error {
var content string
for _, relay := range relays {
content += relay.uri.String() + "\n"
}
return ioutil.WriteFile(file, []byte(content), 0777)
}
func createTestCertificate() tls.Certificate {
@@ -505,7 +633,7 @@ func createTestCertificate() tls.Certificate {
}
certFile, keyFile := filepath.Join(tmpDir, "cert.pem"), filepath.Join(tmpDir, "key.pem")
cert, err := tlsutil.NewCertificate(certFile, keyFile, "relaypoolsrv", 3072)
cert, err := tlsutil.NewCertificate(certFile, keyFile, "relaypoolsrv")
if err != nil {
log.Fatalln("Failed to create test X509 key pair:", err)
}
@@ -514,6 +642,8 @@ func createTestCertificate() tls.Certificate {
}
func getLocation(host string) location {
timer := prometheus.NewTimer(locationLookupSeconds)
defer timer.ObserveDuration()
db, err := geoip2.Open(geoipPath)
if err != nil {
return location{}
@@ -531,7 +661,24 @@ func getLocation(host string) location {
}
return location{
Latitude: city.Location.Latitude,
Longitude: city.Location.Longitude,
Latitude: city.Location.Latitude,
City: city.City.Names["en"],
Country: city.Country.IsoCode,
Continent: city.Continent.Code,
}
}
type loggingResponseWriter struct {
http.ResponseWriter
statusCode int
}
func NewLoggingResponseWriter(w http.ResponseWriter) *loggingResponseWriter {
return &loggingResponseWriter{w, http.StatusOK}
}
func (lrw *loggingResponseWriter) WriteHeader(code int) {
lrw.statusCode = code
lrw.ResponseWriter.WriteHeader(code)
}

259
cmd/strelaypoolsrv/stats.go Normal file
View File

@@ -0,0 +1,259 @@
// Copyright (C) 2018 Audrius Butkevicius and Contributors (see the CONTRIBUTORS file).
package main
import (
"encoding/json"
"net"
"net/http"
"os"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/syncthing/syncthing/lib/sync"
)
func init() {
processCollectorOpts := prometheus.ProcessCollectorOpts{
Namespace: "syncthing_relaypoolsrv",
PidFn: func() (int, error) {
return os.Getpid(), nil
},
}
prometheus.MustRegister(
prometheus.NewProcessCollector(processCollectorOpts),
)
}
var (
statusClient = http.Client{
Timeout: 5 * time.Second,
}
apiRequestsTotal = makeCounter("api_requests_total", "Number of API requests.", "type", "result")
apiRequestsSeconds = makeSummary("api_requests_seconds", "Latency of API requests.", "type")
relayTestsTotal = makeCounter("tests_total", "Number of relay tests.", "result")
relayTestActionsSeconds = makeSummary("test_actions_seconds", "Latency of relay test actions.", "type")
locationLookupSeconds = makeSummary("location_lookup_seconds", "Latency of location lookups.").WithLabelValues()
metricsRequestsSeconds = makeSummary("metrics_requests_seconds", "Latency of metric requests.").WithLabelValues()
scrapeSeconds = makeSummary("relay_scrape_seconds", "Latency of metric scrapes from remote relays.", "result")
relayUptime = makeGauge("relay_uptime", "Uptime of relay", "relay")
relayPendingSessionKeys = makeGauge("relay_pending_session_keys", "Number of pending session keys (two keys per session, one per each side of the connection)", "relay")
relayActiveSessions = makeGauge("relay_active_sessions", "Number of sessions that are happening, a session contains two parties", "relay")
relayConnections = makeGauge("relay_connections", "Number of devices connected to the relay", "relay")
relayProxies = makeGauge("relay_proxies", "Number of active proxy routines sending data between peers (two proxies per session, one for each way)", "relay")
relayBytesProxied = makeGauge("relay_bytes_proxied", "Number of bytes proxied by the relay", "relay")
relayGoRoutines = makeGauge("relay_go_routines", "Number of Go routines in the process", "relay")
relaySessionRate = makeGauge("relay_session_rate", "Rate applied per session", "relay")
relayGlobalRate = makeGauge("relay_global_rate", "Global rate applied on the whole relay", "relay")
relayBuildInfo = makeGauge("relay_build_info", "Build information about a relay", "relay", "go_version", "go_os", "go_arch")
relayLocationInfo = makeGauge("relay_location_info", "Location information about a relay", "relay", "city", "country", "continent")
lastStats = make(map[string]stats)
)
func makeGauge(name string, help string, labels ...string) *prometheus.GaugeVec {
gauge := prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "relaypoolsrv",
Name: name,
Help: help,
},
labels,
)
prometheus.MustRegister(gauge)
return gauge
}
func makeSummary(name string, help string, labels ...string) *prometheus.SummaryVec {
summary := prometheus.NewSummaryVec(
prometheus.SummaryOpts{
Namespace: "syncthing",
Subsystem: "relaypoolsrv",
Name: name,
Help: help,
Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
},
labels,
)
prometheus.MustRegister(summary)
return summary
}
func makeCounter(name string, help string, labels ...string) *prometheus.CounterVec {
counter := prometheus.NewCounterVec(
prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "relaypoolsrv",
Name: name,
Help: help,
},
labels,
)
prometheus.MustRegister(counter)
return counter
}
func statsRefresher(interval time.Duration) {
ticker := time.NewTicker(interval)
for range ticker.C {
refreshStats()
}
}
type statsFetchResult struct {
relay *relay
stats *stats
}
func refreshStats() {
mut.RLock()
relays := append(permanentRelays, knownRelays...)
mut.RUnlock()
now := time.Now()
wg := sync.NewWaitGroup()
results := make(chan statsFetchResult, len(relays))
for _, rel := range relays {
wg.Add(1)
go func(rel *relay) {
t0 := time.Now()
stats := fetchStats(rel)
duration := time.Since(t0).Seconds()
result := "success"
if stats == nil {
result = "failed"
}
scrapeSeconds.WithLabelValues(result).Observe(duration)
results <- statsFetchResult{
relay: rel,
stats: fetchStats(rel),
}
wg.Done()
}(rel)
}
wg.Wait()
close(results)
mut.Lock()
relayBuildInfo.Reset()
relayLocationInfo.Reset()
for result := range results {
result.relay.StatsRetrieved = now
result.relay.Stats = result.stats
if result.stats == nil {
deleteMetrics(result.relay.uri.Host)
} else {
updateMetrics(result.relay.uri.Host, *result.stats, result.relay.Location)
}
}
mut.Unlock()
}
func fetchStats(relay *relay) *stats {
statusAddr := relay.uri.Query().Get("statusAddr")
if statusAddr == "" {
statusAddr = ":22070"
}
statusHost, statusPort, err := net.SplitHostPort(statusAddr)
if err != nil {
return nil
}
if statusHost == "" {
if host, _, err := net.SplitHostPort(relay.uri.Host); err != nil {
return nil
} else {
statusHost = host
}
}
url := "http://" + net.JoinHostPort(statusHost, statusPort) + "/status"
response, err := statusClient.Get(url)
if err != nil {
return nil
}
var stats stats
if json.NewDecoder(response.Body).Decode(&stats); err != nil {
return nil
}
return &stats
}
func updateMetrics(host string, stats stats, location location) {
if stats.GoVersion != "" || stats.GoOS != "" || stats.GoArch != "" {
relayBuildInfo.WithLabelValues(host, stats.GoVersion, stats.GoOS, stats.GoArch).Add(1)
}
if location.City != "" || location.Country != "" || location.Continent != "" {
relayLocationInfo.WithLabelValues(host, location.City, location.Country, location.Continent).Add(1)
}
if lastStat, ok := lastStats[host]; ok {
stats = mergeStats(stats, lastStat)
}
relayUptime.WithLabelValues(host).Set(float64(stats.UptimeSeconds))
relayPendingSessionKeys.WithLabelValues(host).Set(float64(stats.PendingSessionKeys))
relayActiveSessions.WithLabelValues(host).Set(float64(stats.ActiveSessions))
relayConnections.WithLabelValues(host).Set(float64(stats.Connections))
relayProxies.WithLabelValues(host).Set(float64(stats.Proxies))
relayBytesProxied.WithLabelValues(host).Set(float64(stats.BytesProxied))
relayGoRoutines.WithLabelValues(host).Set(float64(stats.GoRoutines))
relaySessionRate.WithLabelValues(host).Set(float64(stats.Options.SessionRate))
relayGlobalRate.WithLabelValues(host).Set(float64(stats.Options.GlobalRate))
lastStats[host] = stats
}
func deleteMetrics(host string) {
relayUptime.DeleteLabelValues(host)
relayPendingSessionKeys.DeleteLabelValues(host)
relayActiveSessions.DeleteLabelValues(host)
relayConnections.DeleteLabelValues(host)
relayProxies.DeleteLabelValues(host)
relayBytesProxied.DeleteLabelValues(host)
relayGoRoutines.DeleteLabelValues(host)
relaySessionRate.DeleteLabelValues(host)
relayGlobalRate.DeleteLabelValues(host)
delete(lastStats, host)
}
// Due to some unexplainable behaviour, some of the numbers sometimes travel slightly backwards (by less than 1%)
// This happens between scrapes, which is 30s, so this can't be a race.
// This causes prometheus to assume a "rate reset", hence causes phenomenal spikes.
// One of the number that moves backwards is BytesProxied, which atomically increments a counter with numeric value
// returned by net.Conn.Read(). I don't think that can return a negative value, so I have no idea what's going on.
func mergeStats(new stats, old stats) stats {
new.UptimeSeconds = mergeValue(new.UptimeSeconds, old.UptimeSeconds)
new.PendingSessionKeys = mergeValue(new.PendingSessionKeys, old.PendingSessionKeys)
new.ActiveSessions = mergeValue(new.ActiveSessions, old.ActiveSessions)
new.Connections = mergeValue(new.Connections, old.Connections)
new.Proxies = mergeValue(new.Proxies, old.Proxies)
new.BytesProxied = mergeValue(new.BytesProxied, old.BytesProxied)
new.GoRoutines = mergeValue(new.GoRoutines, old.GoRoutines)
new.Options.SessionRate = mergeValue(new.Options.SessionRate, old.Options.SessionRate)
new.Options.GlobalRate = mergeValue(new.Options.GlobalRate, old.Options.GlobalRate)
return new
}
func mergeValue(new, old int) int {
if new >= old {
return new // normal increase
}
if float64(new) > 0.99*float64(old) {
return old // slight backward movement
}
return new // reset (relay restart)
}

View File

@@ -0,0 +1,21 @@
// Copyright (C) 2015 Audrius Butkevicius and Contributors (see the CONTRIBUTORS file).
package main
import (
"testing"
)
func TestMerge(t *testing.T) {
if mergeValue(1001, 1000) != 1001 {
t.Error("the computer says no")
}
if mergeValue(999, 1000) != 1000 {
t.Error("the computer says no")
}
if mergeValue(1, 1000) != 1 {
t.Error("the computer says no")
}
}

View File

@@ -0,0 +1,9 @@
[strelaysrv]
title=Syncthing relay server
description=Proxies traffic of syncthing client behind firewalls
ports=22067/tcp
[strelaysrv-metrics]
title=Syncthing relay metrics
description=Provides metrics about the syncthing relay server
ports=22070/tcp

View File

@@ -0,0 +1,5 @@
# Default settings for syncthing-relaysrv (strelaysrv).
NAT=true
## Add Options here:
RELAYSRV_OPTS=

View File

@@ -1,17 +1,25 @@
[Unit]
Description=Syncthing relay server
Description=Syncthing Relay Server
After=network.target
Documentation=man:strelaysrv(1)
[Service]
User=strelaysrv
Group=strelaysrv
ExecStart=/usr/bin/strelaysrv
WorkingDirectory=/var/lib/strelaysrv
WorkingDirectory=/var/lib/syncthing-relaysrv
EnvironmentFile=/etc/default/syncthing-relaysrv
ExecStart=/usr/bin/strelaysrv -nat=${NAT} $RELAYSRV_OPTS
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
# Hardening
User=syncthing-relaysrv
Group=syncthing
ProtectSystem=strict
ReadWritePaths=/var/lib/syncthing-relaysrv
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
[Install]
WantedBy=multi-user.target
Alias=syncthing-relaysrv.service

View File

@@ -86,6 +86,12 @@ var (
pprofEnabled bool
)
// httpClient is the HTTP client we use for outbound requests. It has a
// timeout and may get further options set during initialization.
var httpClient = &http.Client{
Timeout: 30 * time.Second,
}
func main() {
log.SetFlags(log.Lshortfile | log.LstdFlags)
@@ -129,14 +135,14 @@ func main() {
if err != nil {
log.Fatal(err)
}
if laddr.IP != nil && !laddr.IP.IsUnspecified() {
// We bind to a specific address. Our outgoing HTTP requests should
// also come from that address.
laddr.Port = 0
transport, ok := http.DefaultTransport.(*http.Transport)
if ok {
transport.Dial = (&net.Dialer{
Timeout: 30 * time.Second,
LocalAddr: laddr,
}).Dial
boundDialer := &net.Dialer{LocalAddr: laddr}
httpClient.Transport = &http.Transport{
DialContext: boundDialer.DialContext,
}
}
@@ -160,7 +166,7 @@ func main() {
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
log.Println("Failed to load keypair. Generating one, this might take a while...")
cert, err = tlsutil.NewCertificate(certFile, keyFile, "strelaysrv", 3072)
cert, err = tlsutil.NewCertificate(certFile, keyFile, "strelaysrv")
if err != nil {
log.Fatalln("Failed to generate X509 key pair:", err)
}

View File

@@ -7,7 +7,6 @@ import (
"encoding/json"
"io/ioutil"
"log"
"net/http"
"net/url"
"time"
)
@@ -27,7 +26,7 @@ func poolHandler(pool string, uri *url.URL, mapping mapping) {
uriCopy.String(),
})
resp, err := http.Post(pool, "application/json", &b)
resp, err := httpClient.Post(pool, "application/json", &b)
if err != nil {
log.Println("Error joining pool", pool, err)
} else if resp.StatusCode == 500 {

View File

@@ -0,0 +1,4 @@
#!/bin/bash
addgroup --system syncthing
adduser --system --home /var/lib/syncthing-relaysrv --ingroup syncthing syncthing-relaysrv

View File

@@ -22,7 +22,7 @@ import (
var (
sessionMut = sync.RWMutex{}
activeSessions = make([]*session, 0)
pendingSessions = make(map[string]*session, 0)
pendingSessions = make(map[string]*session)
numProxies int64
bytesProxied int64
)

View File

@@ -40,6 +40,10 @@ func getStatus(w http.ResponseWriter, r *http.Request) {
sessionMut.Lock()
// This can potentially be double the number of pending sessions, as each session has two keys, one for each side.
status["version"] = Version
status["buildHost"] = BuildHost
status["buildUser"] = BuildUser
status["buildDate"] = BuildDate
status["startTime"] = rc.startTime
status["uptimeSeconds"] = time.Since(rc.startTime) / time.Second
status["numPendingSessionKeys"] = len(pendingSessions)
@@ -82,9 +86,9 @@ func getStatus(w http.ResponseWriter, r *http.Request) {
}
type rateCalculator struct {
counter *int64 // atomic, must remain 64-bit aligned
rates []int64
prev int64
counter *int64
startTime time.Time
}

View File

@@ -1,69 +0,0 @@
// Copyright (C) 2015 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"encoding/json"
"io"
"github.com/syncthing/syncthing/lib/events"
)
// The auditService subscribes to events and writes these in JSON format, one
// event per line, to the specified writer.
type auditService struct {
w io.Writer // audit destination
stop chan struct{} // signals time to stop
started chan struct{} // signals startup complete
stopped chan struct{} // signals stop complete
}
func newAuditService(w io.Writer) *auditService {
return &auditService{
w: w,
stop: make(chan struct{}),
started: make(chan struct{}),
stopped: make(chan struct{}),
}
}
// Serve runs the audit service.
func (s *auditService) Serve() {
defer close(s.stopped)
sub := events.Default.Subscribe(events.AllEvents)
defer events.Default.Unsubscribe(sub)
enc := json.NewEncoder(s.w)
// We're ready to start processing events.
close(s.started)
for {
select {
case ev := <-sub.C():
enc.Encode(ev)
case <-s.stop:
return
}
}
}
// Stop stops the audit service.
func (s *auditService) Stop() {
close(s.stop)
}
// WaitForStart returns once the audit service is ready to receive events, or
// immediately if it's already running.
func (s *auditService) WaitForStart() {
<-s.started
}
// WaitForStop returns once the audit service has stopped.
// (Needed by the tests.)
func (s *auditService) WaitForStop() {
<-s.stopped
}

View File

@@ -22,11 +22,15 @@ func init() {
panic("Couldn't find block profiler")
}
l.Debugln("Starting block profiling")
go saveBlockingProfiles(profiler)
go func() {
err := saveBlockingProfiles(profiler) // Only returns on error
l.Warnln("Block profiler failed:", err)
panic("Block profiler failed")
}()
}
}
func saveBlockingProfiles(profiler *pprof.Profile) {
func saveBlockingProfiles(profiler *pprof.Profile) error {
runtime.SetBlockProfileRate(1)
t0 := time.Now()
@@ -35,16 +39,16 @@ func saveBlockingProfiles(profiler *pprof.Profile) {
fd, err := os.Create(fmt.Sprintf("block-%05d-%07d.pprof", syscall.Getpid(), startms))
if err != nil {
panic(err)
return err
}
err = profiler.WriteTo(fd, 0)
if err != nil {
panic(err)
return err
}
err = fd.Close()
if err != nil {
panic(err)
return err
}
}
return nil
}

View File

@@ -0,0 +1,152 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"context"
"fmt"
"io/ioutil"
"net/http"
"os"
"path/filepath"
"sort"
"strings"
"time"
"github.com/syncthing/syncthing/lib/sha256"
)
const (
headRequestTimeout = 10 * time.Second
putRequestTimeout = time.Minute
)
// uploadPanicLogs attempts to upload all the panic logs in the named
// directory to the crash reporting server as urlBase. Uploads are attempted
// with the newest log first.
//
// This can can block for a long time. The context can set a final deadline
// for this.
func uploadPanicLogs(ctx context.Context, urlBase, dir string) {
files, err := filepath.Glob(filepath.Join(dir, "panic-*.log"))
if err != nil {
l.Warnln("Failed to list panic logs:", err)
return
}
sort.Sort(sort.Reverse(sort.StringSlice(files)))
for _, file := range files {
if strings.Contains(file, ".reported.") {
// We've already sent this file. It'll be cleaned out at some
// point.
continue
}
if err := uploadPanicLog(ctx, urlBase, file); err != nil {
l.Warnln("Reporting crash:", err)
} else {
// Rename the log so we don't have to try to report it again. This
// succeeds, or it does not. There is no point complaining about it.
_ = os.Rename(file, strings.Replace(file, ".log", ".reported.log", 1))
}
}
}
// uploadPanicLog attempts to upload the named panic log to the crash
// reporting server at urlBase. The panic ID is constructed as the sha256 of
// the log contents. A HEAD request is made to see if the log has already
// been reported. If not, a PUT is made with the log contents.
func uploadPanicLog(ctx context.Context, urlBase, file string) error {
data, err := ioutil.ReadFile(file)
if err != nil {
return err
}
// Remove log lines, for privacy.
data = filterLogLines(data)
hash := fmt.Sprintf("%x", sha256.Sum256(data))
l.Infof("Reporting crash found in %s (report ID %s) ...\n", filepath.Base(file), hash[:8])
url := fmt.Sprintf("%s/%s", urlBase, hash)
headReq, err := http.NewRequest(http.MethodHead, url, nil)
if err != nil {
return err
}
// Set a reasonable timeout on the HEAD request
headCtx, headCancel := context.WithTimeout(ctx, headRequestTimeout)
defer headCancel()
headReq = headReq.WithContext(headCtx)
resp, err := http.DefaultClient.Do(headReq)
if err != nil {
return err
}
resp.Body.Close()
if resp.StatusCode == http.StatusOK {
// It's known, we're done
return nil
}
putReq, err := http.NewRequest(http.MethodPut, url, bytes.NewReader(data))
if err != nil {
return err
}
// Set a reasonable timeout on the PUT request
putCtx, putCancel := context.WithTimeout(ctx, putRequestTimeout)
defer putCancel()
putReq = putReq.WithContext(putCtx)
resp, err = http.DefaultClient.Do(putReq)
if err != nil {
return err
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("upload: %s", resp.Status)
}
return nil
}
// filterLogLines returns the data without any log lines between the first
// line and the panic trace. This is done in-place: the original data slice
// is destroyed.
func filterLogLines(data []byte) []byte {
filtered := data[:0]
matched := false
for _, line := range bytes.Split(data, []byte("\n")) {
switch {
case !matched && bytes.HasPrefix(line, []byte("Panic ")):
// This begins the panic trace, set the matched flag and append.
matched = true
fallthrough
case len(filtered) == 0 || matched:
// This is the first line or inside the panic trace.
if len(filtered) > 0 {
// We add the newline before rather than after because
// bytes.Split sees the \n as *separator* and not line
// ender, so ir will generate a last empty line that we
// don't really want. (We want to keep blank lines in the
// middle of the trace though.)
filtered = append(filtered, '\n')
}
// Remove the device ID prefix. The "plus two" stuff is because
// the line will look like "[foo] whatever" and the end variable
// will end up pointing at the ] and we want to step over that
// and the following space.
if end := bytes.Index(line, []byte("]")); end > 1 && end < len(line)-2 && bytes.HasPrefix(line, []byte("[")) {
line = line[end+2:]
}
filtered = append(filtered, line...)
}
}
return filtered
}

View File

@@ -0,0 +1,37 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"testing"
)
func TestFilterLogLines(t *testing.T) {
in := []byte(`[ABCD123] syncthing version whatever
here is more log data
and more
...
and some more
yet more
Panic detected at like right now
here is panic data
and yet more panic stuff
`)
filtered := []byte(`syncthing version whatever
Panic detected at like right now
here is panic data
and yet more panic stuff
`)
result := filterLogLines(in)
if !bytes.Equal(result, filtered) {
t.Logf("%q\n", result)
t.Error("it should have been filtered")
}
}

View File

@@ -14,15 +14,9 @@ import (
)
var (
l = logger.DefaultLogger.NewFacility("main", "Main package")
httpl = logger.DefaultLogger.NewFacility("http", "REST API")
l = logger.DefaultLogger.NewFacility("main", "Main package")
)
func shouldDebugHTTP() bool {
return l.ShouldDebug("http")
}
func init() {
l.SetDebug("main", strings.Contains(os.Getenv("STTRACE"), "main") || os.Getenv("STTRACE") == "all")
l.SetDebug("http", strings.Contains(os.Getenv("STTRACE"), "http") || os.Getenv("STTRACE") == "all")
}

View File

@@ -23,11 +23,15 @@ func init() {
rate = i
}
l.Debugln("Starting heap profiling")
go saveHeapProfiles(rate)
go func() {
err := saveHeapProfiles(rate) // Only returns on error
l.Warnln("Heap profiler failed:", err)
panic("Heap profiler failed")
}()
}
}
func saveHeapProfiles(rate int) {
func saveHeapProfiles(rate int) error {
runtime.MemProfileRate = rate
var memstats, prevMemstats runtime.MemStats
@@ -38,21 +42,21 @@ func saveHeapProfiles(rate int) {
if memstats.HeapInuse > prevMemstats.HeapInuse {
fd, err := os.Create(name + ".tmp")
if err != nil {
panic(err)
return err
}
err = pprof.WriteHeapProfile(fd)
if err != nil {
panic(err)
return err
}
err = fd.Close()
if err != nil {
panic(err)
return err
}
_ = os.Remove(name) // Error deliberately ignored
os.Remove(name) // Error deliberately ignored
err = os.Rename(name+".tmp", name)
if err != nil {
panic(err)
return err
}
prevMemstats = memstats

View File

@@ -1,125 +0,0 @@
// Copyright (C) 2015 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"os"
"path/filepath"
"runtime"
"strings"
"time"
"github.com/syncthing/syncthing/lib/fs"
)
type locationEnum string
// Use strings as keys to make printout and serialization of the locations map
// more meaningful.
const (
locConfigFile locationEnum = "config"
locCertFile = "certFile"
locKeyFile = "keyFile"
locHTTPSCertFile = "httpsCertFile"
locHTTPSKeyFile = "httpsKeyFile"
locDatabase = "database"
locLogFile = "logFile"
locCsrfTokens = "csrfTokens"
locPanicLog = "panicLog"
locAuditLog = "auditLog"
locGUIAssets = "GUIAssets"
locDefFolder = "defFolder"
)
// Platform dependent directories
var baseDirs = map[string]string{
"config": defaultConfigDir(), // Overridden by -home flag
"home": homeDir(), // User's home directory, *not* -home flag
}
// Use the variables from baseDirs here
var locations = map[locationEnum]string{
locConfigFile: "${config}/config.xml",
locCertFile: "${config}/cert.pem",
locKeyFile: "${config}/key.pem",
locHTTPSCertFile: "${config}/https-cert.pem",
locHTTPSKeyFile: "${config}/https-key.pem",
locDatabase: "${config}/index-v0.14.0.db",
locLogFile: "${config}/syncthing.log", // -logfile on Windows
locCsrfTokens: "${config}/csrftokens.txt",
locPanicLog: "${config}/panic-${timestamp}.log",
locAuditLog: "${config}/audit-${timestamp}.log",
locGUIAssets: "${config}/gui",
locDefFolder: "${home}/Sync",
}
// expandLocations replaces the variables in the location map with actual
// directory locations.
func expandLocations() error {
for key, dir := range locations {
for varName, value := range baseDirs {
dir = strings.Replace(dir, "${"+varName+"}", value, -1)
}
var err error
dir, err = fs.ExpandTilde(dir)
if err != nil {
return err
}
locations[key] = dir
}
return nil
}
// defaultConfigDir returns the default configuration directory, as figured
// out by various the environment variables present on each platform, or dies
// trying.
func defaultConfigDir() string {
switch runtime.GOOS {
case "windows":
if p := os.Getenv("LocalAppData"); p != "" {
return filepath.Join(p, "Syncthing")
}
return filepath.Join(os.Getenv("AppData"), "Syncthing")
case "darwin":
dir, err := fs.ExpandTilde("~/Library/Application Support/Syncthing")
if err != nil {
l.Fatalln(err)
}
return dir
default:
if xdgCfg := os.Getenv("XDG_CONFIG_HOME"); xdgCfg != "" {
return filepath.Join(xdgCfg, "syncthing")
}
dir, err := fs.ExpandTilde("~/.config/syncthing")
if err != nil {
l.Fatalln(err)
}
return dir
}
}
// homeDir returns the user's home directory, or dies trying.
func homeDir() string {
home, err := fs.ExpandTilde("~")
if err != nil {
l.Fatalln(err)
}
return home
}
func timestampedLoc(key locationEnum) string {
// We take the roundtrip via "${timestamp}" instead of passing the path
// directly through time.Format() to avoid issues when the path we are
// expanding contains numbers; otherwise for example
// /home/user2006/.../panic-20060102-150405.log would get both instances of
// 2006 replaced by 2015...
tpl := locations[key]
now := time.Now().Format("20060102-150405")
return strings.Replace(tpl, "${timestamp}", now, -1)
}

View File

File diff suppressed because it is too large Load Diff

View File

@@ -1,65 +0,0 @@
// Copyright (C) 2016 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/util"
)
type mockedConfig struct {
gui config.GUIConfiguration
}
func (c *mockedConfig) GUI() config.GUIConfiguration {
return c.gui
}
func (c *mockedConfig) ListenAddresses() []string {
return nil
}
func (c *mockedConfig) RawCopy() config.Configuration {
cfg := config.Configuration{}
util.SetDefaults(&cfg.Options)
return cfg
}
func (c *mockedConfig) Options() config.OptionsConfiguration {
return config.OptionsConfiguration{}
}
func (c *mockedConfig) Replace(cfg config.Configuration) (config.Waiter, error) {
return nil, nil
}
func (c *mockedConfig) Subscribe(cm config.Committer) {}
func (c *mockedConfig) Folders() map[string]config.FolderConfiguration {
return nil
}
func (c *mockedConfig) Devices() map[protocol.DeviceID]config.DeviceConfiguration {
return nil
}
func (c *mockedConfig) SetDevice(config.DeviceConfiguration) (config.Waiter, error) {
return nil, nil
}
func (c *mockedConfig) SetDevices([]config.DeviceConfiguration) (config.Waiter, error) {
return nil, nil
}
func (c *mockedConfig) Save() error {
return nil
}
func (c *mockedConfig) RequiresRestart() bool {
return false
}

View File

@@ -1,17 +0,0 @@
// Copyright (C) 2016 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
type mockedConnections struct{}
func (m *mockedConnections) Status() map[string]interface{} {
return nil
}
func (m *mockedConnections) NATType() string {
return ""
}

View File

@@ -8,6 +8,7 @@ package main
import (
"bufio"
"context"
"io"
"os"
"os/exec"
@@ -17,7 +18,9 @@ import (
"syscall"
"time"
"github.com/syncthing/syncthing/lib/locations"
"github.com/syncthing/syncthing/lib/osutil"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/sync"
)
@@ -32,6 +35,8 @@ const (
loopThreshold = 60 * time.Second
logFileAutoCloseDelay = 5 * time.Second
logFileMaxOpenTime = time.Minute
panicUploadMaxWait = 30 * time.Second
panicUploadNoticeWait = 10 * time.Second
)
func monitorMain(runtimeOptions RuntimeOptions) {
@@ -71,6 +76,8 @@ func monitorMain(runtimeOptions RuntimeOptions) {
childEnv := childEnv()
first := true
for {
maybeReportPanics()
if t := time.Since(restarts[0]); t < loopThreshold {
l.Warnf("%d restarts in %v; not retrying further", countRestarts, t)
os.Exit(exitError)
@@ -84,18 +91,19 @@ func monitorMain(runtimeOptions RuntimeOptions) {
stderr, err := cmd.StderrPipe()
if err != nil {
l.Fatalln("stderr:", err)
panic(err)
}
stdout, err := cmd.StdoutPipe()
if err != nil {
l.Fatalln("stdout:", err)
panic(err)
}
l.Infoln("Starting syncthing")
err = cmd.Start()
if err != nil {
l.Fatalln(err)
l.Warnln("Error starting the main Syncthing process:", err)
panic("Error starting the main Syncthing process")
}
stdoutMut.Lock()
@@ -127,7 +135,7 @@ func monitorMain(runtimeOptions RuntimeOptions) {
select {
case s := <-stopSign:
l.Infof("Signal %d received; exiting", s)
cmd.Process.Kill()
cmd.Process.Signal(sigTerm)
<-exit
return
@@ -172,6 +180,13 @@ func copyStderr(stderr io.Reader, dst io.Writer) {
br := bufio.NewReader(stderr)
var panicFd *os.File
defer func() {
if panicFd != nil {
_ = panicFd.Close()
maybeReportPanics()
}
}()
for {
line, err := br.ReadString('\n')
if err != nil {
@@ -198,7 +213,7 @@ func copyStderr(stderr io.Reader, dst io.Writer) {
}
if strings.HasPrefix(line, "panic:") || strings.HasPrefix(line, "fatal error:") {
panicFd, err = os.Create(timestampedLoc(locPanicLog))
panicFd, err = os.Create(locations.GetTimestamped(locations.PanicLog))
if err != nil {
l.Warnln("Create panic log:", err)
continue
@@ -418,10 +433,10 @@ func (f *autoclosedFile) closerLoop() {
func childEnv() []string {
var env []string
for _, str := range os.Environ() {
if strings.HasPrefix("STNORESTART=", str) {
if strings.HasPrefix(str, "STNORESTART=") {
continue
}
if strings.HasPrefix("STMONITORED=", str) {
if strings.HasPrefix(str, "STMONITORED=") {
continue
}
env = append(env, str)
@@ -429,3 +444,39 @@ func childEnv() []string {
env = append(env, "STMONITORED=yes")
return env
}
// maybeReportPanics tries to figure out if crash reporting is on or off,
// and reports any panics it can find if it's enabled. We spend at most
// panicUploadMaxWait uploading panics...
func maybeReportPanics() {
// Try to get a config to see if/where panics should be reported.
cfg, err := loadOrDefaultConfig(protocol.EmptyDeviceID)
if err != nil {
l.Warnln("Couldn't load config; not reporting crash")
return
}
// Bail if we're not supposed to report panics.
opts := cfg.Options()
if !opts.CREnabled {
return
}
// Set up a timeout on the whole operation.
ctx, cancel := context.WithTimeout(context.Background(), panicUploadMaxWait)
defer cancel()
// Print a notice if the upload takes a long time.
go func() {
select {
case <-ctx.Done():
return
case <-time.After(panicUploadNoticeWait):
l.Warnln("Uploading crash reports is taking a while, please wait...")
}
}()
// Report the panics.
dir := locations.GetBaseDir(locations.ConfigBaseDir)
uploadPanicLogs(ctx, opts.CRURL, dir)
}

View File

@@ -38,7 +38,10 @@ func savePerfStats(file string) {
t0 := time.Now()
for t := range time.NewTicker(250 * time.Millisecond).C {
syscall.Getrusage(syscall.RUSAGE_SELF, &rusage)
if err := syscall.Getrusage(syscall.RUSAGE_SELF, &rusage); err != nil {
continue
}
curTime := time.Now().UnixNano()
timeDiff := curTime - prevTime
curUsage := rusage.Utime.Nano() + rusage.Stime.Nano()

View File

@@ -1,229 +0,0 @@
// Copyright (C) 2015 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"time"
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/sync"
"github.com/thejerf/suture"
)
// The folderSummaryService adds summary information events (FolderSummary and
// FolderCompletion) into the event stream at certain intervals.
type folderSummaryService struct {
*suture.Supervisor
cfg configIntf
model modelIntf
stop chan struct{}
immediate chan string
// For keeping track of folders to recalculate for
foldersMut sync.Mutex
folders map[string]struct{}
// For keeping track of when the last event request on the API was
lastEventReq time.Time
lastEventReqMut sync.Mutex
}
func newFolderSummaryService(cfg configIntf, m modelIntf) *folderSummaryService {
service := &folderSummaryService{
Supervisor: suture.NewSimple("folderSummaryService"),
cfg: cfg,
model: m,
stop: make(chan struct{}),
immediate: make(chan string),
folders: make(map[string]struct{}),
foldersMut: sync.NewMutex(),
lastEventReqMut: sync.NewMutex(),
}
service.Add(serviceFunc(service.listenForUpdates))
service.Add(serviceFunc(service.calculateSummaries))
return service
}
func (c *folderSummaryService) Stop() {
c.Supervisor.Stop()
close(c.stop)
}
// listenForUpdates subscribes to the event bus and makes note of folders that
// need their data recalculated.
func (c *folderSummaryService) listenForUpdates() {
sub := events.Default.Subscribe(events.LocalIndexUpdated | events.RemoteIndexUpdated | events.StateChanged | events.RemoteDownloadProgress | events.DeviceConnected)
defer events.Default.Unsubscribe(sub)
for {
// This loop needs to be fast so we don't miss too many events.
select {
case ev := <-sub.C():
if ev.Type == events.DeviceConnected {
// When a device connects we schedule a refresh of all
// folders shared with that device.
data := ev.Data.(map[string]string)
deviceID, _ := protocol.DeviceIDFromString(data["id"])
c.foldersMut.Lock()
nextFolder:
for _, folder := range c.cfg.Folders() {
for _, dev := range folder.Devices {
if dev.DeviceID == deviceID {
c.folders[folder.ID] = struct{}{}
continue nextFolder
}
}
}
c.foldersMut.Unlock()
continue
}
// The other events all have a "folder" attribute that they
// affect. Whenever the local or remote index is updated for a
// given folder we make a note of it.
data := ev.Data.(map[string]interface{})
folder := data["folder"].(string)
switch ev.Type {
case events.StateChanged:
if data["to"].(string) == "idle" && data["from"].(string) == "syncing" {
// The folder changed to idle from syncing. We should do an
// immediate refresh to update the GUI. The send to
// c.immediate must be nonblocking so that we can continue
// handling events.
select {
case c.immediate <- folder:
c.foldersMut.Lock()
delete(c.folders, folder)
c.foldersMut.Unlock()
default:
}
}
default:
// This folder needs to be refreshed whenever we do the next
// refresh.
c.foldersMut.Lock()
c.folders[folder] = struct{}{}
c.foldersMut.Unlock()
}
case <-c.stop:
return
}
}
}
// calculateSummaries periodically recalculates folder summaries and
// completion percentage, and sends the results on the event bus.
func (c *folderSummaryService) calculateSummaries() {
const pumpInterval = 2 * time.Second
pump := time.NewTimer(pumpInterval)
for {
select {
case <-pump.C:
t0 := time.Now()
for _, folder := range c.foldersToHandle() {
c.sendSummary(folder)
}
// We don't want to spend all our time calculating summaries. Lets
// set an arbitrary limit at not spending more than about 30% of
// our time here...
wait := 2*time.Since(t0) + pumpInterval
pump.Reset(wait)
case folder := <-c.immediate:
c.sendSummary(folder)
case <-c.stop:
return
}
}
}
// foldersToHandle returns the list of folders needing a summary update, and
// clears the list.
func (c *folderSummaryService) foldersToHandle() []string {
// We only recalculate summaries if someone is listening to events
// (a request to /rest/events has been made within the last
// pingEventInterval).
c.lastEventReqMut.Lock()
last := c.lastEventReq
c.lastEventReqMut.Unlock()
if time.Since(last) > defaultEventTimeout {
return nil
}
c.foldersMut.Lock()
res := make([]string, 0, len(c.folders))
for folder := range c.folders {
res = append(res, folder)
delete(c.folders, folder)
}
c.foldersMut.Unlock()
return res
}
// sendSummary send the summary events for a single folder
func (c *folderSummaryService) sendSummary(folder string) {
// The folder summary contains how many bytes, files etc
// are in the folder and how in sync we are.
data, err := folderSummary(c.cfg, c.model, folder)
if err != nil {
return
}
events.Default.Log(events.FolderSummary, map[string]interface{}{
"folder": folder,
"summary": data,
})
for _, devCfg := range c.cfg.Folders()[folder].Devices {
if devCfg.DeviceID.Equals(myID) {
// We already know about ourselves.
continue
}
if _, ok := c.model.Connection(devCfg.DeviceID); !ok {
// We're not interested in disconnected devices.
continue
}
// Get completion percentage of this folder for the
// remote device.
comp := jsonCompletion(c.model.Completion(devCfg.DeviceID, folder))
comp["folder"] = folder
comp["device"] = devCfg.DeviceID.String()
events.Default.Log(events.FolderCompletion, comp)
}
}
func (c *folderSummaryService) gotEventRequest() {
c.lastEventReqMut.Lock()
c.lastEventReq = time.Now()
c.lastEventReqMut.Unlock()
}
// serviceFunc wraps a function to create a suture.Service without stop
// functionality.
type serviceFunc func()
func (f serviceFunc) Serve() { f() }
func (f serviceFunc) Stop() {}

330
cmd/uraggregate/main.go Normal file
View File

@@ -0,0 +1,330 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"database/sql"
"log"
"os"
"time"
_ "github.com/lib/pq"
)
var dbConn = getEnvDefault("UR_DB_URL", "postgres://user:password@localhost/ur?sslmode=disable")
func getEnvDefault(key, def string) string {
if val := os.Getenv(key); val != "" {
return val
}
return def
}
func main() {
log.SetFlags(log.Ltime | log.Ldate)
log.SetOutput(os.Stdout)
db, err := sql.Open("postgres", dbConn)
if err != nil {
log.Fatalln("database:", err)
}
err = setupDB(db)
if err != nil {
log.Fatalln("database:", err)
}
for {
runAggregation(db)
// Sleep until one minute past next midnight
sleepUntilNext(24*time.Hour, 1*time.Minute)
}
}
func runAggregation(db *sql.DB) {
since := maxIndexedDay(db, "VersionSummary")
log.Println("Aggregating VersionSummary data since", since)
rows, err := aggregateVersionSummary(db, since)
if err != nil {
log.Fatalln("aggregate:", err)
}
log.Println("Inserted", rows, "rows")
log.Println("Aggregating UserMovement data")
rows, err = aggregateUserMovement(db)
if err != nil {
log.Fatalln("aggregate:", err)
}
log.Println("Inserted", rows, "rows")
log.Println("Aggregating Performance data")
since = maxIndexedDay(db, "Performance")
rows, err = aggregatePerformance(db, since)
if err != nil {
log.Fatalln("aggregate:", err)
}
log.Println("Inserted", rows, "rows")
log.Println("Aggregating BlockStats data")
since = maxIndexedDay(db, "BlockStats")
rows, err = aggregateBlockStats(db, since)
if err != nil {
log.Fatalln("aggregate:", err)
}
log.Println("Inserted", rows, "rows")
}
func sleepUntilNext(intv, margin time.Duration) {
now := time.Now().UTC()
next := now.Truncate(intv).Add(intv).Add(margin)
log.Println("Sleeping until", next)
time.Sleep(next.Sub(now))
}
func setupDB(db *sql.DB) error {
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS VersionSummary (
Day TIMESTAMP NOT NULL,
Version VARCHAR(8) NOT NULL,
Count INTEGER NOT NULL
)`)
if err != nil {
return err
}
_, err = db.Exec(`CREATE TABLE IF NOT EXISTS UserMovement (
Day TIMESTAMP NOT NULL,
Added INTEGER NOT NULL,
Bounced INTEGER NOT NULL,
Removed INTEGER NOT NULL
)`)
if err != nil {
return err
}
_, err = db.Exec(`CREATE TABLE IF NOT EXISTS Performance (
Day TIMESTAMP NOT NULL,
TotFiles INTEGER NOT NULL,
TotMiB INTEGER NOT NULL,
SHA256Perf DOUBLE PRECISION NOT NULL,
MemorySize INTEGER NOT NULL,
MemoryUsageMiB INTEGER NOT NULL
)`)
if err != nil {
return err
}
_, err = db.Exec(`CREATE TABLE IF NOT EXISTS BlockStats (
Day TIMESTAMP NOT NULL,
Reports INTEGER NOT NULL,
Total INTEGER NOT NULL,
Renamed INTEGER NOT NULL,
Reused INTEGER NOT NULL,
Pulled INTEGER NOT NULL,
CopyOrigin INTEGER NOT NULL,
CopyOriginShifted INTEGER NOT NULL,
CopyElsewhere INTEGER NOT NULL
)`)
if err != nil {
return err
}
var t string
row := db.QueryRow(`SELECT 'UniqueDayVersionIndex'::regclass`)
if err := row.Scan(&t); err != nil {
_, err = db.Exec(`CREATE UNIQUE INDEX UniqueDayVersionIndex ON VersionSummary (Day, Version)`)
}
row = db.QueryRow(`SELECT 'VersionDayIndex'::regclass`)
if err := row.Scan(&t); err != nil {
_, err = db.Exec(`CREATE INDEX VersionDayIndex ON VersionSummary (Day)`)
}
row = db.QueryRow(`SELECT 'MovementDayIndex'::regclass`)
if err := row.Scan(&t); err != nil {
_, err = db.Exec(`CREATE INDEX MovementDayIndex ON UserMovement (Day)`)
}
row = db.QueryRow(`SELECT 'PerformanceDayIndex'::regclass`)
if err := row.Scan(&t); err != nil {
_, err = db.Exec(`CREATE INDEX PerformanceDayIndex ON Performance (Day)`)
}
row = db.QueryRow(`SELECT 'BlockStatsDayIndex'::regclass`)
if err := row.Scan(&t); err != nil {
_, err = db.Exec(`CREATE INDEX BlockStatsDayIndex ON BlockStats (Day)`)
}
return err
}
func maxIndexedDay(db *sql.DB, table string) time.Time {
var t time.Time
row := db.QueryRow("SELECT MAX(Day) FROM " + table)
err := row.Scan(&t)
if err != nil {
return time.Time{}
}
return t
}
func aggregateVersionSummary(db *sql.DB, since time.Time) (int64, error) {
res, err := db.Exec(`INSERT INTO VersionSummary (
SELECT
DATE_TRUNC('day', Received) AS Day,
SUBSTRING(Version FROM '^v\d.\d+') AS Ver,
COUNT(*) AS Count
FROM Reports
WHERE
DATE_TRUNC('day', Received) > $1
AND DATE_TRUNC('day', Received) < DATE_TRUNC('day', NOW())
AND Version like 'v_.%'
GROUP BY Day, Ver
);
`, since)
if err != nil {
return 0, err
}
return res.RowsAffected()
}
func aggregateUserMovement(db *sql.DB) (int64, error) {
rows, err := db.Query(`SELECT
DATE_TRUNC('day', Received) AS Day,
UniqueID
FROM Reports
WHERE
DATE_TRUNC('day', Received) < DATE_TRUNC('day', NOW())
AND Version like 'v_.%'
ORDER BY Day
`)
if err != nil {
return 0, err
}
defer rows.Close()
firstSeen := make(map[string]time.Time)
lastSeen := make(map[string]time.Time)
var minTs time.Time
minTs = minTs.In(time.UTC)
for rows.Next() {
var ts time.Time
var id string
if err := rows.Scan(&ts, &id); err != nil {
return 0, err
}
if minTs.IsZero() {
minTs = ts
}
if _, ok := firstSeen[id]; !ok {
firstSeen[id] = ts
}
lastSeen[id] = ts
}
type sumRow struct {
day time.Time
added int
removed int
bounced int
}
var sumRows []sumRow
for t := minTs; t.Before(time.Now().Truncate(24 * time.Hour)); t = t.AddDate(0, 0, 1) {
var added, removed, bounced int
old := t.Before(time.Now().AddDate(0, 0, -30))
for id, first := range firstSeen {
last := lastSeen[id]
if first.Equal(t) && last.Equal(t) && old {
bounced++
continue
}
if first.Equal(t) {
added++
}
if last == t && old {
removed++
}
}
sumRows = append(sumRows, sumRow{t, added, removed, bounced})
}
tx, err := db.Begin()
if err != nil {
return 0, err
}
if _, err := tx.Exec("DELETE FROM UserMovement"); err != nil {
tx.Rollback()
return 0, err
}
for _, r := range sumRows {
if _, err := tx.Exec("INSERT INTO UserMovement (Day, Added, Removed, Bounced) VALUES ($1, $2, $3, $4)", r.day, r.added, r.removed, r.bounced); err != nil {
tx.Rollback()
return 0, err
}
}
return int64(len(sumRows)), tx.Commit()
}
func aggregatePerformance(db *sql.DB, since time.Time) (int64, error) {
res, err := db.Exec(`INSERT INTO Performance (
SELECT
DATE_TRUNC('day', Received) AS Day,
AVG(TotFiles) As TotFiles,
AVG(TotMiB) As TotMiB,
AVG(SHA256Perf) As SHA256Perf,
AVG(MemorySize) As MemorySize,
AVG(MemoryUsageMiB) As MemoryUsageMiB
FROM Reports
WHERE
DATE_TRUNC('day', Received) > $1
AND DATE_TRUNC('day', Received) < DATE_TRUNC('day', NOW())
AND Version like 'v_.%'
GROUP BY Day
);
`, since)
if err != nil {
return 0, err
}
return res.RowsAffected()
}
func aggregateBlockStats(db *sql.DB, since time.Time) (int64, error) {
// Filter out anything prior 0.14.41 as that has sum aggregations which
// made no sense.
res, err := db.Exec(`INSERT INTO BlockStats (
SELECT
DATE_TRUNC('day', Received) AS Day,
COUNT(1) As Reports,
SUM(BlocksTotal) AS Total,
SUM(BlocksRenamed) AS Renamed,
SUM(BlocksReused) AS Reused,
SUM(BlocksPulled) AS Pulled,
SUM(BlocksCopyOrigin) AS CopyOrigin,
SUM(BlocksCopyOriginShifted) AS CopyOriginShifted,
SUM(BlocksCopyElsewhere) AS CopyElsewhere
FROM Reports
WHERE
DATE_TRUNC('day', Received) > $1
AND DATE_TRUNC('day', Received) < DATE_TRUNC('day', NOW())
AND ReportVersion = 3
AND Version like 'v_.%'
AND Version NOT LIKE 'v0.14.40%'
AND Version NOT LIKE 'v0.14.39%'
AND Version NOT LIKE 'v0.14.38%'
GROUP BY Day
);
`, since)
if err != nil {
return 0, err
}
return res.RowsAffected()
}

240
cmd/ursrv/analytics.go Normal file
View File

@@ -0,0 +1,240 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"regexp"
"sort"
"strconv"
"strings"
)
type analytic struct {
Key string
Count int
Percentage float64
Items []analytic `json:",omitempty"`
}
type analyticList []analytic
func (l analyticList) Less(a, b int) bool {
if l[a].Key == "Others" {
return false
}
if l[b].Key == "Others" {
return true
}
return l[b].Count < l[a].Count // inverse
}
func (l analyticList) Swap(a, b int) {
l[a], l[b] = l[b], l[a]
}
func (l analyticList) Len() int {
return len(l)
}
// Returns a list of frequency analytics for a given list of strings.
func analyticsFor(ss []string, cutoff int) []analytic {
m := make(map[string]int)
t := 0
for _, s := range ss {
m[s]++
t++
}
l := make([]analytic, 0, len(m))
for k, c := range m {
l = append(l, analytic{
Key: k,
Count: c,
Percentage: 100 * float64(c) / float64(t),
})
}
sort.Sort(analyticList(l))
if cutoff > 0 && len(l) > cutoff {
c := 0
for _, i := range l[cutoff:] {
c += i.Count
}
l = append(l[:cutoff], analytic{
Key: "Others",
Count: c,
Percentage: 100 * float64(c) / float64(t),
})
}
return l
}
// Find the points at which certain penetration levels are met
func penetrationLevels(as []analytic, points []float64) []analytic {
sort.Slice(as, func(a, b int) bool {
return versionLess(as[b].Key, as[a].Key)
})
var res []analytic
idx := 0
sum := 0.0
for _, a := range as {
sum += a.Percentage
if sum >= points[idx] {
a.Count = int(points[idx])
a.Percentage = sum
res = append(res, a)
idx++
if idx == len(points) {
break
}
}
}
return res
}
func statsForInts(data []int) [4]float64 {
var res [4]float64
if len(data) == 0 {
return res
}
sort.Ints(data)
res[0] = float64(data[int(float64(len(data))*0.05)])
res[1] = float64(data[len(data)/2])
res[2] = float64(data[int(float64(len(data))*0.95)])
res[3] = float64(data[len(data)-1])
return res
}
func statsForFloats(data []float64) [4]float64 {
var res [4]float64
if len(data) == 0 {
return res
}
sort.Float64s(data)
res[0] = data[int(float64(len(data))*0.05)]
res[1] = data[len(data)/2]
res[2] = data[int(float64(len(data))*0.95)]
res[3] = data[len(data)-1]
return res
}
func group(by func(string) string, as []analytic, perGroup int) []analytic {
var res []analytic
next:
for _, a := range as {
group := by(a.Key)
for i := range res {
if res[i].Key == group {
res[i].Count += a.Count
res[i].Percentage += a.Percentage
if len(res[i].Items) < perGroup {
res[i].Items = append(res[i].Items, a)
}
continue next
}
}
res = append(res, analytic{
Key: group,
Count: a.Count,
Percentage: a.Percentage,
Items: []analytic{a},
})
}
sort.Sort(analyticList(res))
return res
}
func byVersion(s string) string {
parts := strings.Split(s, ".")
if len(parts) >= 2 {
return strings.Join(parts[:2], ".")
}
return s
}
func byPlatform(s string) string {
parts := strings.Split(s, "-")
if len(parts) >= 2 {
return parts[0]
}
return s
}
var numericGoVersion = regexp.MustCompile(`^go[0-9]\.[0-9]+`)
func byCompiler(s string) string {
if m := numericGoVersion.FindString(s); m != "" {
return m
}
return "Other"
}
func versionLess(a, b string) bool {
arel, apre := versionParts(a)
brel, bpre := versionParts(b)
minlen := len(arel)
if l := len(brel); l < minlen {
minlen = l
}
for i := 0; i < minlen; i++ {
if arel[i] != brel[i] {
return arel[i] < brel[i]
}
}
// Longer version is newer, when the preceding parts are equal
if len(arel) != len(brel) {
return len(arel) < len(brel)
}
if apre != bpre {
// "(+dev)" versions are ahead
if apre == plusStr {
return false
}
if bpre == plusStr {
return true
}
return apre < bpre
}
// don't actually care how the prerelease stuff compares for our purposes
return false
}
// Split a version as returned from transformVersion into parts.
// "1.2.3-beta.2" -> []int{1, 2, 3}, "beta.2"}
func versionParts(v string) ([]int, string) {
parts := strings.SplitN(v[1:], " ", 2) // " (+dev)" versions
if len(parts) == 1 {
parts = strings.SplitN(parts[0], "-", 2) // "-rc.1" type versions
}
fields := strings.Split(parts[0], ".")
release := make([]int, len(fields))
for i, s := range fields {
v, _ := strconv.Atoi(s)
release[i] = v
}
var prerelease string
if len(parts) > 1 {
prerelease = parts[1]
}
return release, prerelease
}

View File

@@ -0,0 +1,31 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import "testing"
func TestCompilerRe(t *testing.T) {
tests := [][3]string{
{`syncthing v0.11.0 (xgcc (Ubuntu 4.9.3-0ubuntu4) 4.9.3 linux-amd64 default) niklas@Niklas-Netbook 2015-04-26 13:15:08 UTC`, "xgcc (Ubuntu 4.9.3-0ubuntu4) 4.9.3", "niklas@Niklas-Netbook"},
{`syncthing v0.12.0-rc5 "Beryllium Bedbug" (go1.4.2 linux-arm android) unknown-user@Felix-T420 2015-10-22 18:32:15 UTC`, "go1.4.2", "unknown-user@Felix-T420"},
{`syncthing v0.13.0-beta.0+39-ge267bf3 "Copper Cockroach" (go1.4.2 linux-amd64) portage@slevermann.de 2016-01-20 08:41:52 UTC`, "go1.4.2", "portage@slevermann.de"},
}
for _, tc := range tests {
m := compilerRe.FindStringSubmatch(tc[0])
if len(m) != 3 {
t.Errorf("Regexp didn't match %q", tc[0])
continue
}
if m[1] != tc[1] {
t.Errorf("Compiler %q != %q", m[1], tc[1])
}
if m[2] != tc[2] {
t.Errorf("Builder %q != %q", m[2], tc[2])
}
}
}

131
cmd/ursrv/formatting.go Normal file
View File

@@ -0,0 +1,131 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"fmt"
"strings"
)
type NumberType int
const (
NumberMetric NumberType = iota
NumberBinary
NumberDuration
)
func number(ntype NumberType, v float64) string {
switch ntype {
case NumberMetric:
return metric(v)
case NumberDuration:
return duration(v)
case NumberBinary:
return binary(v)
default:
return metric(v)
}
}
type suffix struct {
Suffix string
Multiplier float64
}
var metricSuffixes = []suffix{
{"G", 1e9},
{"M", 1e6},
{"k", 1e3},
}
var binarySuffixes = []suffix{
{"Gi", 1 << 30},
{"Mi", 1 << 20},
{"Ki", 1 << 10},
}
var durationSuffix = []suffix{
{"year", 365 * 24 * 60 * 60},
{"month", 30 * 24 * 60 * 60},
{"day", 24 * 60 * 60},
{"hour", 60 * 60},
{"minute", 60},
{"second", 1},
}
func metric(v float64) string {
return withSuffix(v, metricSuffixes, false)
}
func binary(v float64) string {
return withSuffix(v, binarySuffixes, false)
}
func duration(v float64) string {
return withSuffix(v, durationSuffix, true)
}
func withSuffix(v float64, ps []suffix, pluralize bool) string {
for _, p := range ps {
if v >= p.Multiplier {
suffix := p.Suffix
if pluralize && v/p.Multiplier != 1.0 {
suffix += "s"
}
// If the number only has decimal zeroes, strip em off.
num := strings.TrimRight(strings.TrimRight(fmt.Sprintf("%.1f", v/p.Multiplier), "0"), ".")
return fmt.Sprintf("%s %s", num, suffix)
}
}
return strings.TrimRight(strings.TrimRight(fmt.Sprintf("%.1f", v), "0"), ".")
}
// commatize returns a number with sep as thousands separators. Handles
// integers and plain floats.
func commatize(sep, s string) string {
// If no dot, don't do anything.
if !strings.ContainsRune(s, '.') {
return s
}
var b bytes.Buffer
fs := strings.SplitN(s, ".", 2)
l := len(fs[0])
for i := range fs[0] {
b.Write([]byte{s[i]})
if i < l-1 && (l-i)%3 == 1 {
b.WriteString(sep)
}
}
if len(fs) > 1 && len(fs[1]) > 0 {
b.WriteString(".")
b.WriteString(fs[1])
}
return b.String()
}
func proportion(m map[string]int, count int) float64 {
total := 0
isMax := true
for _, n := range m {
total += n
if n > count {
isMax = false
}
}
pct := (100 * float64(count)) / float64(total)
// To avoid rounding errors in the template, surpassing 100% and breaking
// the progress bars.
if isMax && len(m) > 1 && count != total {
pct -= 0.01
}
return pct
}

1663
cmd/ursrv/main.go Normal file
View File

File diff suppressed because it is too large Load Diff

View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.4 KiB

View File

File diff suppressed because one or more lines are too long

View File

File diff suppressed because one or more lines are too long

View File

File diff suppressed because one or more lines are too long

View File

Binary file not shown.

View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

View File

Binary file not shown.

View File

Binary file not shown.

656
cmd/ursrv/static/index.html Normal file
View File

@@ -0,0 +1,656 @@
<!DOCTYPE html>
<!--
Copyright (C) 2014 Jakob Borg and other contributors. All rights reserved.
Use of this source code is governed by an MIT-style license that can be
found in the LICENSE file.
-->
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="">
<meta name="author" content="">
<link rel="shortcut icon" href="static/assets/img/favicon.png">
<title>Syncthing Usage Reports</title>
<link href="static/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script type="text/javascript" src="static/bootstrap/js/bootstrap.min.js"></script>
<script type="text/javascript" src="https://maps.googleapis.com/maps/api/js?libraries=visualization&key=AIzaSyDk5WJ8s7ueLKb99X5DbQ-vkWtPDAKqYs0"></script>
<style type="text/css">
body {
margin: 40px;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, "Noto Sans", sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji";
}
tr.main td {
font-weight: bold;
}
tr.child td.first {
padding-left: 2em;
}
.progress-bar {
overflow:hidden;
white-space:nowrap;
text-overflow: ellipsis;
}
</style>
<script type="text/javascript"
src="https://www.google.com/jsapi?autoload={
'modules':[{
'name':'visualization',
'version':'1',
'packages':['corechart']
}]
}"></script>
<script type="text/javascript">
google.setOnLoadCallback(drawVersionChart);
google.setOnLoadCallback(drawMovementChart);
google.setOnLoadCallback(drawBlockStatsChart);
google.setOnLoadCallback(drawPerformanceCharts);
google.setOnLoadCallback(drawHeatMap);
function drawVersionChart() {
var jsonData = $.ajax({url: "summary.json", dataType:"json", async: false}).responseText;
var rows = JSON.parse(jsonData);
var data = new google.visualization.DataTable();
data.addColumn('date', 'Day');
for (var i = 1; i < rows[0].length; i++){
data.addColumn('number', rows[0][i]);
}
for (var i = 1; i < rows.length; i++){
rows[i][0] = new Date(rows[i][0]);
data.addRow(rows[i]);
};
var options = {
legend: { position: 'bottom', alignment: 'center' },
isStacked: true,
colors: ['rgb(102,194,165)','rgb(252,141,98)','rgb(141,160,203)','rgb(231,138,195)','rgb(166,216,84)','rgb(255,217,47)'],
chartArea: {left: 80, top: 20, width: '1020', height: '300'},
};
var chart = new google.visualization.AreaChart(document.getElementById('versionChart'));
chart.draw(data, options);
}
function drawMovementChart() {
var jsonData = $.ajax({url: "movement.json", dataType:"json", async: false}).responseText;
var rows = JSON.parse(jsonData);
var data = new google.visualization.DataTable();
data.addColumn('date', 'Day');
for (var i = 1; i < rows[0].length; i++){
data.addColumn('number', rows[0][i]);
}
for (var i = 1; i < rows.length; i++){
rows[i][0] = new Date(rows[i][0]);
if (rows[i][1] > 500) {
rows[i][1] = null;
}
if (rows[i][2] < -500) {
rows[i][2] = null;
}
data.addRow(rows[i]);
};
var options = {
legend: { position: 'bottom', alignment: 'center' },
colors: ['rgb(102,194,165)','rgb(252,141,98)','rgb(141,160,203)','rgb(231,138,195)','rgb(166,216,84)','rgb(255,217,47)'],
chartArea: {left: 80, top: 20, width: '1020', height: '300'},
};
var chart = new google.visualization.AreaChart(document.getElementById('movementChart'));
chart.draw(data, options);
}
function formatGibibytes(gibibytes, decimals) {
if(gibibytes == 0) return '0 GiB';
var k = 1024,
dm = decimals || 2,
sizes = ['GiB', 'TiB', 'PiB', 'EiB', 'ZiB', 'YiB'],
i = Math.floor(Math.log(gibibytes) / Math.log(k));
if (i < 0) {
sizes = 'MiB';
} else {
sizes = sizes[i];
}
return parseFloat((gibibytes / Math.pow(k, i)).toFixed(dm)) + ' ' + sizes;
}
function drawBlockStatsChart() {
var jsonData = $.ajax({url: "blockstats.json", dataType:"json", async: false}).responseText;
var rows = JSON.parse(jsonData);
var data = new google.visualization.DataTable();
data.addColumn('date', 'Day');
for (var i = 1; i < rows[0].length; i++){
data.addColumn('number', rows[0][i]);
}
var totals = [0, 0, 0, 0, 0, 0];
for (var i = 1; i < rows.length; i++){
rows[i][0] = new Date(rows[i][0]);
for (var j = 2; j < rows[i].length; j++) {
totals[j-2] += rows[i][j];
}
data.addRow(rows[i]);
};
var totalTotals = totals.reduce(function(a, b) { return a + b; }, 0);
if (totalTotals > 0) {
var content = "<table class='table'>\n"
for (var j = 2; j < rows[0].length; j++) {
content += "<tr><td><b>" + rows[0][j].replace(' (GiB)', '') + "</b></td><td>" + formatGibibytes(totals[j-2].toFixed(2)) + " (" + ((100*totals[j-2])/totalTotals).toFixed(2) +"%)</td></tr>\n";
}
content += "</table>";
document.getElementById("data-to-date").innerHTML = content;
} else {
// No data, hide it.
document.getElementById("block-stats").outerHTML = "";
return;
}
var options = {
focusTarget: 'category',
vAxes: {0: {}, 1: {}},
series: {0: {type: 'line', targetAxisIndex:1}},
isStacked: true,
legend: {position: 'none'},
colors: ['rgb(102,194,165)','rgb(252,141,98)','rgb(141,160,203)','rgb(231,138,195)','rgb(166,216,84)','rgb(255,217,47)'],
chartArea: {left: 80, top: 20, width: '1020', height: '300'},
};
var chart = new google.visualization.AreaChart(document.getElementById('blockStatsChart'));
chart.draw(data, options);
}
function drawPerformanceCharts() {
var jsonData = $.ajax({url: "/performance.json", dataType:"json", async: false}).responseText;
var rows = JSON.parse(jsonData);
for (var i = 1; i < rows.length; i++){
rows[i][0] = new Date(rows[i][0]);
}
drawChart(rows, 1, 'Total Number of Files', 'totFilesChart', 1e6, 1);
drawChart(rows, 2, 'Total Folder Size (GiB)', 'totMiBChart', 1e6, 1024);
drawChart(rows, 3, 'Hash Performance (MiB/s)', 'hashPerfChart', 1000, 1);
drawChart(rows, 4, 'System RAM Size (GiB)', 'memSizeChart', 1e6, 1024);
drawChart(rows, 5, 'Memory Usage (MiB)', 'memUsageChart', 250, 1);
}
function drawChart(rows, index, title, id, cutoff, divisor) {
var data = new google.visualization.DataTable();
data.addColumn('date', 'Day');
data.addColumn('number', title);
var row;
for (var i = 1; i < rows.length; i++){
row = [rows[i][0], rows[i][index] / divisor];
if (row[1] > cutoff) {
row[1] = null;
}
data.addRow(row);
}
var options = {
legend: { position: 'bottom', alignment: 'center' },
colors: ['rgb(102,194,165)','rgb(252,141,98)','rgb(141,160,203)','rgb(231,138,195)','rgb(166,216,84)','rgb(255,217,47)'],
chartArea: {left: 80, top: 20, width: '1020', height: '300'},
vAxes: {0: {minValue: 0}},
};
var chart = new google.visualization.LineChart(document.getElementById(id));
chart.draw(data, options);
}
var locations = [];
{{range $location, $weight := .locations}}
locations.push({location: new google.maps.LatLng({{- $location.Latitude -}}, {{- $location.Longitude -}}), weight: {{- $weight -}}});
{{- end}}
function drawHeatMap() {
if (locations.length == 0) {
return;
}
var mapBounds = new google.maps.LatLngBounds();
var map = new google.maps.Map(document.getElementById('map'), {
zoom: 1,
mapTypeId: google.maps.MapTypeId.ROADMAP
});
var heatmap = new google.maps.visualization.HeatmapLayer({
data: locations
});
heatmap.set('radius', 10);
heatmap.set('maxIntensity', 20);
heatmap.set('gradient', [
'rgba(0, 255, 255, 0)',
'rgba(0, 255, 255, 1)',
'rgba(0, 191, 255, 1)',
'rgba(0, 127, 255, 1)',
'rgba(0, 63, 255, 1)',
'rgba(0, 0, 255, 1)',
'rgba(0, 0, 223, 1)',
'rgba(0, 0, 191, 1)',
'rgba(0, 0, 159, 1)',
'rgba(0, 0, 127, 1)',
'rgba(63, 0, 91, 1)',
'rgba(127, 0, 63, 1)',
'rgba(191, 0, 31, 1)',
'rgba(255, 0, 0, 1)'
]);
heatmap.setMap(map);
for (var x = 0; x < locations.length; x++) {
mapBounds.extend(locations[x].location);
}
map.fitBounds(mapBounds);
if (locations.length == 1) {
map.setZoom(13);
}
}
</script>
</head>
<body>
<div class="container">
<div class="row">
<div class="col-md-12">
<h1>Syncthing Usage Data</h1>
<h4 id="active-users">Active Users per Day and Version</h4>
<p>
This is the total number of unique users with reporting enabled, per day. Area color represents the major version.
</p>
<div class="img-thumbnail" id="versionChart" style="width: 1130px; height: 400px; padding: 10px;"></div>
<h4 id="joining-leaving">Users Joining and Leaving per Day</h4>
<p>
This is the total number of unique users joining and leaving per day. A user is counted as "joined" on first the day their unique ID is seen, and as "left" on the last day the unique ID was seen before a two weeks or longer absence. "Bounced" refers to users who joined and left on the same day.
</p>
<div class="img-thumbnail" id="movementChart" style="width: 1130px; height: 400px; padding: 10px;"></div>
<p class="text-muted">
Reappearance of users cause the "left" data to shrink retroactively.
</p>
<div id="block-stats">
<h4>Data Transfers per Day</h4>
<p>
This is total data transferred per day. Also shows how much data was saved (not transferred) by each of the methods syncthing uses.
</p>
<div class="img-thumbnail" id="blockStatsChart" style="width: 1130px; height: 400px; padding: 10px;"></div>
<h4 id="totals-to-date">Totals to date</h4>
<p id="data-to-date">
No data
</p>
</div>
<h4 id="metrics">Usage Metrics</h4>
<p>
This is the aggregated usage report data for the last 24 hours. Data based on <b>{{.nodes}}</b> devices that have reported in.
</p>
{{if .locations}}
<div class="img-thumbnail" id="map" style="width: 1130px; height: 400px; padding: 10px;"></div>
<p class="text-muted">
Heatmap max intensity is capped at 20 reports within a location.
</p>
<div class="panel panel-default">
<div class="panel-heading">
<h4 class="panel-title">
<a data-toggle="collapse" href="#collapseTwo">Break down per country</a>
</h4>
</div>
<div id="collapseTwo" class="panel-collapse collapse">
<div class="panel-body">
<div class="row">
<div class="col-md-6">
<table class="table table-striped">
<tbody>
{{range .contries | slice 2 1}}
<tr>
<td style="width: 45%">{{.Key}}</td>
<td style="width: 5%" class="text-right">{{if ge .Pct 10.0}}{{.Pct | printf "%.0f"}}{{else if ge .Pct 1.0}}{{.Pct | printf "%.01f"}}{{else}}{{.Pct | printf "%.02f"}}{{end}}%</td>
<td style="width: 5%" class="text-right">{{.Count}}</td>
<td>
<div class="progress-bar" role="progressbar" aria-valuenow="{{.Pct | printf "%.02f"}}" aria-valuemin="0" aria-valuemax="100" style="width: {{.Pct | printf "%.02f"}}%; height:20px"></div>
</td>
</tr>
{{end}}
</tbody>
</table>
</div>
<div class="col-md-6">
<table class="table table-striped">
<tbody>
{{range .contries | slice 2 2}}
<tr>
<td style="width: 45%">{{.Key}}</td>
<td style="width: 5%" class="text-right">{{if ge .Pct 10.0}}{{.Pct | printf "%.0f"}}{{else if ge .Pct 1.0}}{{.Pct | printf "%.01f"}}{{else}}{{.Pct | printf "%.02f"}}{{end}}%</td>
<td style="width: 5%" class="text-right">{{.Count}}</td>
<td>
<div class="progress-bar" role="progressbar" aria-valuenow="{{.Pct | printf "%.02f"}}" aria-valuemin="0" aria-valuemax="100" style="width: {{.Pct | printf "%.02f"}}%; height:20px"></div>
</td>
</tr>
{{end}}
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
{{end}}
<table class="table table-striped">
<thead>
<tr>
<th></th>
<th colspan="4" class="text-center">
<a href="https://en.wikipedia.org/wiki/Percentile">Percentile</a>
</th>
</tr>
<tr>
<th></th>
<th class="text-right">5%</th>
<th class="text-right">50%</th>
<th class="text-right">95%</th>
<th class="text-right">100%</th>
</tr>
</thead>
<tbody>
{{range .categories}}
<tr>
<td>{{.Descr}}</td>
<td class="text-right">{{index .Values 0 | number .Type | commatize " "}}{{.Unit}}</td>
<td class="text-right">{{index .Values 1 | number .Type | commatize " "}}{{.Unit}}</td>
<td class="text-right">{{index .Values 2 | number .Type | commatize " "}}{{.Unit}}</td>
<td class="text-right">{{index .Values 3 | number .Type | commatize " "}}{{.Unit}}</td>
</tr>
{{end}}
</tbody>
</table>
</div>
</div>
<div class="row">
<div class="col-md-6">
<table class="table table-striped">
<thead>
<tr>
<th>Version</th><th class="text-right">Devices</th><th class="text-right">Share</th>
</tr>
</thead>
<tbody>
{{range .versions}}
{{if gt .Percentage 0.1}}
<tr class="main">
<td>{{.Key}}</td>
<td class="text-right">{{.Count}}</td>
<td class="text-right">{{.Percentage | printf "%.01f"}}%</td>
</tr>
{{range .Items}}
{{if gt .Percentage 0.1}}
<tr class="child">
<td class="first">{{.Key}}</td>
<td class="text-right">{{.Count}}</td>
<td class="text-right">{{.Percentage | printf "%.01f"}}%</td>
</tr>
{{end}}
{{end}}
{{end}}
{{end}}
</tbody>
</table>
<table class="table table-striped">
<thead>
<tr>
<th>Penetration Level</th>
<th>Version</th>
<th class="text-right">Actual</th>
</tr>
</thead>
<tbody>
{{range .versionPenetrations}}
<tr>
<td>{{.Count}}%</td>
<td>&ge; {{.Key}}</td>
<td class="text-right">{{.Percentage | printf "%.01f"}}%</td>
</tr>
{{end}}
</tbody>
</table>
</div>
<div class="col-md-6">
<table class="table table-striped">
<thead>
<tr>
<th>Platform</th>
<th class="text-right">Devices</th>
<th class="text-right">Share</th>
</tr>
</thead>
<tbody>
{{range .platforms}}
<tr class="main">
<td>{{.Key}}</td>
<td class="text-right">{{.Count}}</td>
<td class="text-right">{{.Percentage | printf "%.01f"}}%</td>
</tr>
{{range .Items}}
<tr class="child">
<td class="first">{{.Key}}</td>
<td class="text-right">{{.Count}}</td>
<td class="text-right">{{.Percentage | printf "%.01f"}}%</td>
</tr>
{{end}}
{{end}}
</tbody>
</table>
</div>
</div>
<div class="row">
<div class="col-md-6">
<table class="table table-striped">
<thead>
<tr>
<th>Compiler</th>
<th class="text-right">Devices</th>
<th class="text-right">Share</th>
</tr>
</thead>
<tbody>
{{range .compilers}}
<tr class="main">
<td>{{.Key}}</td>
<td class="text-right">{{.Count}}</td>
<td class="text-right">{{.Percentage | printf "%.01f"}}%</td>
</tr>
{{range .Items}}
{{if or (gt .Percentage 0.1) (eq .Key "Others")}}
<tr class="child">
<td class="first">{{.Key}}</td>
<td class="text-right">{{.Count}}</td>
<td class="text-right">{{.Percentage | printf "%.01f"}}%</td>
</tr>
{{end}}
{{end}}
{{end}}
</tbody>
</table>
</div>
<div class="col-md-6">
<table class="table table-striped">
<thead>
<tr>
<th>Builder</th>
<th class="text-right">Devices</th>
<th class="text-right">Share</th>
</tr>
</thead>
<tbody>
{{range .builders}}
<tr>
<td>{{.Key}}</td>
<td class="text-right">{{.Count}}</td>
<td class="text-right">{{.Percentage | printf "%.01f"}}%</td>
</tr>
{{end}}
</tbody>
</table>
</div>
<div class="col-md-6">
<table class="table table-striped">
<thead>
<tr>
<th>Distribution Channel</th>
<th class="text-right">Devices</th>
<th class="text-right">Share</th>
</tr>
</thead>
<tbody>
{{range .distributions}}
<tr>
<td>{{.Key}}</td>
<td class="text-right">{{.Count}}</td>
<td class="text-right">{{.Percentage | printf "%.01f"}}%</td>
</tr>
{{end}}
</tbody>
</table>
</div>
</div>
<div class="row">
<div class="col-md-12">
<h4 id="features">Feature Usage</h4>
<p>
The following lists feature usage. Some features are reported per report, some are per sum of units within report (eg. devices with static addresses among all known devices per report).
Currently there are <b>{{.versionNodes.v2}}</b> devices reporting for version 2 and <b>{{.versionNodes.v3}}</b> for version 3.
</p>
</div>
</div>
<div class="row">
{{$i := counter}}
{{range $featureName := .featureOrder}}
{{$featureValues := index $.features $featureName }}
{{if $i.DrawTwoDivider}}
</div>
<div class="row">
{{end}}
{{ $i.Increment }}
<div class="col-md-6">
<table class="table table-striped">
<thead><tr>
<th>{{$featureName}} Features</th><th colspan="2" class="text-center">Usage</th>
</tr></thead>
<tbody>
{{range $featureValues}}
<tr>
<td style="width: 50%">{{.Key}} ({{.Version}})</td>
<td style="width: 10%" class="text-right">{{if ge .Pct 10.0}}{{.Pct | printf "%.0f"}}{{else if ge .Pct 1.0}}{{.Pct | printf "%.01f"}}{{else}}{{.Pct | printf "%.02f"}}{{end}}%</td>
<td style="width: 40%" {{if lt .Pct 5.0}}data-toggle="tooltip" title='{{.Count}}'{{end}}>
<div class="progress-bar" role="progressbar" aria-valuenow="{{.Pct | printf "%.02f"}}" aria-valuemin="0" aria-valuemax="100" style="width: {{.Pct | printf "%.02f"}}%; height:20px" {{if ge .Pct 5.0}}data-toggle="tooltip" title='{{.Count}}'{{end}}></div>
</td>
</tr>
{{end}}
</tbody>
</table>
</div>
{{end}}
</div>
<div class="row">
<div class="col-md-12">
<h4 id="features">Feature Group Usage</h4>
<p>
The following lists feature usage groups, which might include multiple occourances of a feature use per report.
</p>
</div>
</div>
<div class="row">
{{$i := counter}}
{{range $featureName := .featureOrder}}
{{$featureValues := index $.featureGroups $featureName }}
{{if $i.DrawTwoDivider}}
</div>
<div class="row">
{{end}}
{{ $i.Increment }}
<div class="col-md-6">
<table class="table table-striped">
<thead><tr>
<th>{{$featureName}} Group Features</th><th colspan="2" class="text-center">Usage</th>
</tr></thead>
<tbody>
{{range $featureValues}}
{{$counts := .Counts}}
<tr>
<td style="width: 50%">
<div data-toggle="tooltip" title='{{range $key, $value := .Counts}}{{$key}} ({{$value | proportion $counts | printf "%.02f"}}% - {{$value}})</br>{{end}}'>
{{.Key}} ({{.Version}})
</div>
</td>
<td style="width: 50%">
<div class="progress" role="progressbar" style="width: 100%">
{{$j := counter}}
{{range $key, $value := .Counts}}
{{with $valuePct := $value | proportion $counts}}
<div class="progress-bar {{ $j.Current | progressBarClassByIndex }}" style='width: {{$valuePct | printf "%.02f"}}%' data-toggle="tooltip" title='{{$key}} ({{$valuePct | printf "%.02f"}}% - {{$value}})'>
{{if ge $valuePct 30.0}}{{$key}}{{end}}
</div>
{{end}}
{{ $j.Increment }}
{{end}}
</div>
</td>
</tr>
{{end}}
</tbody>
</table>
</div>
{{end}}
</div>
<div class="row">
<div class="col-md-12">
<h1 id="performance-charts">Historical Performance Data</h1>
<p>These charts are all the average of the corresponding metric, for the entire population of a given day.</p>
<h4 id="hash-performance">Hash Performance (MiB/s)</h4>
<div class="img-thumbnail" id="hashPerfChart" style="width: 1130px; height: 400px; padding: 10px;"></div>
<h4 id="memory-usage">Memory Usage (MiB)</h4>
<div class="img-thumbnail" id="memUsageChart" style="width: 1130px; height: 400px; padding: 10px;"></div>
<h4 id="total-files">Total Number of Files</h4>
<div class="img-thumbnail" id="totFilesChart" style="width: 1130px; height: 400px; padding: 10px;"></div>
<h4 id="total-size">Total Folder Size (GiB)</h4>
<div class="img-thumbnail" id="totMiBChart" style="width: 1130px; height: 400px; padding: 10px;"></div>
<h4 id="system-ram">System RAM Size (GiB)</h4>
<div class="img-thumbnail" id="memSizeChart" style="width: 1130px; height: 400px; padding: 10px;"></div>
</div>
</div>
</div>
<hr>
<p>
This product includes GeoLite2 data created by MaxMind, available from
<a href="http://www.maxmind.com">http://www.maxmind.com</a>.
</p>
<script type="text/javascript">
$('[data-toggle="tooltip"]').tooltip({html:true});
</script>
</body>
</html>

View File

@@ -0,0 +1,12 @@
# Desktop Entries
This directory contains files to integrate Syncthing in your desktop environment (DE).
Specifically this works for DEs that implement the [XDG Desktop Menu Specification][1], which
is virtually every DE.
To add Syncthing to desktop menus for all users, copy the `.desktop` files to
`/usr/local/share/applications` (root required). To add it for just your user, copy them to `~/.local/share/applications`.
To start Syncthing automatically, you have two options: Either you go to the autostart settings of your DE and choose Syncthing or you copy the `syncthing-start.desktop` file to `~/.config/autostart`.
For more information refer to the [ArchWiki page on Desktop entries][2]
[1]: https://specifications.freedesktop.org/menu-spec/menu-spec-latest.html
[2]: https://wiki.archlinux.org/index.php/Desktop_entries

View File

@@ -0,0 +1,10 @@
[Desktop Entry]
Name=Start Syncthing
GenericName=File synchronization
Comment=Starts the main syncthing process in the background.
Exec=/usr/bin/syncthing -no-browser
Icon=syncthing
Terminal=false
Type=Application
Keywords=synchronization;daemon;
Categories=Network;FileTransfer;P2P

View File

@@ -0,0 +1,10 @@
[Desktop Entry]
Name=Syncthing Web UI
GenericName=File synchronization UI
Comment=Opens Syncthing's Web UI in the default browser (Syncthing must already be started).
Exec=/usr/bin/syncthing -browser-only
Icon=syncthing
Terminal=false
Type=Application
Keywords=synchronization;interface;
Categories=Network;FileTransfer;P2P

View File

@@ -2,7 +2,6 @@
Description=Syncthing - Open Source Continuous File Synchronization for %I
Documentation=man:syncthing(1)
After=network.target
Wants=syncthing-inotify@.service
[Service]
User=%i
@@ -11,5 +10,12 @@ Restart=on-failure
SuccessExitStatus=3 4
RestartForceExitStatus=3 4
# Hardening
ProtectSystem=full
PrivateTmp=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target

View File

@@ -1,7 +1,6 @@
[Unit]
Description=Syncthing - Open Source Continuous File Synchronization
Documentation=man:syncthing(1)
Wants=syncthing-inotify.service
[Service]
ExecStart=/usr/bin/syncthing -no-browser -no-restart -logflags=0
@@ -9,5 +8,10 @@ Restart=on-failure
SuccessExitStatus=3 4
RestartForceExitStatus=3 4
# Hardening
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
NoNewPrivileges=true
[Install]
WantedBy=default.target

Some files were not shown because too many files have changed in this diff Show More