Compare commits

...

652 Commits

Author SHA1 Message Date
Ross Smith II
7f3c8dbff1 build: move nightly build schedule to separate workflow (#10000)
This allows users to easily disable nightly builds in their forks,
simply by disabling the
build-nightly action.

### Testing

I tested it in my fork, and it works.
2025-03-27 09:31:51 +00:00
Jakob Borg
7762e39fb3 chore(syncthing): use file lock on certificate to prevent multiple instances (#10003)
This adds the locking from the SQLite branch, in preparation, so that we
do not inadvertently permit running an instance of each.
2025-03-27 09:26:21 +00:00
Jakob Borg
4235b2c406 chore(ur): add RSS to reported stats (#10002)
For easier comparison in the future.
2025-03-27 10:17:10 +01:00
Syncthing Release Automation
3fd090bfa7 chore(gui, man, authors): update docs, translations, and contributors 2025-03-24 03:49:44 +00:00
Syncthing Release Automation
6dfa54efa6 chore(gui, man, authors): update docs, translations, and contributors 2025-03-17 03:49:35 +00:00
mathias4833
67575e1736 fix(api): prevent tilde expansion in path suggestions (fixes #9990) (#9992)
### Purpose

Path autocompletion wasn't working when using `~` as a shortcut for the
home directory. The issue occurred because the tilde was expanded to
/home/user, which caused the suggestion to no longer match the input
(thus preventing the autocompletion from appearing in the suggestion
list).

To fix this, I replaced the custom `parentAndBase` function, which
handled path splitting in a more complex way, with `filepath.Split` from
the standard `path/filepath` package. This prevents tilde expansion
while keeping the expected behavior for path splitting.

### Testing

The issue has been tested manually on Linux.

### Screenshots


![screenshot](https://github.com/user-attachments/assets/49dd96e2-6d75-4476-946d-0dfb2ac474ef)
2025-03-15 21:06:38 +01:00
Jakob Borg
65923fc255 fix(syncthing): don't auto upgrade to higher major on startup (#9989)
We avoided upgrading to newer major versions during normal auto upgrade
procedures, but currently not in the initial upgrade check on startup.
2025-03-13 07:59:19 +00:00
Jakob Borg
aea763868f build(deps): update dependencies (#9988) 2025-03-13 07:41:05 +00:00
Syncthing Release Automation
26b134ae7b chore(gui, man, authors): update docs, translations, and contributors 2025-03-10 03:45:18 +00:00
Emil Lundberg
893071d2ba refactor(api): extract method configMuxBuilder.postAdjustGui and add test coverage (#9979)
This is extracted from PR #9175. This deduplicates `SetPassword` calls
and makes `postAdjustGui` a single place where PR #9175 can add
another adjustment step for sanitizing changes to WebAuthn
credentials.

This also adds tests to validate that the refactored logic was not
broken.
2025-03-09 21:31:06 +01:00
Emil Lundberg
435f2d2178 refactor(api): make shutdown timeout configurable for tests (#9980)
This is extracted from PR #9175, which adds some tests that seem to
need a longer shutdown timeout when running on GitHub Actions.
2025-03-07 12:50:33 +01:00
Emil Lundberg
8461ca539b refactor(api): deduplicate HTTP test helpers and allow session cookie access (#9977)
These refactorizations were made in [PR #9175][1] to accommodate a few
new variants of authentication method and body content. On request from
reviewers, this PR extracts it as a smaller refactorization to review in
isolation.

### Purpose

This extracts a shared `httpRequest` base function from `httpGet` and
`httpPost`, which will be used in PR #9175 for new helper functions
`httpGetCsrf` (hiding all optional parameters except the CSRF token),
`httpPostCsrf` (same) and `httpPostCsrfAuth` (hiding basic auth
parameters). A `getSessionCookie` function is also extracted from
`hasSessionCookie` and will be used to test that concurrent WebAuthn
authentications result in separate sessions (indicated by different
session cookies).
2025-03-07 11:07:01 +01:00
Jakob Borg
fb977dc61d build: correct API call for Weblate statistics
Something changed...
2025-03-03 08:11:29 +01:00
Jakob Borg
ee7ab4ce25 build(deps): update dependencies (#9978) 2025-03-01 21:33:13 +00:00
polyfloyd
c3ce9713d9 chore(etc): remove /usr/bin prefix from Linux .desktop files (#9966)
Exec accepts just the program name and will look for it in $PATH. This
makes these files work on NixOS which does not create a /usr/bin
directory.
2025-02-28 20:59:07 +01:00
Jakob Borg
6a147091c5 build: use Go 1.24, minimum is Go 1.23 (#9960) 2025-02-12 10:16:46 +01:00
Jakob Borg
6208c36417 fix(policy): do not require multiple maintainers for build changes 2025-02-12 09:47:09 +01:00
Syncthing Release Automation
453fd20eeb chore(gui, man, authors): update docs, translations, and contributors 2025-02-10 03:46:14 +00:00
Tommy van der Vorst
28f0cffdb6 chore(fs): build kqueue instead of fsevents watcher on iOS (#9950)
### Purpose

On iOS, the FSEvents API for watching files (also used on macOS) is not
available, but `kqueue` is. This PR ensures `kqueue` support is built on
iOS instead of the FSEvents based watcher implementation.

Before this PR, you could already use the `kqueue` build option to force
its usage. Unfortunately `gomobile`, the tool that I use to build
Syncthing for iOS and macOS for Synctrain, does not support setting
different build flags for iOS and macOS (unless I build separately for
each, which is a bit of a hassle because XCode nonsense). I am assuming
there are good reasons to support FSEvents even though `kqueue` is also
available on macOS (but I'm not sure why?). I do know FSEvents has been
working fine for me on macOS so it seems best to use FSEvents on macOS
and kqueue on iOS.

Note that this also requires https://github.com/syncthing/notify/pull/4
to be merged in `synchting/notify` (until that is done, this PR will
fail to build on iOS due to `notify` still trying to link to `fsevents`
stuff when the `kqueue` build flag is not set).

### Testing

I compiled both `syncthing/notify` and syncthing with this PR applied,
and used that to successfully build the Synctrain iOS app, which after
this PR works fine and should follow up file changes a bit quicker.

### Screenshots

n/a

### Documentation

n/a

## Authorship

Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.

---------

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2025-02-07 15:40:53 +00:00
Jakob Borg
87c16c6cf5 build(deps): update dependencies (#9951) 2025-02-07 08:44:26 +00:00
dashangcun
5495c98e63 refactor: using slices.Contains to simplify the code (#9918)
### Purpose

This is a [new function](https://pkg.go.dev/slices@go1.21.0#Contains)
added in the go1.21 standard library, which can make the code more
concise and easy to read.

### Testing

Describe what testing has been done, and how the reviewer can test the
change
if new tests are not included.

### Screenshots

If this is a GUI change, include screenshots of the change. If not,
please
feel free to just delete this section.

### Documentation

If this is a user visible change (including API and protocol changes),
add a link here
to the corresponding pull request on https://github.com/syncthing/docs
or describe
the documentation changes necessary.

## Authorship

Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.

Signed-off-by: dashangcun <907225865@qq.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
2025-02-07 08:21:24 +00:00
Jakob Borg
da7d5ce608 build: switch to cloud code signing for Windows (#9948)
The requirements for Windows code signing changed in 2023, so that newly
generated certificates can only be stored in hardware modules. Luckily,
I managed to snag a three year certificate before that so it hasn't
affected us so much. Now though, it does, because our cert is expiring
in March.

This changes the code signing process for Windows to use a cloud
service, Azure Trusted Signing. This appears to work equally well and
outsources the problem entirely, while also being cheaper than the
actual certificate was to begin with. 🤷

The signing entity will be Kastelo AB and not the Syncthing Foundation,
because the latter is almost impossible to get a certificate for as it's
not a normal corporate entity whose existence can be verified, etc. This
is also how it was prior to the latest certificate; it's not ideal, but
I think it's acceptable under the circumstances.
2025-02-06 10:43:23 +01:00
Syncthing Release Automation
b300c297c6 chore(gui, man, authors): update docs, translations, and contributors 2025-02-03 03:45:29 +00:00
Syncthing Release Automation
124673f7a8 chore(gui, man, authors): update docs, translations, and contributors 2025-01-27 03:45:14 +00:00
Jakob Borg
0395cf2bc0 fix(model): clarify errors on Windows user/group lookup (fixes #9929) (#9930)
Currently, this just results in a very ambiguous `setting metadata: lookup
failed` while it could report what it's looking up and why it failed
(not found, etc).
2025-01-20 09:59:05 +01:00
Syncthing Release Automation
36cd70040a chore(gui, man, authors): update docs, translations, and contributors 2025-01-20 03:45:10 +00:00
Jakob Borg
1fbd396ffa chore(scanner): don't warn about cancelled scan (#9920)
We expect a context cancellation when a folder is restarted during a
scan.
2025-01-13 17:35:55 +00:00
Syncthing Release Automation
2834bad85e chore(gui, man, authors): update docs, translations, and contributors 2025-01-13 03:47:27 +00:00
Jakob Borg
516f3e29e8 chore(proto): change symlinktarget to be byte sequence (fixes #9913) (#9914) 2025-01-11 17:38:29 +01:00
Jakob Borg
ab20c16982 fix(api): don't crash requests after failing to unmarshal tokens (fixes #9909) (#9912)
Unclear why this would happen, but apparently it does?
2025-01-09 21:47:10 +01:00
Jakob Borg
0c1df81ee9 fix(scanner): don't warn when legitimately skipping a directory (#9911)
Such as ignored directories etc.
2025-01-09 18:42:18 +00:00
Jakob Borg
d324b2ac86 fix(db): correct unsafe RLock order (fixes #9906) (#9910)
Marshal() was called with the read lock held, and in turn called
Created() which also takes the read lock. This is fine by itself, but
there is a risk of deadlock if another call to lock the mutex happens
concurrently, as the lock call will block the inner rlock and the outer
rlock can never become unlocked.

It's an easy fix as marshalling is guaranteed to be called with a read
lock and does not need to call any methods that read lock themselves.
2025-01-09 19:33:04 +01:00
Jakob Borg
0231089b99 fix(db): add JSON tags to types used in API returns (fixes #9907) (#9908)
These types are also used in the API, and hence need JSON tags.
2025-01-09 09:26:34 +00:00
Jakob Borg
74ffb85467 fix(model): correct type of fileInfoType in API browse response (fixes #9904) (#9905)
This was broken in the Protobuf refactor. The reason is that with the
previous library, generated types would have a JSON marshalling method
that would automatically return strings for enums. In the current
library you need to use the jsonpb marshaller for that, but these are
hand crafted structs so we can't do that. The easy solution is to just
use strings directly, since this is an API-only type anyway.
2025-01-08 11:08:33 +01:00
Syncthing Release Automation
dcd280e6e2 chore(gui, man, authors): update docs, translations, and contributors 2025-01-06 03:47:27 +00:00
Jakob Borg
4e56dbd883 build(deps): update dependencies (#9900) 2025-01-01 19:49:52 +00:00
Syncthing Release Automation
13d7881b80 chore(gui, man, authors): update docs, translations, and contributors 2024-12-30 03:46:20 +00:00
Simon Frei
1d2c53bf3c chore(scanner): avoid scan failures and report if they happen (#9888) 2024-12-28 17:09:38 +01:00
Syncthing Release Automation
9c449c966b chore(gui, man, authors): update docs, translations, and contributors 2024-12-23 03:46:12 +00:00
Jakob Borg
ec2d4638e3 build: fix publish-nightly tools checkout 2024-12-22 09:02:11 +01:00
André Colomb
0ce92befc8 chore(gui): fix merge conflict on Weblate (#9882)
Commit dee920a840 was merged into
Weblate's repository, but later replaced on main by
2167ce9656. This led to a merge conflict,
which this commit fixes cleanly. After merging, we can unlock Weblate
again.
2024-12-21 21:54:17 +00:00
Jakob Borg
2167ce9656 build: reinstate docker push for main 2024-12-21 16:17:21 +01:00
Jakob Borg
0dc85d74aa build: also push to ghcr.io 2024-12-21 15:43:49 +01:00
Jakob Borg
b6e3f8037b build: workaround for tag builds
The GitHub checkout action does weird stuff with tags which breaks `git
describe`. This works around that so we get proper release builds on tag
pushes.
2024-12-21 14:35:04 +01:00
Jakob Borg
00e7161a8f build: compat.json should be in the signed packages bundle 2024-12-20 10:34:29 +01:00
Jakob Borg
b5a7879eca fix(stdiscosrv): handle announcements properly :p (#9881)
Further protobuf refactor damage, also adding some better debugging
2024-12-19 20:43:46 +00:00
Jakob Borg
4355dc69ea fix(discovery): properly unmarshal local discovery (#9880)
Damage from recent protobuf refactoring
2024-12-19 20:16:44 +00:00
Jakob Borg
371ba69447 build: slightly streamline build dependencies 2024-12-19 15:08:17 +01:00
Jakob Borg
79ef57d0ea fix(scanner): continue walk after special files (fixes #9872) (#9877)
We must skip unix sockets, fifos, etc when scanning as these are not
filetypes we can handle. Currently we return a "bug" error, which
results in the walk being aborted and the rest of the tree being
essentially invisible to Syncthing. Instead, just ignore these files and
continue onwards.

This might well be #9859 as well but I can't confirm.
2024-12-19 08:27:53 +01:00
Jakob Borg
8bd6bdd397 chore(model): clarify log message (fixes #9875) (#9876) 2024-12-18 07:56:06 +00:00
Simon Frei
ce3248cea7 chore(fs): add debug logging for case cache registry (#9869)
Clearly something is up with it, but I still have no clue what. This
might give some clue when affected user enable debug logging.
2024-12-16 12:04:24 +01:00
Jakob Borg
99a6f3a5b6 docs: update section on code signing 2024-12-16 11:42:34 +01:00
Jakob Borg
c2e10dc156 build: skip Docker push for main, reserve for release 2024-12-16 11:39:13 +01:00
Jakob Borg
fc914f3237 build: sign asc files using ezapt
And same keys as APT archive
2024-12-16 11:35:41 +01:00
Jakob Borg
811d3752d0 build: loki vars are in secrets 2024-12-16 09:26:21 +01:00
Jakob Borg
00827dd5c1 build: consolidate release environment for actions
signing & docker -> release
2024-12-16 08:57:35 +01:00
Syncthing Release Automation
a981c21d27 chore(gui, man, authors): update docs, translations, and contributors 2024-12-16 03:50:41 +00:00
Jakob Borg
163dc122f3 build: update compat.yaml for Go 1.24 (rc1) (fixes #9870) 2024-12-15 11:09:14 +01:00
Jakob Borg
83727e0824 build(deps): update dependencies (#9866) 2024-12-10 14:33:47 +01:00
Jakob Borg
529d3ef764 ci: reduce frequency of dependabot nags 2024-12-10 09:53:46 +01:00
Jakob Borg
fefbf4dcc8 build: run release flows on tag pushes 2024-12-09 08:32:15 +01:00
André Colomb
b9c6d3ae09 fix(config): skip GUI port probing for UNIX sockets (fixes #9855) (#9858)
When creating an initial default config, we usually probe for a free
TCP port.  But when a UNIX socket is specified via the `STGUIADDRESS=`
override or the `--gui-address=unix:///...` command line syntax, parsing
that option will fail during port probing.

The solution is to just skip the port probing when the address is
determined to specify something other than a TCP socket.

### Testing

Start with a fresh home directory each time.
1. Specify a UNIX socket for the GUI (works with this PR):

TMPHOME=$(mktemp -d); ./syncthing --home=$TMPHOME
--gui-address=unix://$TMPHOME/socket

2. Specify no GUI address (probes for a free port if default is taken,
   as before):

       TMPHOME=$(mktemp -d); ./syncthing --home=$TMPHOME

3. Specify a TCP GUI address (probes whether the given port is taken,
   as before):

TMPHOME=$(mktemp -d); ./syncthing --home=$TMPHOME
--gui-address=127.0.0.1:8385
2024-12-09 07:24:42 +00:00
Syncthing Release Automation
7bea8c758a chore(gui, man, authors): update docs, translations, and contributors 2024-12-09 03:50:45 +00:00
Alex Ionescu
479c0d3f16 fix(gui): reflect folder password visibility by changing button icon (#9857)
Make the "toggle password visibility" button on encrypted
folders change its icon from `fa-eye` to `fa-eye-slash` while the
password is visible.
2024-12-08 21:20:31 +01:00
Jakob Borg
d9ce7c3166 build: build all the things (#9845)
Build packages for stdiscosrv and strelaysrv as well.
2024-12-05 10:09:33 -05:00
Jakob Borg
69979996d9 build(infra): also push Docker images to ghcr.io 2024-12-03 07:54:06 -05:00
Jakob Borg
da58e5c50c build(deps): update dependencies (#9852) 2024-12-03 12:48:10 +00:00
Syncthing Release Automation
44e259142f chore(gui, man, authors): update docs, translations, and contributors 2024-12-02 03:50:30 +00:00
Jakob Borg
77970d5113 refactor: use modern Protobuf encoder (#9817)
At a high level, this is what I've done and why:

- I'm moving the protobuf generation for the `protocol`, `discovery` and
`db` packages to the modern alternatives, and using `buf` to generate
because it's nice and simple.
- After trying various approaches on how to integrate the new types with
the existing code, I opted for splitting off our own data model types
from the on-the-wire generated types. This means we can have a
`FileInfo` type with nicer ergonomics and lots of methods, while the
protobuf generated type stays clean and close to the wire protocol. It
does mean copying between the two when required, which certainly adds a
small amount of inefficiency. If we want to walk this back in the future
and use the raw generated type throughout, that's possible, this however
makes the refactor smaller (!) as it doesn't change everything about the
type for everyone at the same time.
- I have simply removed in cold blood a significant number of old
database migrations. These depended on previous generations of generated
messages of various kinds and were annoying to support in the new
fashion. The oldest supported database version now is the one from
Syncthing 1.9.0 from Sep 7, 2020.
- I changed config structs to be regular manually defined structs.

For the sake of discussion, some things I tried that turned out not to
work...

### Embedding / wrapping

Embedding the protobuf generated structs in our existing types as a data
container and keeping our methods and stuff:

```
package protocol

type FileInfo struct {
  *generated.FileInfo
}
```

This generates a lot of problems because the internal shape of the
generated struct is quite different (different names, different types,
more pointers), because initializing it doesn't work like you'd expect
(i.e., you end up with an embedded nil pointer and a panic), and because
the types of child types don't get wrapped. That is, even if we also
have a similar wrapper around a `Vector`, that's not the type you get
when accessing `someFileInfo.Version`, you get the `*generated.Vector`
that doesn't have methods, etc.

### Aliasing

```
package protocol

type FileInfo = generated.FileInfo
```

Doesn't help because you can't attach methods to it, plus all the above.

### Generating the types into the target package like we do now and
attaching methods

This fails because of the different shape of the generated type (as in
the embedding case above) plus the generated struct already has a bunch
of methods that we can't necessarily override properly (like `String()`
and a bunch of getters).

### Methods to functions

I considered just moving all the methods we attach to functions in a
specific package, so that for example

```
package protocol

func (f FileInfo) Equal(other FileInfo) bool
```

would become

```
package fileinfos

func Equal(a, b *generated.FileInfo) bool
```

and this would mostly work, but becomes quite verbose and cumbersome,
and somewhat limits discoverability (you can't see what methods are
available on the type in auto completions, etc). In the end I did this
in some cases, like in the database layer where a lot of things like
`func (fv *FileVersion) IsEmpty() bool` becomes `func fvIsEmpty(fv
*generated.FileVersion)` because they were anyway just internal methods.

Fixes #8247
2024-12-01 16:50:17 +01:00
André Colomb
2b8ee4c7a5 chore(gui): cache input type in each advanced settings category (#9802)
Each section in the advanced settings dialog has similar code to insert
repeated input fields for each option. But only the first section (GUI
options) was adjusted in #9743 to avoid calling the `inputTypeFor()`
function repeatedly.

Apply the same caching to a locally scoped variable for each ng-repeat
entry by defining it in an ng-init directive.
2024-12-01 12:30:05 +00:00
bt90
be952e5f2d chore(config): add Chinese STUN servers (#9843) 2024-11-30 08:33:55 +01:00
Jakob Borg
43ebac4242 fix(model): create fileset under lock (#9840)
I came accross this in another context and didn't investigate fully, but
literally ten lines above this code, in another method, we say that
filesets _must_ be created under the lock. It's either one or the other
and I'm taking the safer route here.

---------

Co-authored-by: Simon Frei <freisim93@gmail.com>
2024-11-28 13:41:44 +00:00
Jakob Borg
f08a0ed01c build(deps): update dependencies (#9833) 2024-11-25 07:25:24 +00:00
Jakob Borg
612fdff377 build: automatically update APT repository on release
This uses https://github.com/kastelo/ezapt to generate and sign the
archive, and uploads it to blob storage.
2024-11-24 22:55:12 +01:00
Jakob Borg
8ccb7f1924 fix(protocol): allow encrypted-to-encrypted connections again
Encrypted-to-encrypted connections (i.e., ones where both sides set a
password) used to work but were broken in the 1.28.0 release. The
culprit is the 5342bec1b refactor which slightly changed how the request
was constructed, resulting in a bad block hash field.

Co-authored-by: Simon Frei <freisim93@gmail.com>
2024-11-24 22:55:12 +01:00
André Colomb
65d0ca8aa9 fix(config): respect GUI address override in fresh default config (fixes #9783) (#9675)
### Purpose

When generating a new `config.xml` file with default options, the GUI
address is populated with a hard-coded default value of
`127.0.0.1:8384`, except for a random free port if that default one is
occupied. This is independent from the GUI configuration default address
defined in the protobuf description. More importantly, it ignores any
`STGUIADDRESS` override given via environment variable or command-line
option, thus probing for the default port instead of the one specified
via override.

The `ProbeFreePorts()` function now respects the override, by reading
the `GUIConfiguration.Address()` method instead of using hard-coded
defaults.

When not calling `ProbeFreePorts()`, the override should still be
persisted rather than the default address. This happens only when
generating a fresh default `config.xml`, never on an existing one.
2024-11-19 11:01:43 +00:00
Jakob Borg
e82ed6e3d3 style: gofumpt all the things (#9829)
Literally `gofumpt -w .` from the top level dir. Guaranteed to be minor
style changes only and nothing else.

@imsodin per request?
2024-11-19 11:32:56 +01:00
Syncthing Release Automation
4b815fc086 chore(gui, man, authors): update docs, translations, and contributors 2024-11-18 03:49:35 +00:00
Jakob Borg
7eaf843de2 chore(api): add block and goroutine profiles to support bundle (#9824) 2024-11-16 09:43:17 +01:00
Jakob Borg
110e1ae6f9 fix(model): don't panic in index consistency print (fixes #9821) (#9823)
We try to compare to the last fileinfo, but apparently we can end up
here with an empty file list and crash on out of index.
2024-11-14 19:59:34 +00:00
Jakob Borg
896f9725ec chore: update policy to allow approvals by contributors (#9818)
This adds `allow_contributor: true` which allows approvals by
contributors to the PR (but still not the author themself, which is a
different thing). This allows things like pushing minor fixups while
also approving.

The `ignore_update_merges: true` option makes it so that someone is not
considered a "contributor" just because they push the merge button to
update the branch. In principle this is not needed given the above, but
I like it for clarity.
2024-11-12 10:50:19 +01:00
Tobias Frölich
1a529e9d5d fix(gui): expand tildes for subdir check (fixes #9400) (#9788)
### Purpose

This closes #9400 by always expanding tildes when parent/subdir checks
are done.

### Testing

I tested this by creating folders with paths to parent or subdirectories
of the default folder that include a tilde in their path as shown in the
attached screenshots.
With this change, overlap will be detected regardless of wether or not
tildes are used in other folder paths.

### Screenshots

Default Folder:

![2024-10-26-At-08h40m33s](https://github.com/user-attachments/assets/07df090c-4481-41ec-b741-d2785fc848d5)
Newly created folder (parent directory in this case)

![2024-10-26-At-08h40m13s](https://github.com/user-attachments/assets/636fa1fd-41dc-44d9-ac90-0a4937c9921c)

---------

Signed-off-by: tobifroe <froeltob@pm.me>
2024-11-12 09:24:00 +01:00
Hireworks
36ef17df8f fix(model): check if remote folder state before pulling files (fixes #9686) (#9732)
### Purpose

As discussed in #9686 
Syncthing currently does not check folderstate on remote device before
pulling. If no devices have a valid folderstate (i.e all devices have
the folder paused) it will still attempt to pull. On large folders this
will cause a hanging "Syncing" status.

This checks whether at least one connected device has the file available
and has a valid folderstate.

### Testing
Tested locally on multiple devices.
We're new to Go (all our stuff is Python) so please bear with!
Interested if there may be a better place to slot this in.

Thanks,
Jon

---------

Co-authored-by: Simon Frei <freisim93@gmail.com>
2024-11-12 08:51:52 +01:00
Syncthing Release Automation
955ac7775e chore(gui, man, authors): update docs, translations, and contributors 2024-11-11 03:46:06 +00:00
tomasz1986
8f69e874c4 chore(gui): group logout/restart/shutdown buttons together in Actions menu (#9801)
Currently, the "Restart" and "Shutdown" buttons are displayed in the
middle of the Actions menu. On the other hand, the "Log Out" button is
displayed at the very bottom. However, in other cases, e.g. the menus in
operating systems like Windows or macOS, these kind of buttons are
usually grouped together.

Therefore, move the "Restart" and "Shutdown" buttons down, so that they
are listed together with the "Log Out" button. Also, change the order,
so that it goes from the least impactful ("Log Out") to the most
impactful ("Shutdown").

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>

### Screenshots

#### Before


![image](https://github.com/user-attachments/assets/a51438ef-bb6f-4535-a972-8c1bc1dffa02)

#### After


![image](https://github.com/user-attachments/assets/535762d6-6f26-44ab-a402-db87bdcbfb36)

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-11-07 16:47:00 +01:00
Syncthing Release Automation
ac06fd97e9 chore(gui, man, authors): update docs, translations, and contributors 2024-11-04 03:47:49 +00:00
Syncthing Release Automation
3726b7d112 chore(gui, man, authors): update docs, translations, and contributors 2024-10-28 03:48:15 +00:00
Ross Smith II
377200591e fix(fs): fix directory junction handling (fixes #9775) (#9786)
### Purpose

This fixes #9775. I also improved the comments as they were lacking.

My apologies for introducing this bug. In summary, the bug was
```
mode = mode ^ (ModeSymlink | ModeIrregular)
```
didn't correctly reset those bits. This correctly resets them:
```
mode = mode &^ ModeSymlink &^ ModeIrregular
```
Tested and working in Windows 11 version 10.0.22631.4317. I didn't test
in other versions, but I'm sure this is the only issue.
2024-10-27 16:08:38 +01:00
Kapil Sareen
4afc898c2f fix(model): don't sync symbolic links on Android (fixes #9725) (#9782) 2024-10-26 09:29:38 +00:00
Simon Frei
ff7e4fef55 chore(nat, upnp): Make failure logging less reptitive (ref #9324) (#9785)
Currently we log on every single one of 10 retries deep in the upnp
stack. However we also return the failure as an error, which is bubbled
up a while until it's logged at debug level. Switch that around, such
that the repeat logging happens at debug level but the top-level happens
at info. There's some chance that this will newly log errors from
nat-pmp that were previously hidden in debug level - I hope those are
useful and not too numerous.

Also potentially this can even close #9324, my (very limited)
understanding of the reports/discussion there is that there's likely no
problem with syncthing beyond the excessive logging, it's some weird
router behaviour.
2024-10-25 21:04:22 +00:00
Simon Frei
9ffddb1923 fix(gui): apply small screen CSS changes earlier (fixes #9590) (#9756)
These CSS overrides address issues that are already present on wider
screens, so apply it there. Some experiments show we might even want to
up the limit more, but I am chicken and lazy, so I propose to use the
existing 470px media block.

Supersedes another PR after not getting any reaction to feedback there:
https://github.com/syncthing/syncthing/pull/9591#issuecomment-2212586134

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2024-10-25 22:49:22 +02:00
André Colomb
896b857fc4 fix(gui): hide lists with [object Object] in advanced settings (#9743)
As discussed in
https://github.com/syncthing/syncthing/pull/9175#discussion_r1730431703,
entries in advanced settings are unusable if they are comprised of a
list of objects. It just displays `[object Object], [object Object],
[object Object]`, e.g. for the devices a folder is shared with.

Filter out these config elements by detecting an array whose members are
not all strings or numbers, and setting them to `skip` type.

Fix some unnecessary repetition in calling `inputTypeFor()`, since it is
already cached in the `ng-init` directive.
2024-10-25 22:22:12 +02:00
Terrance
acc5d2675b fix(gui): add dark scheme styles for disabled checkboxes (fixes #9776) (#9777)
### Purpose

Fixes #9776 by tweaking the text/background colours of disabled checkbox
panels when dark mode is enabled.

It was [noted on that
issue](https://github.com/syncthing/syncthing/issues/9776#issuecomment-2424828520)
that there's a bigger issue around the correctness of using the
`disabled` attribute on a `<div>` in the first place, but this PR does
not attempt to change that.

### Testing

I've hooked up the GUI files against a release build as suggested below.

### Screenshots

Using the dark theme, or the default theme with a system dark scheme:


![image](https://github.com/user-attachments/assets/3c6bfa77-cc7a-4f3e-a5c2-83daf54dcc34)

Using the black theme:


![image](https://github.com/user-attachments/assets/768db657-aa52-4db0-8455-5194a00fc143)

These borrow the colours from dark theme text inputs and black theme
tabs for a consistent look (initially I tried the text colour of
disabled text inputs, but that produced some poor contrast).
2024-10-22 14:03:32 +02:00
Syncthing Release Automation
6ece4c1fd2 chore(gui, man, authors): update docs, translations, and contributors 2024-10-21 03:47:52 +00:00
Jakob Borg
cc09f0170d build(deps): update dependencies (#9773) 2024-10-17 13:05:53 +00:00
Syncthing Release Automation
bb234d6c0e chore(gui, man, authors): update docs, translations, and contributors 2024-10-14 03:47:49 +00:00
Syncthing Release Automation
e6acc64758 chore(gui, man, authors): update docs, translations, and contributors 2024-10-07 03:47:13 +00:00
André Colomb
f18cf545b9 fix(gui): improve device ID readability in black and dark themes (fixes #9757) (#9758) 2024-10-06 00:48:19 +02:00
Jakob Borg
6d64daaba3 chore(db): process "unchanged" files anyway (#9755)
Skipping these makes the sequence numbering inconcistent; we've received
a file and suppsedly added it to the database, but if you check the
sequence number afterwards it didn't increase, i.e., we trigger [this
failure
condition](47f48faed7/lib/model/indexhandler.go (L447-L459))
and, similarly, a future update will look like there was a hole in the
numbering.

I propose to at least temporarily remove this optimisation in order for
things to make more sense. Is there a reason to keep this beyond saving
some database operations?
2024-10-04 19:47:57 +00:00
Jakob Borg
47f48faed7 fix(upgrades): avoid clobbering cache when filtering (#9752)
The slice is shared, can't overwrite elements of it. (Upgrade server
only thing.)
2024-10-02 18:56:39 +00:00
Jakob Borg
cfa834177b build(deps): update dependencies (#9751) 2024-10-02 12:53:14 +00:00
tomasz1986
c454fc8baa chore(build): use conventional commit title in update script (#9747) 2024-09-30 15:59:14 -05:00
Jakob Borg
dbe7fa9155 Merge branch 'infrastructure'
* infrastructure:
  feat(ursrv): new metrics based approach
2024-09-30 14:17:02 -05:00
Jakob Borg
4d842f7d3b feat(ursrv): new metrics based approach 2024-09-30 14:16:27 -05:00
Syncthing Release Automation
0e68221c91 gui, man, authors: Update docs, translations, and contributors 2024-09-30 03:48:06 +00:00
Jakob Borg
19f63c7ea3 chore(model): improve tracking sentPrevSeq for index debugging (#9740) 2024-09-29 22:18:24 +00:00
Emil Lundberg
fb939ec496 fix(model): prevent division by zero in numHashers (#9744)
This should prevent the panic that occurred in this test run:
https://github.com/syncthing/syncthing/actions/runs/11095876010/job/30825046810

```
2024-09-29T21:01:53.5425372Z === RUN   TestIssue4357
2024-09-29T21:01:53.5505943Z panic: runtime error: integer divide by zero [recovered]
2024-09-29T21:01:53.5512200Z 	panic: runtime error: integer divide by zero
2024-09-29T21:01:53.5516633Z
2024-09-29T21:01:53.5523018Z goroutine 2655 [running]:
2024-09-29T21:01:53.5524157Z github.com/thejerf/suture/v4.(*Supervisor).runService.func2.2()
2024-09-29T21:01:53.5527176Z 	/home/runner/go/pkg/mod/github.com/thejerf/suture/v4@v4.0.5/supervisor.go:563 +0xd0
2024-09-29T21:01:53.5530556Z panic({0x1080d20?, 0x1851290?})
2024-09-29T21:01:53.5564723Z 	/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.1.linux-amd64/src/runtime/panic.go:785 +0x132
2024-09-29T21:01:53.5566616Z github.com/syncthing/syncthing/lib/model.(*model).numHashers(0xc0006f6180, {0x117dc1a, 0x7})
2024-09-29T21:01:53.5568061Z 	/home/runner/work/syncthing/syncthing/lib/model/model.go:2581 +0x210
2024-09-29T21:01:53.5569912Z github.com/syncthing/syncthing/lib/model.(*folder).scanSubdirsChangedAndNew(0xc00c38c808, {0x0, 0x0, 0x0}, 0xc0003fc060)
2024-09-29T21:01:53.5571612Z 	/home/runner/work/syncthing/syncthing/lib/model/folder.go:653 +0x250
2024-09-29T21:01:53.5573010Z github.com/syncthing/syncthing/lib/model.(*folder).scanSubdirs(0xc00c38c808, {0x0, 0x0, 0x0})
2024-09-29T21:01:53.5574447Z 	/home/runner/work/syncthing/syncthing/lib/model/folder.go:512 +0xd0f
2024-09-29T21:01:53.5576011Z github.com/syncthing/syncthing/lib/model.(*folder).scanTimerFired(0xc00c38c808)
2024-09-29T21:01:53.5577367Z 	/home/runner/work/syncthing/syncthing/lib/model/folder.go:916 +0x46
2024-09-29T21:01:53.5579010Z github.com/syncthing/syncthing/lib/model.(*folder).Serve(0xc00c38c808, {0x1307650, 0xc0006a0910})
2024-09-29T21:01:53.5580428Z 	/home/runner/work/syncthing/syncthing/lib/model/folder.go:205 +0xd7e
2024-09-29T21:01:53.5581624Z github.com/thejerf/suture/v4.(*Supervisor).runService.func2()
2024-09-29T21:01:53.5582978Z 	/home/runner/go/pkg/mod/github.com/thejerf/suture/v4@v4.0.5/supervisor.go:567 +0x249
2024-09-29T21:01:53.5584400Z created by github.com/thejerf/suture/v4.(*Supervisor).runService in goroutine 2651
2024-09-29T21:01:53.5585872Z 	/home/runner/go/pkg/mod/github.com/thejerf/suture/v4@v4.0.5/supervisor.go:541 +0x32a
2024-09-29T21:01:53.5661413Z FAIL	github.com/syncthing/syncthing/lib/model	5.510s
```

### Testing

I have not been able to reproduce the panic throughout a few minutes of
continuously running the test without this fix, but judging by the
traceback it seems to only happen if the test happens to delete the
folder from config at the same time `scanTimerFired` triggers.
2024-09-30 00:01:52 +02:00
Jakob Borg
39df3173d4 chore(model): log sequence anomaly when update appears not to "take" (#9741)
I hope this doesn't fire, but 👻  I'm Seeing Things I Can't Explain 👻
2024-09-29 15:04:06 +00:00
maxice8
429672e0b4 docs(docker): add healthcheck to docker-compose (#9742)
### Purpose

Syncthing had a healthcheck API for a while, and the example Dockerfile
for it has it in the form of:

HEALTHCHECK --interval=1m --timeout=10s \
CMD curl -fkLsS -m 2 127.0.0.1:8384/rest/noauth/health | grep -o
--color=never OK || exit 1

Let's add it to the docker-compose as well

### Testing

I use this docker-compose.yml file to deploy via ansible (using
community.docker.docker_compose_v2) to my machine with success, using
`wait: true` in ansible for it to use `docker compose up --wait`.

```yml
- name: Enable syncthing docker
  community.docker.docker_compose_v2:
    project_src: /srv/syncthing
    wait: true
    wait_timeout: 90
```
2024-09-29 09:53:13 -05:00
Simon Frei
605fd6d726 fix(ignore): ensure normalization of patterns and paths match (fixes #9597) (#9717)
In ignores, normalize the input when parsing it.
When scanning, normalize earlier such that the path is already
normalized when checking ignores. This requires splitting normalization
of the string from normalization of the file, as we don't want to
attempt the latter if the file is ignored.

Closes #9598

---------

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2024-09-28 17:16:44 +02:00
Jakob Borg
3c476542d2 fix(ur): actually send usage report directly when enabled (#9736)
There was a bug that the unique ID was not set when reporting was
enabled, and thus the reports where rejected by the server. The unique
ID got set only on startup, so next time Syncthing restarted.

This makes sure to set the unique ID when blank.
2024-09-28 17:02:05 +02:00
Jakob Borg
31874f3ebb chore(model): remove GUI/log warning on sequence anomaly (#9738)
I can see already in our Sentry data that there are a fair amount of
these warnings, and mostly the shape of it. Asking users to report them
will likely cause a lot of reporting effort to fairly little additional
value. We can do that when/if we have something more targeted to ask
for.
2024-09-28 16:38:07 +02:00
Jakob Borg
77942747db Merge branch 'infrastructure'
* infrastructure:
  feat(stupgrades): filter returned releases per compatibility
2024-09-26 10:25:02 +02:00
Jakob Borg
fe01b396ba feat(stupgrades): filter returned releases per compatibility 2024-09-26 10:22:23 +02:00
Jakob Borg
3583949706 refactor(upgrade): rename insecureGet which is no longer insecure (#9735) 2024-09-25 15:50:22 +00:00
Jakob Borg
23fc22ebc5 chore: add more advanced policy configuration (#9726)
This codifies a review policy which is closer to what I always
envisioned, but which isn't expressible using the normal checks in the
GitHub GUI. It would move the commit approval check from GitHub into the
policy-bot check which is already present to enforce the
conventional-commits standard. Approvals in general would still work the
same -- it's just that the bot picks it up and toggles the status
accordingly. From a GitHub side when this is enabled we'd remove the
requires-review check from there and let the bot decide that part. We
would still require builds and tests to pass of course.

There are a couple of relexations from the current policy, details in
the code but briefly:

- Changes to translations or dependencies by a trusted person don't
require review
- Trivial changes by a trusted person, explicitly marked as such, don't
require review

This enables less bureaucracy for things like adding new translated
languages and updating dependencies, and enables the trivial-change
workflow to a larger audience than, like, me, who could always just
bypass the rules by way of being admin.
2024-09-25 17:41:56 +02:00
Jakob Borg
cba163a1fd chore: enable TLS client cache for HTTPS where appropriate (#9721)
https://forum.syncthing.net/t/infrastructure-report-discovery-stuff/22819/4
2024-09-24 08:55:04 +02:00
Jakob Borg
a8e2c8edb6 fix(connections): announce PtP links again (fixes #9730) (#9731) 2024-09-23 14:32:19 +02:00
Syncthing Release Automation
3e501d9036 gui, man, authors: Update docs, translations, and contributors 2024-09-23 03:46:01 +00:00
bt90
9ca101756d chore(ursrv): add Nix detection (#9729)
Classify the builder `nix@nix` as [Nix](https://nixos.org/)

![369684243-172cab09-df6f-449a-a638-1f0a0c080ab3](https://github.com/user-attachments/assets/37a6e0a5-bdcb-4b31-8b36-eaaa42423382)
2024-09-22 14:03:40 +02:00
bt90
a873d12c65 chore(ursrv): extend F-Droid detection (#9728)
Our f-droid apps are currently built using `vagrant@bookworm`:

![grafik](https://github.com/user-attachments/assets/172cab09-df6f-449a-a638-1f0a0c080ab3)
2024-09-22 13:48:38 +02:00
Emil Lundberg
8ff670c564 fix(gui): get version from header when not authenticated (#9724)
### Purpose

Since #8757, the Syncthing GUI now has an unauthenticated state. One
consequence of this is that `$scope.versionBase()` is not initialized
while unauthenticated, which causes the `docsURL` function to truncate
links to just `https://docs.syncthing.net`/, discarding the section
path. This currently affects at least the "Help > Introduction" link
reachable both while logged in and not. The issue is exacerbated in
https://github.com/syncthing/syncthing/pull/9175 where we sometimes want
to show additional contextual help links from the login page to
particular sections of the docs.

I don't think it's any worse to try to preserve the section path even
without an explicit version tag, than to fall back to just the host and
lose all context the link was attempting to provide.

### Testing

- On commit b1ed2802fb (before):
  - Open the GUI, set a username and log out.
- Open the "Help" drop-down. The "Introduction" item links to:
https://docs.syncthing.net/
  - Log in.
- Open the "Help" drop-down. The "Introduction" item links to:
https://docs.syncthing.net/v1.27.10/intro/gui
- On commit 44fef31780 (after):
  - Open the GUI, set a username and log out.
- Open the "Help" drop-down. The "Introduction" item links to:
https://docs.syncthing.net/intro/gui
  - Log in.
- Open the "Help" drop-down. The "Introduction" item links to:
https://docs.syncthing.net/v1.27.10/intro/gui

### Screenshots

This is a GUI change, but affecting only URLs in the markup with no
visual changes.


### Drawbacks

If a `docsURL` call generates a versionless link to a docs page that
doesn't exist on https://docs.syncthing.net - presumably because
Syncthing is not the latest version and links to a deleted page? - then
this will lead to GitHub's generic 404 page with no link to the
Syncthing docs root. Before, any versionless link would also be a
pathless link, leading to the Syncthing docs root instead of a 404 page.
2024-09-22 09:47:02 +02:00
Jakob Borg
b1ed2802fb fix(connections): skip point-to-point interfaces when listing LANs (fixes #9719) (#9720)
Point-to-point interfaces are typically VPNs and similar which, for our
purposes, do not qualify as LANs.
2024-09-21 09:27:23 +02:00
Jakob Borg
b70cb580c8 build(deps): update all dependencies (#9723) 2024-09-21 09:25:27 +02:00
Sonu Kumar Saw
28be3ba788 chore(connections): lower log level from INFO to DEBUG for "already connected to this device" messages (fixes #9715) (#9722)
### Purpose

The primary aim of this change is to minimize log clutter in production
environments. There are many lines in the logs coming from an expected
race condition when two devices connect `already connected to this
device`. These messages do not indicate errors and can overwhelm the log
files with unnecessary noise.

By lowering the logging level, we enhance the usability of the logs,
making it easier for users and developers to identify actual issues
without being distracted

### Testing
1. Build syncthing locally
2. Start two Syncthing instances
```bash
./syncthing -no-browser -home=~/.config/syncthing1
./syncthing -no-browser -home=~/.config/syncthing2
```
3. Enable the DEBUG logs from UI for `connections` package
4. Connect the synching instances by adding remote devices from the UI
5. Observe the logs for the message `XXXX already connected to this
device`

### Screenshots


![image](https://github.com/user-attachments/assets/882ccb4c-d39d-463a-8f66-2aad97010700)

## Authorship

Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
2024-09-21 07:19:58 +00:00
Jakob Borg
d4770ddc77 chore(cmd): clean up commands (#9705)
Move infrastructure related commands to under `cmd/infra` and
development stuff to `cmd/dev`. The default build command builds the
regular user facing binaries: syncthing, stdiscosrv, and strelaysrv.
2024-09-21 09:04:22 +02:00
Simon Frei
cbe1220680 chore(fs): put the caseFS as the outermost layer again (#9716)
Reasoning in comments. The main motivation is to avoid all the case
checks when walking the filesystem.

"again" as we already tried once, but it caused a major issue ragarding
mtimefs layer. The root of this problem has been fixed in the meantime
in ac8b3342a
2024-09-18 20:31:19 +02:00
André Colomb
0b95c5fa76 fix(meta): return read error in forbidden_words_test (#9706)
When reading a file fails, the error is currently swallowed / hidden.
Probably just a typo.
2024-09-18 19:12:24 +02:00
Jakob Borg
0343bca257 Merge branch 'infrastructure'
* infrastructure:
  chore(stdiscosrv): ensure incoming addresses are sorted and unique
  chore(stdiscosrv): use zero-allocation merge in the common case
  chore(stdiscosrv): properly clean out old addresses from memory
  chore(stdiscosrv): calculate IPv6 GUA
2024-09-16 09:33:15 +02:00
Syncthing Release Automation
878016db39 gui, man, authors: Update docs, translations, and contributors 2024-09-16 03:47:01 +00:00
Simon Frei
1f4fde9525 chore(protocol): prioritize closing a connection (#9711)
The read/write loops may keep going for a while on a closing connection
with lots of read/write activity, as it's random which select case is
chosen. And if the connection is slow or even broken, a single
read/write
can take a long time/until timeout. Add initial non-blocking selects
with only the cases relevant to closing, to prioritize those.
2024-09-15 21:13:56 +02:00
Jakob Borg
5b9d8a838f chore(stdiscosrv): ensure incoming addresses are sorted and unique 2024-09-15 17:01:16 +02:00
Jakob Borg
8b19cb1e11 chore(stdiscosrv): use zero-allocation merge in the common case 2024-09-15 15:26:40 +02:00
Jakob Borg
ce1e259bb4 chore(stdiscosrv): properly clean out old addresses from memory 2024-09-15 14:20:59 +02:00
Jakob Borg
2238a288d9 fix(model): shut down index sender faster (#9704) 2024-09-15 11:37:49 +02:00
Jakob Borg
c8ee2a5cf6 chore(stdiscosrv): calculate IPv6 GUA 2024-09-15 10:48:16 +02:00
Ross Smith II
1704827d04 chore(gui): update HumanDuration.js (#9710)
Relevant changes:

ko: Use correct names for month and hour in Korean (465eaed)
Hide unit count if 2 in Arabic (f90d847)
2024-09-15 10:21:18 +02:00
Jakob Borg
a156e88eef Merge branch 'infrastructure'
* infrastructure:
  chore(stdiscosrv): hide internal/undocumented flags
  chore(stdiscosrv): remove legacy replication
  chore(stdiscosrv): clean up s3 handling
  chore(stdiscosrv): less garbage in statistics
  chore(stdiscosrv): improve expire, logging
  chore(stdiscosrv): sched in loop
  chore(stdiscosrv): database writing logging
  chore(stdiscosrv): use order-preserving expire
  chore(stdiscosrv): simplify sorting
  chore(stdiscosrv): reduce allocations in cert handling
  chore(stdiscosrv): reduce unnecessary allocations in merge
  feat(stdiscosrv): enable HTTP profiler
  feat(discosrv): in-memory storage with S3 backing
  feat(stdiscosrv): make compression optional (and faster)
2024-09-13 08:50:24 +02:00
Jakob Borg
94d0195b63 chore(stdiscosrv): hide internal/undocumented flags 2024-09-13 08:49:13 +02:00
Jakob Borg
1616edcee3 chore(stdiscosrv): remove legacy replication 2024-09-13 08:49:13 +02:00
Jakob Borg
6505e123bb chore(stdiscosrv): clean up s3 handling 2024-09-13 08:48:04 +02:00
Jakob Borg
63e4659282 chore(stdiscosrv): less garbage in statistics 2024-09-13 08:48:04 +02:00
Jakob Borg
f3f5557c8e chore(stdiscosrv): improve expire, logging 2024-09-13 08:48:04 +02:00
Jakob Borg
b794726e1f chore(stdiscosrv): sched in loop 2024-09-13 08:48:04 +02:00
Jakob Borg
3d59740a0a chore(stdiscosrv): database writing logging 2024-09-13 08:48:04 +02:00
Jakob Borg
66fb65b01f chore(stdiscosrv): use order-preserving expire 2024-09-13 08:48:04 +02:00
Jakob Borg
5c2fcbfd19 chore(stdiscosrv): simplify sorting 2024-09-13 08:48:03 +02:00
Jakob Borg
f9b72330a8 chore(stdiscosrv): reduce allocations in cert handling 2024-09-13 08:48:03 +02:00
Jakob Borg
822b6ac36b chore(stdiscosrv): reduce unnecessary allocations in merge 2024-09-13 08:48:03 +02:00
Jakob Borg
77f7778292 feat(stdiscosrv): enable HTTP profiler 2024-09-13 08:48:03 +02:00
Jakob Borg
aed2c66e52 feat(discosrv): in-memory storage with S3 backing 2024-09-13 08:48:03 +02:00
Jakob Borg
68a1fd010f feat(stdiscosrv): make compression optional (and faster) 2024-09-13 08:33:03 +02:00
Simon Frei
ac8b3342ac chore(fs): only cache the cache for case FS, not the entire FS (#9701)
This would have addressed a recent issue that arose when re-ordering our
"filesystem layers". Specifically moving the caseFilesystem to the
outermost layer. The previous cache included the filesystem, and as such
all the layers below. This isn't desirable (to put it mildly), as you
can create different variants of filesystems with different layers for
the same path and options. Concretely this did happen with the mtime
layer, which isn't always present. A test for the mtime related breakage
was added in #9687, and I intend to redo the caseFilesystem reordering
after this.

Ref: #9677
Followup to: #9687
2024-09-12 20:35:21 +02:00
Jakob Borg
0ea90dd932 build: add generating compat.json (#9700)
This is to add the generation of `compat.json` as a release artifact. It
describes the runtime requirements of the release in question. The next
step is to have the upgrade server use this information to filter
releases provided to clients. This is per the discussion in #9656

---------

Co-authored-by: Ross Smith II <ross@smithii.com>
2024-09-11 09:29:49 +02:00
Jakob Borg
718b1ce2b7 chore(discovery,upgrade): use regular TLS certificate verification (#9673)
This changes the two remaining instances where we use insecure HTTPS to
use standard HTTPS certificate verification.

When we introduced these things, almost a decade ago, HTTPS certificates
were expensive and annoying to get, much of the web was still HTTP, and
many devices seemed to not have up-to-date CA bundles.

Nowadays _all_ of the web is HTTPS and I'm skeptical that any device can
work well without understanding LetsEncrypt certificates in particular.

Our current discovery servers use hardcoded certificates which has
several issues:
- Not great for security if it leaks as there is no way to rotate it
- Not great for infrastructure flexibility as we can't use many load
balancer or TLS termination services
- The certificate is a very oddball ECDSA-SHA384 type certificate which
has higher CPU cost than a more regular certificate, which has real
effects on our infrastructure

Using normal TLS certificates here improves these things.

I expect there will be some very few devices out there for which this
doesn't work. For the foreseeable future they can simply change the
config to use the old URLs and parameters -- it'll be years before we
can retire those entirely.

For the upgrade client this simply seems like better hygiene. While our
releases are signed anyway, protecting the metadata exchange is _better_
and, again, I doubt many clients will fail this today.
2024-09-11 09:29:19 +02:00
Simon Frei
29f7510f5a lib/fs: Add test reproducing missing mtimefs issue (ref #9677) (#9687)
The test is quite odd and specific, but it does reproduce the issue that
caused #9677, so I'd propose to add it to have a simple regression test
for the basic scenario. Also the option to the fakefs might come handy
for other scenarios where you want to quickly test some behaviour on a
filesystem without nanosecond precision, without actually needing access
to one.
2024-09-10 13:36:17 +02:00
Syncthing Release Automation
a7f9ed4a80 gui, man, authors: Update docs, translations, and contributors 2024-09-09 03:45:15 +00:00
tomasz1986
1baefea410 gui: Actually filter scope ID out of IPv6 address when using Remote GUI (ref #8084) (#9688)
gui: Actually filter scope ID out of IPv6 address when using Remote GUI
(ref #8084)

The current code does not work, because it uses a string in the
replace() method instead of regex. Thus, change it to a proper regex.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-09-08 12:42:32 +02:00
Jakob Borg
563cec8923 Merge branch 'release'
* release:
  Revert "lib/fs: Put the caseFS as the outermost layer (#9648)"
2024-09-06 09:39:09 +02:00
Jakob Borg
a3c340ece9 Revert "lib/fs: Put the caseFS as the outermost layer (#9648)"
This reverts commit 7517d18fbb.

Fixes #9677
2024-09-06 09:15:45 +02:00
André Colomb
cb24638ec9 lib/api: Correct ordering of Accept-Language codes by weight (fixes #9670) (#9671)
The preference for languages in the Accept-Language header field
should not be deduced from the listed order, but from the passed
"quality values", according to the HTTP specification:
https://httpwg.org/specs/rfc9110.html#field.accept-language

This implements the parsing of q=values and ordering within the API
backend, to not complicate things further in the GUI code.  Entries
with invalid (unparseable) quality values are discarded completely.

* gui: Fix API endpoint in comment.
2024-09-02 10:15:04 +02:00
Syncthing Release Automation
2fb24dc2cc gui, man, authors: Update docs, translations, and contributors 2024-09-02 03:45:15 +00:00
tomasz1986
9aa2d2c92f gui: Fix incorrect UI language auto detection (fixes #9668) (#9669)
gui: Fix incorrect UI language auto detection (fixes #9668)

Currently, the code only checks whether the detected language partially
matches one of the available languages. This means that if the detected
language is "fi" but among the available languages there is only "fil"
and no "fi", then it will match "fi" with "fil", even though the two are
completely different languages.

With this change, the matching is only done when there is a hyphen in
the language code, e.g. "en" will match with "en-US".

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-08-31 22:47:10 +02:00
Jakob Borg
d1c5100c98 Merge branch 'release'
* release:
  lib/upgrade: Send OS version header to upgrade server (#9663)
2024-08-30 11:32:49 +02:00
Jakob Borg
42e677c055 lib/model, lib/protocol: Index sending/receiving debugging (#9657)
This adds guardrails to the index sending and receiving, to verify that
what we thinks is happening is what actually happens.
2024-08-28 15:00:19 +02:00
Jakob Borg
27bba2c0c2 lib/upgrade: Send OS version header to upgrade server (#9663)
This adds a header with the operating system version, verbatim in
whatever format the operating system reports it, to the upgrade check.
The intention is that the upgrade server can use this information to
filter out (or maybe just mark) potentially unsupported upgrades.
2024-08-28 08:32:03 +02:00
Jakob Borg
feff334547 lib/upgrade: Send OS version header to upgrade server (#9663)
This adds a header with the operating system version, verbatim in
whatever format the operating system reports it, to the upgrade check.
The intention is that the upgrade server can use this information to
filter out (or maybe just mark) potentially unsupported upgrades.
2024-08-28 08:31:10 +02:00
Syncthing Release Automation
713cf357ce gui, man, authors: Update docs, translations, and contributors 2024-08-26 03:45:23 +00:00
Jakob Borg
5342bec1b7 lib/protocol: Further interface refactor (#9396)
This is a symmetric change to #9375 -- where that PR changed the
protocol->model interface, this changes the model->protocol one.
2024-08-24 12:45:10 +02:00
Emil Lundberg
7df75e681d gui: Replace global "Panel padding decrease" style with targeted class (#9659)
Transplanted from https://github.com/emlun/syncthing/pull/8 (meta-PR
into https://github.com/syncthing/syncthing/pull/9175) by request of
@acolomb (see:
https://github.com/emlun/syncthing/pull/8#discussion_r1724470574).

This padding decrease currently applies to _all_ collapsible panels, but
this padding decrease may not be appropriate for all collapsible panels.
In particular, it will not be appropriate for the collapsible panels
introduced in https://github.com/emlun/syncthing/pull/8.
2024-08-21 15:02:45 +02:00
Jakob Borg
8dc826b234 build: use Go 1.23, require minimum 1.22 (#9651)
🥳

---------

Co-authored-by: Ross Smith II <ross@smithii.com>
2024-08-19 20:26:08 +02:00
Syncthing Release Automation
9ef37e1485 gui, man, authors: Update docs, translations, and contributors 2024-08-19 03:45:18 +00:00
Simon Frei
7517d18fbb lib/fs: Put the caseFS as the outermost layer (#9648)
Reasoning in comments. The main motivation is to avoid all the case
checks when walking the filesystem.
2024-08-13 10:59:31 +02:00
André Colomb
42d0fee536 gui: Add Irish (ga) translation template (#9646)
Based on user request from Weblate, user `@aindriu80`.

Looks promising based on the profile:
https://hosted.weblate.org/user/aindriu80/ Not sure whether almost
30.000 translations in about one month is realistic for a human though.
2024-08-12 13:32:29 +02:00
Syncthing Release Automation
2ca9d3b5c5 gui, man, authors: Update docs, translations, and contributors 2024-08-12 03:45:16 +00:00
Tommy van der Vorst
9cde068f2a lib/syncthing: Add wrapper for access to model (#9627)
### Purpose

Wrap access to Model for users that use the syncthing Go package. See
discussion:
https://github.com/syncthing/syncthing/pull/9619#pullrequestreview-2212484910

### Testing

It works with the iOS app. Other than that, there are no current users
of this API (to my knowledge) as Model was only exposed recently form
the iOS app.
2024-08-11 20:20:43 +02:00
Gusted
1243083831 cli: Remove go-shlex dependency (#9644) 2024-08-11 11:37:18 +02:00
Gusted
356c5055ad lib/sha256: Remove it (#9643)
### Purpose

Remove the `lib/sha256` package, because it's no longer necessary. Go's
standard library now has the same performance and is on par with
`sha256-simd` since [Since Go
1.21](1a64574f42).
Therefore using `sha256-simd` has no benefits anymore.

ARM already has optimized sha256 assembly code since
7b8a7f8272,
`sha256-simd` published their results before that optimized assembly was
implemented,
f941fedda8.
The assembly looks very similar and the benchmarks in the Go commit
match that of `sha256-simd`.

This patch removes all of the related code of `lib/sha256` and makes
`crypto/sha256` the 'default'.

Benchmark of `sha256-simd` and `crypto/sha256`:
<details>

```
cpu: AMD Ryzen 5 3600X 6-Core Processor
                │  simd.txt   │               go.txt                │
                │   sec/op    │    sec/op     vs base               │
Hash/8Bytes-12    63.25n ± 1%    73.38n ± 1%  +16.02% (p=0.002 n=6)
Hash/64Bytes-12   98.73n ± 1%   105.30n ± 1%   +6.65% (p=0.002 n=6)
Hash/1K-12        567.2n ± 1%    572.8n ± 1%   +0.99% (p=0.002 n=6)
Hash/8K-12        4.062µ ± 1%    4.062µ ± 1%        ~ (p=0.396 n=6)
Hash/1M-12        512.1µ ± 0%    510.6µ ± 1%        ~ (p=0.485 n=6)
Hash/5M-12        2.556m ± 1%    2.564m ± 0%        ~ (p=0.093 n=6)
Hash/10M-12       5.112m ± 0%    5.127m ± 0%        ~ (p=0.093 n=6)
geomean           13.82µ         14.27µ        +3.28%

                │   simd.txt   │               go.txt                │
                │     B/s      │     B/s       vs base               │
Hash/8Bytes-12    120.6Mi ± 1%   104.0Mi ± 1%  -13.81% (p=0.002 n=6)
Hash/64Bytes-12   618.2Mi ± 1%   579.8Mi ± 1%   -6.22% (p=0.002 n=6)
Hash/1K-12        1.682Gi ± 1%   1.665Gi ± 1%   -0.98% (p=0.002 n=6)
Hash/8K-12        1.878Gi ± 1%   1.878Gi ± 1%        ~ (p=0.310 n=6)
Hash/1M-12        1.907Gi ± 0%   1.913Gi ± 1%        ~ (p=0.485 n=6)
Hash/5M-12        1.911Gi ± 1%   1.904Gi ± 0%        ~ (p=0.093 n=6)
Hash/10M-12       1.910Gi ± 0%   1.905Gi ± 0%        ~ (p=0.093 n=6)
geomean           1.066Gi        1.032Gi        -3.18%
```

</details>


### Testing

Compiled and tested on Linux.

### Documentation

https://github.com/syncthing/docs/pull/874
2024-08-10 12:58:20 +01:00
Jakob Borg
19693734a3 build: Update dependencies (#9640) 2024-08-09 16:04:51 +02:00
Ross Smith II
17e60b9e0c Chmod -x non-executable files (fixes #9629) (#9630)
Fixed via
```
git ls-files -s | grep 100755 | grep -E -v '(run|sh)$' | cut -f 2 | xargs git update-index --chmod -x
```
See #9629.
2024-08-05 18:32:23 +02:00
Syncthing Release Automation
ac22b2d00a gui, man, authors: Update docs, translations, and contributors 2024-08-05 03:45:25 +00:00
Tommy van der Vorst
de0b4270df all: minimal set of changes for iOS app (#9619)
### Purpose

This PR contains the set of changes needed to make Syncthing work on iOS
for [my iOS app for
Syncthing](https://github.com/pixelspark/sushitrain).

Most changes originate from [the Mobius Sync
fork](http://github.com/MobiusSync/syncthing/tree/ios). I have removed
the changes from their fork that are not strictly needed for my app
(i.e. their changes to the GUI and command line utilities, for instance)
and squashed it all in a single commit.

In summary, the changes are:

* Resolve non-absolute paths to the 'Documents' folder (basically the
only one an app can/should write user data to by default on iOS)
* Tweaking of build flags/conditions for iOS (i.e. determine which
basicfs_watch, ignoreresult variant to build for iOS)
* Disable upgrade mechanism on iOS
* Make `RequestGlobal` and `PullerProgress` public symbols
* Expose syncthing.app's Model instance (app.M)
* Add no-op stub for SetLowPriority on iOS

I would very much appreciate these changes to be (eventually) merged to
mainline syncthing, as this would allow my iOS app to track the mainline
source code directly and removes the need (for me at least) for
maintaining a separate fork. Perhaps the Mobius folks can also benefit
from this (although as noted this branch does not contain their changes
to e.g. the GUI).

### Testing

This branch has been tested with the iOS app and appears to work fine.
The full set of MobiusSync changes has been used before with success.

### Screenshots

n/a

### Documentation

There should be no visible changes for users due to this set of changes.

---------

Co-authored-by: Simon Pickup <simon@pickupinfinity.com>
2024-07-31 07:31:14 +02:00
Syncthing Release Automation
e738af7c56 gui, man, authors: Update docs, translations, and contributors 2024-07-29 03:45:25 +00:00
Syncthing Release Automation
a28441a9bf gui, man, authors: Update docs, translations, and contributors 2024-07-22 03:45:28 +00:00
Simon Frei
2e313716e5 etc: Remove restart on suspend systemd service (ref #8448) (#9611)
The option to do the same in our own monitor process has been removed a
long time ago.
2024-07-17 09:29:49 +03:00
Syncthing Release Automation
0b5ff1f5f7 gui, man, authors: Update docs, translations, and contributors 2024-07-15 03:45:20 +00:00
Simon Frei
0fe6d97d3d lib/fs: Add missing locks to fakeFile methods (fixes #9499) (#9603)
fixes #9499
2024-07-09 10:33:30 +02:00
Simon Frei
0756e42a85 lib/api: Increase test request timeout (fixes #9455) (#9602)
Fixes #9455
2024-07-09 00:37:44 +02:00
Syncthing Release Automation
13ebe1c87f gui, man, authors: Update docs, translations, and contributors 2024-07-08 03:45:16 +00:00
Simon Frei
aea7fa5f22 lib/ignore: Remove unused patterns in cache (#9601)
Tiny cleanup I noticed while trying to fix/test another issue
(https://github.com/syncthing/syncthing/pull/9600). I shortly tried to
figure out what it was used for in the past, but gave up without
results.
2024-07-02 11:01:00 +00:00
Simon Frei
403ce7e597 lib/ignore: Fix caching of filenames with path separators on windows (#9600)
Previously we queried cache with backslashes, and stored entries with
slashes. As in no cache hits ever for non-toplevel files. I also
eventually remembered that cache is disabled by default, so this is a
bit pointless, but still right :P
2024-07-02 10:58:06 +00:00
Syncthing Release Automation
4704d3bc48 gui, man, authors: Update docs, translations, and contributors 2024-07-01 03:45:16 +00:00
André Colomb
2794b04243 gui: Add Filipino (fil) translation template (#9599) 2024-06-29 17:53:32 +02:00
Syncthing Release Automation
1ce64971fd gui, man, authors: Update docs, translations, and contributors 2024-06-24 03:45:42 +00:00
Syncthing Release Automation
eb6d80eac4 gui, man, authors: Update docs, translations, and contributors 2024-06-17 03:45:27 +00:00
Syncthing Release Automation
a8db3351ae gui, man, authors: Update docs, translations, and contributors 2024-06-10 03:45:25 +00:00
Ross Smith II
23a900e096 gui: Use localised time in duration (#9552)
https://github.com/syncthing/syncthing/pull/8291 inpired me to develop
this. I tested it with all the languages Syncthing currently supports,
and they all work.

The only issue is that when you change the language in the GUI, you have
to either refresh the page, or wait a few seconds for the page to
refresh by itself, before the duration is translated into the new
language.

### Screenshots


![2024-05-20_21-47-58](https://github.com/syncthing/syncthing/assets/220772/7e3b371e-3495-4e3e-853a-b5a41215e6c7)

### Documentation

The documentation for the translation widget is at
https://github.com/EvanHahn/HumanizeDuration.js/blob/main/README.md
2024-06-05 06:05:51 -04:00
Jakob Borg
5a304cf295 build: Skip autoprocs for build script 2024-06-04 08:55:37 -04:00
Jakob Borg
136b3742bf build: Update dependencies (#9565) 2024-06-04 13:58:49 +02:00
Jakob Borg
21e0f98fe2 Merge branch 'infrastructure'
* infrastructure:
  cmd/stupgrades: Basic process metrics
  cmd/stcrashreceiver: Ignore patterns, improve metrics
  cmd/strelaypoolsrv: More compact response, improved metrics
  cmd/stdiscosrv: Add AMQP replication
2024-06-04 07:18:35 -04:00
Jakob Borg
2bb5b2244b cmd/stupgrades: Basic process metrics 2024-06-03 19:50:28 +02:00
Jakob Borg
2f281799c1 cmd/stcrashreceiver: Ignore patterns, improve metrics 2024-06-03 19:50:28 +02:00
Jakob Borg
18a58a2ddc cmd/strelaypoolsrv: More compact response, improved metrics 2024-06-03 19:50:28 +02:00
Jakob Borg
f283215fce cmd/stdiscosrv: Add AMQP replication 2024-06-03 19:50:28 +02:00
Syncthing Release Automation
495809ac9e gui, man, authors: Update docs, translations, and contributors 2024-06-03 03:45:18 +00:00
Jakob Borg
9ca8addcf7 build: Add missing region attribute for uploads 2024-05-30 10:49:22 +02:00
Jakob Borg
94181ade23 build: Generalise S3 push options 2024-05-27 13:56:32 +02:00
Syncthing Release Automation
e50933433e gui, man, authors: Update docs, translations, and contributors 2024-05-27 03:45:21 +00:00
Jakob Borg
a2b8f2361e lib/config: Add file inside folder marker directory (#9525)
### Purpose

Avoid the issue where the folder marker is deleted by overzealous
cleanup tools because it's just a useless, empty directory.

We create a small file containing a an admonishment to not delete the
directory, and some metadata that is just for human consumption at the
moment. (But it would parse as a valid yaml file if we wanted to read
this, at some point.)

This will only apply when _creating_ a folder marker, that is, existing
setups will not gain the file automatically. Obviously, when using a
custom folder marker none of this applies.

Also, slightly adjust the permission bits for the folder marker directory and file on Unixes, making sure the group & write bits are unset.

### Testing

I've created and deleted a few folders and it appears to behave as I
expect.

### Screenshots

```
jb@ok:~/somefolder % ls -la
total 0
drwxr-xr-x   3 jb  staff   96 May  1 08:52 ./
drwx------  12 jb  staff  384 May  1 08:52 ../
drwxr-xr-x   3 jb  staff   96 May  1 08:52 .stfolder/
jb@ok:~/somefolder % ls -l .stfolder
total 8
-rw-r--r--  1 jb  staff  122 May  1 08:52 syncthing-folder-39a4b0.txt
jb@ok:~/somefolder % cat .stfolder/syncthing-folder-39a4b0.txt
# This directory is a Syncthing folder marker.
# Do not delete.

folderID: xtdca-cudyf
created: 2024-05-01T08:52:49+02:00
```
2024-05-24 08:51:02 +02:00
Jakob Borg
4b60e86d02 lib/config, lib/watchaggregator: Add config for max FS watcher delay (#9558)
Currently the maximum delay is always derived automatically from the
initial delay. This is fine in most cases, but for some use cases (large
files that take a long time to write) we need to be able to set a longer
max delay than the computed value (e.g., 15s delay with 10min timeout).
2024-05-23 16:21:00 +02:00
Jakob Borg
d6b5676603 lib/fs: Watcher should react to xattr-only events on Darwin 2024-05-23 09:39:10 +02:00
Jakob Borg
3821b6ceee build: Update dependencies (#9553) 2024-05-21 12:37:43 +02:00
Syncthing Release Automation
973585e97d gui, man, authors: Update docs, translations, and contributors 2024-05-20 03:45:15 +00:00
Jakob Borg
ba6ac2f604 lib/geoip, cmd/relaypoolsrv, cmd/ursrv: Automatically manage GeoIP updates (#9342)
This adds a small package `geoip` which knows how to download and manage
the Maxmind GeoLite2 database we use. This removes the need for various
scripts to download and manage the geoip database, something that today
happens on Docker startup for the relay pool server and using various
hand written hacks for the usage reporting server.

The database is downloaded when needed and then refreshed on a
best-effort basis weekly.
2024-05-18 20:31:49 +03:00
luchenhan
57d399317e lib/db: Correct function name in comments (#9520) 2024-05-16 07:02:57 +00:00
WangXi
f2d6722348 lib/connections: Use proper errors.Is check (#9538) 2024-05-16 07:01:16 +00:00
dependabot[bot]
7b1b77e50d build(deps): bump github.com/quic-go/quic-go from 0.42.0 to 0.43.0 (#9522)
Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go)
from 0.42.0 to 0.43.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/quic-go/quic-go/releases">github.com/quic-go/quic-go's
releases</a>.</em></p>
<blockquote>
<h2>v0.43.0</h2>
<h2><em>quic-go.net</em>: Launching a new Documentation Site</h2>
<p>With this release, we're launching a new documentation site for the
quic-go projects (quic-go itself, HTTP/3, webtransport-go, and soon,
masque-go): <a href="https://quic-go.net">quic-go.net</a>.</p>
<p>The documentation site aims to explain QUIC concepts and how they are
made accessible using quic-go's API. This site replaces the wiki, and
the ever-growing README files.</p>
<p>A lot of work has gone into the documentation already, but we're by
no means done yet. The entire source is public in <a
href="https://github.com/quic-go/docs/">https://github.com/quic-go/docs/</a>,
and we're happy about community contributions.</p>
<h2>HTTP Datagrams (RFC 9297)</h2>
<p>This release adds support for HTTP Datagrams (<a
href="https://datatracker.ietf.org/doc/html/rfc9297">RFC 9297</a>), both
on the client and on the server side (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4452">#4452</a>).
HTTP Datagrams are used in WebTransport in CONNECT-UDP (<a
href="https://datatracker.ietf.org/doc/html/rfc9298">RFC 9298</a>),
among others.</p>
<p>The new API for HTTP Datagrams is described on the new documentation
page: <a href="https://quic-go.net/docs/http3/datagrams/">HTTP
Datagrams</a>. The integration of HTTP Datagram support necessitated a
comprehensive refactor of the HTTP/3 package, resulting in several
breaking API changes listed below.</p>
<h2>Breaking Changes</h2>
<ul>
<li>quicvarint: functions now return an <code>int</code> instead the
internal <code>protocol.ByteCount</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4365">#4365</a>)</li>
<li>http3: <code>Server.SetQuicHeaders</code> was renamed to
<code>SetQUICHeaders</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4377">#4377</a>)</li>
<li>http3: <code>Server.QuicConfig</code> was renamed to
<code>QUICConfig</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4384">#4384</a>)</li>
<li>http3: <code>RoundTripper.QuicConfig</code> was renamed to
<code>QUICConfig</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4385">#4385</a>)</li>
<li>http3: <code>RoundTripOpt.CheckSettings</code> was removed (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4416">#4416</a>).
Use the new<code>SingleDestinationRoundTripper</code> API instead.</li>
<li>http3: the <code>HTTPStreamer</code> interface is now implemented by
the <code>http.ResponseWriter</code> (and not the
<code>http.Request.Body</code>) (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4469">#4469</a>)</li>
<li>include the maximum payload size in the
<code>DatagramTooLargeError</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4470">#4470</a>)</li>
</ul>
<h2>Other Notable Changes</h2>
<ul>
<li>GSO and ECN is disabled on kernel versions older than 5 (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4456">#4456</a>)</li>
<li>http3: logging can be controlled using an <code>slog.Logger</code>
(<a
href="https://redirect.github.com/quic-go/quic-go/issues/4449">#4449</a>)</li>
<li>http3: HEAD requests can now be sent in 0-RTT (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4378">#4378</a>)</li>
<li>http3: duplicate QPACK encoder and decoder streams are not rejected
as required by the RFC (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4388">#4388</a>)</li>
<li>http3: Extended CONNECT are blocked until the server's SETTINGS are
received, as required by the RFC (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4450">#4450</a>)</li>
<li>http3: HTTP/3 client connections aren't removed if
<code>RoundTrip</code> errors due to a cancelled context (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4448">#4448</a>).
Thanks to <a
href="https://github.com/GeorgeMac"><code>@​GeorgeMac</code></a>!</li>
<li>http3: sniff Content-Type when flushing the ResponseWriter (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4412">#4412</a>).
Thanks to <a
href="https://github.com/WeidiDeng"><code>@​WeidiDeng</code></a>!</li>
<li>The <code>Context</code> exposed on the <code>quic.Stream</code> is
now derived from the connection's context (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4414">#4414</a>)</li>
<li>The UDP send and receive buffer size was increased to 7 MiB (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4455">#4455</a>).
Thanks to <a
href="https://github.com/bt90"><code>@​bt90</code></a>!</li>
</ul>
<h2>Clarifications on the QUIC Stream State Machine</h2>
<h3>Calling CancelWrite after Close</h3>
<p>After a long and fruitful discussion (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4404">#4404</a>),
we decided to clarify that calling <code>CancelWrite</code> after
<code>Close</code> on a <code>SendStream</code> (or a bidirectional
stream) should cause a state transition from the &quot;Data Sent&quot;
to the &quot;Reset Sent&quot; state, as described in <a
href="https://datatracker.ietf.org/doc/html/rfc9000#section-3.1">section
3.1 of RFC 9000</a>. This matches the current behavior of quic-go,
however, it didn't match the API documentation (fixed in <a
href="https://redirect.github.com/quic-go/quic-go/issues/4419">#4419</a>).</p>
<p>This means that stream data will not be delivered reliably if
<code>CancelWrite</code> is called, and that this applies even if
<code>Close</code> was called before.</p>
<h3>Garbage Collection of Streams</h3>
<p>This release also changes the way streams are garbage-collected (and
the peer is granted additional limit to open a new stream), once they're
not needed anymore, in a subtle way:</p>
<ul>
<li>for the send direction of streams: <a
href="https://redirect.github.com/quic-go/quic-go/issues/4445">#4445</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="93c4785521"><code>93c4785</code></a>
http3: sniff Content-Type when flushing the ResponseWriter (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4412">#4412</a>)</li>
<li><a
href="c0250ce824"><code>c0250ce</code></a>
include the maximum payload size in the DatagramTooLargeError (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4470">#4470</a>)</li>
<li><a
href="34f4d1443f"><code>34f4d14</code></a>
http3: implement on the HTTPStreamer on the ResponseWriter, flush header
(<a
href="https://redirect.github.com/quic-go/quic-go/issues/4469">#4469</a>)</li>
<li><a
href="083ceb42f2"><code>083ceb4</code></a>
http3: rename Settings.EnableDatagram to EnableDatagrams (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4466">#4466</a>)</li>
<li><a
href="e1e5b6294d"><code>e1e5b62</code></a>
README: link to the new documentation site (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4464">#4464</a>)</li>
<li><a
href="2a37c53143"><code>2a37c53</code></a>
http3: add support for HTTP Datagrams (RFC 9297) (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4452">#4452</a>)</li>
<li><a
href="11b11594b2"><code>11b1159</code></a>
http3: fix race condition in client unit test (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4463">#4463</a>)</li>
<li><a
href="4b87539b1e"><code>4b87539</code></a>
delay completion of the receive stream until the reset error was read
(<a
href="https://redirect.github.com/quic-go/quic-go/issues/4460">#4460</a>)</li>
<li><a
href="bff131e546"><code>bff131e</code></a>
delay completion of the send stream until the reset error was delivered
(<a
href="https://redirect.github.com/quic-go/quic-go/issues/4445">#4445</a>)</li>
<li><a
href="12aa63824c"><code>12aa638</code></a>
disable GSO and ECN on kernels older than version 5 (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4456">#4456</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/quic-go/quic-go/compare/v0.42.0...v0.43.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/quic-go/quic-go&package-manager=go_modules&previous-version=0.42.0&new-version=0.43.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-16 08:56:44 +02:00
Syncthing Release Automation
06914b872b gui, man, authors: Update docs, translations, and contributors 2024-05-13 03:45:18 +00:00
Jakob Borg
f6df8b40b4 build: Use Go 1.22.3 at minimum 2024-05-08 08:01:46 +02:00
André Colomb
e057f5ee9a gui: Add Hindi (hi) translation template (#9530)
Based on user request from Weblate, user @Scrambled777.

Looks promising based on the profile:
https://hosted.weblate.org/user/Scrambled777/
2024-05-07 10:29:42 +02:00
Syncthing Release Automation
a5bf110d90 gui, man, authors: Update docs, translations, and contributors 2024-05-06 03:45:24 +00:00
DerRockWolf
debbe726e0 lib/connections: Add syncthing_connections_active metric (fixes #9527) (#9528)
### Purpose

Adds a new metric `syncthing_connections_active` which equals to the
amount of active connections per device.

Fixes #9527 

<!--
Describe the purpose of this change. If there is an existing issue that
is
resolved by this pull request, ensure that the commit subject is on the
form
`Some short description (fixes #1234)` where 1234 is the issue number.
-->

### Testing

I've manually tested it by running syncthing with these changes locally
and examining the returned metrics from `/metrics`.
I've done the following things:
- Connect & disconnect a device
- Increase & decrease the number of connections and verify that the
value of the metric matches with the amount displayed in the GUI.

### Documentation

https://github.com/syncthing/docs/blob/main/includes/metrics-list.rst
needs to be regenerated with
[find-metrics.go](https://github.com/syncthing/docs/blob/main/_script/find-metrics/find-metrics.go)

## Authorship

Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.

---------

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2024-05-04 22:31:37 +02:00
otbutz
ec3e474a53 etc: Use 7MiB buffer size (#9524)
### Purpose

In preparation for quic-go v0.43.0. see
https://github.com/quic-go/quic-go/pull/4455
2024-05-01 10:03:07 +02:00
Severin von Wnuck-Lipinski
ebb1edc652 gui: Fix Firefox bookmark favicon (fixes #9506) (#9507)
### Purpose

Firefox uses the last specified favicon link for bookmarks, but only if
it is available on initial page load.
Remove the second link and use ng-href to change the icon instead.

I'm not really familiar with AngularJS, feel free to offer suggestions
for improvements.

### Testing

Briefly tested on Firefox 124.0.2 and Chrome 123.0.6312.105.
2024-05-01 09:00:36 +02:00
Syncthing Release Automation
6204670c66 gui, man, authors: Update docs, translations, and contributors 2024-04-29 03:45:25 +00:00
Syncthing Release Automation
ff9b24f388 gui, man, authors: Update docs, translations, and contributors 2024-04-22 03:45:18 +00:00
Syncthing Release Automation
01b820dc78 gui, man, authors: Update docs, translations, and contributors 2024-04-15 04:10:05 +00:00
Jakob Borg
79ae24df76 lib/nat: Don't crash on empty address list (fixes #9503) (#9504) 2024-04-11 13:23:29 +02:00
Jakob Borg
61b94b9ea5 lib/db: Drop indexes for outgoing data to force refresh (ref #9496) (#9502)
### Purpose

Resend our indexes since we fixed that index-sending issue.

I made a new thing to only drop the non-local-device index IDs, i.e.,
those for other devices. This means we will see a mismatch and resend
all indexes, but they will not. This is somewhat cleaner as it avoids
resending everything twice when two devices are upgraded, and in any
case, we have no reason to force a resend of incoming indexes here.

### Testing

It happens on my computer...
2024-04-08 11:14:27 +02:00
tomasz1986
6fb3c5ccf2 gui: Fix missing link to device editor for names with superscript (ref #9472) (#9494)
The commit 7e4e65ebf5 added links to
devices listed in the Shared With list in the folder info. However, it
only added them to those that had no superscript next to them.

With this change, the links are added to all devices regardless of
whether they have the superscript next to their names or not. The commit
also simplifies the code by using anchors directly instead of wrapping
them in spans.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-04-07 21:37:31 +02:00
Jakob Borg
2e7c03420f lib/db: Hold update lock while taking snapshot (#9496) 2024-04-05 21:32:43 +02:00
Jakob Borg
faa56b4bb7 build: Update dependencies (#9497) 2024-04-05 16:13:20 +02:00
Syncthing Release Automation
d7ba5316b8 gui, man, authors: Update docs, translations, and contributors 2024-04-01 03:45:41 +00:00
Syncthing Release Automation
bdfd0f0548 gui, man, authors: Update docs, translations, and contributors 2024-03-25 03:45:14 +00:00
Tim Nordenfur
5d27185083 Removed no longer relevant Bountysource link (#9480)
### Purpose

Bountysource no longer exists and the readme link 404s. This removes the
Bountysource link and corresponding readme section.

Perhaps the section should instead be replaced by other instructions for
voting on features.
2024-03-23 00:23:02 +01:00
Jakob Borg
4dfb9d7c83 lib/api: Missing return after HTTP error 2024-03-21 08:57:43 -04:00
Emil Lundberg
2f15670094 lib/api: Extract session store (#9425)
This is an extract from PR #9175, which can be reviewed in isolation to
reduce the volume of changes to review all at once in #9175. There are
about to be several services and API handlers that read and set cookies
and session state, so this abstraction will prove helpful.

In particular a motivating cause for this is that with the current
architecture in PR #9175, in `api.go` the [`webauthnService` needs to
access the
session](https://github.com/syncthing/syncthing/pull/9175/files#diff-e2e14f22d818b8e635572ef0ee7718dee875c365e07225d760a6faae8be7772dR309-R310)
for authentication purposes but needs to be instantiated before the
`configMuxBuilder` for config purposes, because the WebAuthn additions
to config management need to perform WebAuthn registration ceremonies,
but currently the session management is embedded in the
`basicAuthAndSessionMiddleware` which is [instantiated much
later](https://github.com/syncthing/syncthing/pull/9175/files#diff-e2e14f22d818b8e635572ef0ee7718dee875c365e07225d760a6faae8be7772dL371-R380)
and only if authentication is enabled in `guiCfg`. This refactorization
extracts the session management out from `basicAuthAndSessionMiddleware`
so that `basicAuthAndSessionMiddleware` and `webauthnService` can both
use the same shared session management service to perform session
management logic.

### Testing

This is a refactorization intended to not change any externally
observable behaviour, so existing tests (e.g., `api_auth_test.go`)
should cover this where appropriate. I have manually verified that:

- Appending `+ "foo"` to the cookie name in `createSession` causes
`TestHtmlFormLogin/invalid_URL_returns_403_before_auth_and_404_after_auth`
and `TestHtmlFormLogin/UTF-8_auth_works` to fail
- Inverting the return value of `hasValidSession` cases a whole bunch of
tests in `TestHTTPLogin` and `TestHtmlFormLogin` to fail
- (Fixed) Changing the cookie to `MaxAge: 1000` in `destroySession` does
NOT cause any tests to fail!
- Added tests `TestHtmlFormLogin/Logout_removes_the_session_cookie`,
`TestHTTPLogin/*/Logout_removes_the_session_cookie`,
`TestHtmlFormLogin/Session_cookie_is_invalid_after_logout` and
`TestHTTPLogin/200_path#01/Session_cookie_is_invalid_after_logout` to
cover this.
- Manually verified that these tests pass both before and after the
changes in this PR, and that changing the cookie to `MaxAge: 1000` or
not calling `m.tokens.Delete(cookie.Value)` in `destroySession` makes
the respective pair of tests fail.
2024-03-21 08:09:47 -04:00
Syncthing Release Automation
b49137ce36 gui, man, authors: Update docs, translations, and contributors 2024-03-18 03:45:22 +00:00
Jaspitta
7e4e65ebf5 gui: Open devices on click in Shared With list in folder info (fixes #8972) (#9472) 2024-03-17 20:13:09 +01:00
Simon Frei
8c8167a4ab lib/model: Don't bump seq on error in index handler (#9459) 2024-03-11 07:30:21 +01:00
Simon Frei
73cc5553b6 lib/model: Prevent infinite index sending loop (fixes #9407) (#9458)
Explanation of what/why in a code comment.

Fixes https://github.com/syncthing/syncthing/issues/9407
2024-03-10 22:28:40 +01:00
Simon Frei
2ab2488274 lib/scanner: Fix ticker leak in scanner (fixes #9417) (#9451)
Move the ticker closer to where it's used and defer stop it to avoid
missing a branch.

Fixes regression introduced in
2f3eacdb6c

Fixes https://github.com/syncthing/syncthing/issues/9417
2024-03-05 19:04:26 +01:00
Jakob Borg
eb9cd363d0 build: Update dependencies (#9448) 2024-03-04 20:39:43 +01:00
Syncthing Release Automation
7fe3906534 gui, man, authors: Update docs, translations, and contributors 2024-03-04 03:54:27 +00:00
André Colomb
5fdab1bf11 gui: Show encryption status for devices sharing folder (ref #8808) (#9355)
This re-implements the stalled enhancement from #8808. Thanks @Craeckie
for the idea and first implementation draft!

If a folder is shared to a device with encryption, add a lock icon in
front of the device name under "Shared With" in the folder details
panel. Be careful not to add whitespace caused by line wraps in HTML
source code, which would defeat the purpose of keeping the icon glued to
the name by a non-breaking space.

Apply the same lock icon for the list of folders shared with a device.
2024-03-03 21:09:57 +01:00
André Colomb
13a6d43f0b gui: Fix wrapping in "Shared With" / "Folders" lists. (#9438)
This change was split off from #9355 as an independent clean-up / fix.
See that PR for review discussion, testing, and screenshots.

Improve the wrapping of folder labels / device names by going back to
word-wrapping, but making sure other spans, such as the trailing comma,
do not get separated from the label span.

* Avoid adding whitespace caused by line wraps in HTML source code.

The different cases within the ng-switch block are separated by
newlines for readability, but that gets parsed as whitespace.  For
wrapping purposes, this should not happen, because then there is no
way to keep other HTML parts glued to the name / label in each list
entry.

* Simplify redundant conditional comma code.

The separating comma after a device name or folder label (all but the
last) should always stick to it.  Use the HTML comment trick to avoid
whitespace and therefore a wrapping opportunity caused by the code
formatting newline.  Thus the conditional comma only needs to be
defined once, not in each ng-switch case.

* Wrap at word boundaries and only break up words if necessary.

Use the overflow-wrap: break-word; style instead of word-break:
break-all;.  While the latter is suitable for longish paths, breaking
device names or folder labels arbitrarily within words is ugly.

This also makes the the <sup> numbers actually stay glued to their
respective neighboring words.

Include legacy CSS alias "word-wrap" in the class definition.

* Fix indentation (unrelated).
2024-03-03 20:55:09 +01:00
Jakob Borg
ac942e2481 github: Convert issue templates into forms (fixes #9442) 2024-03-02 16:27:57 +01:00
Luke Hamburg
bbd2a7fbc5 lib/model: Ignore difference in extended attributes & ownership when deleting (fixes #9371) (#9430)
Adds a bool flag to `scanIfItemChanged()` to indicate when the scan was initiated from a delete function, and if so, tell `IsEquivalentOptional()` to ignore Xattrs and Ownership regardless of the global setting.

I tested this with my sledgehammer and it seems to pass.
2024-03-02 14:55:18 +00:00
Jakob Borg
07a9fa2dbd all: Use own automaxprocs package that doesn't log (ref #9436) (#9437)
### Purpose

🤫
2024-02-27 13:05:19 +01:00
Thomas
aa559bf496 all: Use Linux container CPU quota (fixes #9357, fixes #9435) (#9436)
Go is not cgroup aware and by default will set GOMAXPROCS to the number
of available threads, regardless of whether it is within the allocated
quota. This behaviour causes high amount of CPU throttling and degraded
application performance.
2024-02-26 12:23:14 +00:00
Jakob Borg
2d968d46b7 cmd/syncthing: Remove legacy GOMAXPROCS handling (ref #9436) 2024-02-26 13:12:57 +01:00
Syncthing Release Automation
86c4cafc96 gui, man, authors: Update docs, translations, and contributors 2024-02-26 03:45:28 +00:00
Beat Reichenbach
c4dfb66d84 docker: Add support for setting umask (#9429)
Add support for setting umask value in the Docker `entrypoint.sh`
script. This is useful when
not syncing permissions and working with groups, and needing umask
values like `002` instead of `022`.
2024-02-22 08:47:43 +00:00
Syncthing Release Automation
f4d160684b gui, man, authors: Update docs, translations, and contributors 2024-02-19 03:45:23 +00:00
Syncthing Release Automation
b76e6ce70d gui, man, authors: Update docs, translations, and contributors 2024-02-12 03:45:25 +00:00
Jakob Borg
6b4028eede build: Use correct Go version also for script runs (#9414)
The changes to go.mod in latest Go 1.21/1.22 are not fully understood by
older Go that might be pre-installed on builds, so make sure we always
have a modern one in place even for running small release scripts etc.
2024-02-11 09:24:00 +01:00
Jakob Borg
ad81ac8da7 lib/api: Deflake TestAPIServiceRequests (#9413)
Somewhere along the way, the non-parallel test became parallel, and at
that point, timeouts occurred. Parallel is better, so increase the
timeout on the offending call a bit...
2024-02-11 09:20:29 +01:00
Jakob Borg
7ebeaefe77 lib/model: Deflake new IndexHandlerTest (#9412)
The new test has a flakiness factor on slow platforms, where the close
on the sending connection races with the last index message, potentially
messing up the count. This adds a wait to ensure that all sent messages
are received, or the test will eventually fail with a timeout.
2024-02-11 09:03:12 +01:00
Jakob Borg
e1dd36561d all: Use some Go 1.21 features (#9409) 2024-02-10 21:02:42 +01:00
Jakob Borg
96c30f8387 lib/model, lib/protocol: Remove FileInfoBatch reuse behavior (#9399) 2024-02-10 19:16:27 +01:00
Jakob Borg
fc8b353011 build: Use Go 1.22, minimum is Go 1.21 (#9408)
Also updated dependencies, and an adjustment to build tags for how those
are handled and how irrelevant go1.15 is nowadays...
2024-02-09 16:35:29 +01:00
Jakob Borg
416b9e8924 lib/logger: Reduce API surface (#9404)
There is no need to expose the IsTraced() thing; it's just used in
initialisation, and thereafter ShouldDebug() is the corresponding
correct call.
2024-02-09 11:17:44 +01:00
gudvinr
9f6d732587 lib/logger: Split STTRACE into list of strings (#9402)
Currently `IsTraced("xyz")` will return true for
any inclusion of "xyz" in string.

This change splits `STTRACE` using `','`, `' '` and `';'`
as delimiters. That makes facilities separation
more clear.
2024-02-06 14:07:59 +01:00
Syncthing Release Automation
f2f5786b33 gui, man, authors: Update docs, translations, and contributors 2024-02-05 03:45:33 +00:00
Jakob Borg
eb617865d2 lib/model: Typo in method name (fixes #9389) 2024-02-01 15:13:33 +01:00
Jakob Borg
a49e318d25 lib/model: Typo in debug print (fixes #9386) 2024-02-01 15:11:09 +01:00
Jakob Borg
e74674a019 build: Update dependencies (#9379) 2024-01-31 09:11:04 +01:00
dependabot[bot]
d98fa474ae build(deps): bump actions/cache from 3 to 4 (#9363)
Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/cache/releases">actions/cache's
releases</a>.</em></p>
<blockquote>
<h2>v4.0.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Update action to node20 by <a
href="https://github.com/takost"><code>@​takost</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1284">actions/cache#1284</a></li>
<li>feat: save-always flag by <a
href="https://github.com/to-s"><code>@​to-s</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1242">actions/cache#1242</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/takost"><code>@​takost</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1284">actions/cache#1284</a></li>
<li><a href="https://github.com/to-s"><code>@​to-s</code></a> made their
first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1242">actions/cache#1242</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v3...v4.0.0">https://github.com/actions/cache/compare/v3...v4.0.0</a></p>
<h2>v3.3.3</h2>
<h2>What's Changed</h2>
<ul>
<li>Cache v3.3.3 by <a
href="https://github.com/robherley"><code>@​robherley</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1302">actions/cache#1302</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/robherley"><code>@​robherley</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1302">actions/cache#1302</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v3...v3.3.3">https://github.com/actions/cache/compare/v3...v3.3.3</a></p>
<h2>v3.3.2</h2>
<h2>What's Changed</h2>
<ul>
<li>Fixed readme with new segment timeout values by <a
href="https://github.com/kotewar"><code>@​kotewar</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1133">actions/cache#1133</a></li>
<li>Readme fixes by <a
href="https://github.com/kotewar"><code>@​kotewar</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1134">actions/cache#1134</a></li>
<li>Updated description of the lookup-only input for main action by <a
href="https://github.com/kotewar"><code>@​kotewar</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1130">actions/cache#1130</a></li>
<li>Change two new actions mention as quoted text by <a
href="https://github.com/bishal-pdMSFT"><code>@​bishal-pdMSFT</code></a>
in <a
href="https://redirect.github.com/actions/cache/pull/1131">actions/cache#1131</a></li>
<li>Update Cross-OS Caching tips by <a
href="https://github.com/pdotl"><code>@​pdotl</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1122">actions/cache#1122</a></li>
<li>Bazel example (Take <a
href="https://redirect.github.com/actions/cache/issues/2">#2</a>️⃣) by
<a href="https://github.com/vorburger"><code>@​vorburger</code></a> in
<a
href="https://redirect.github.com/actions/cache/pull/1132">actions/cache#1132</a></li>
<li>Remove actions to add new PRs and issues to a project board by <a
href="https://github.com/jorendorff"><code>@​jorendorff</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1187">actions/cache#1187</a></li>
<li>Consume latest toolkit and fix dangling promise bug by <a
href="https://github.com/chkimes"><code>@​chkimes</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1217">actions/cache#1217</a></li>
<li>Bump action version to 3.3.2 by <a
href="https://github.com/bethanyj28"><code>@​bethanyj28</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1236">actions/cache#1236</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/vorburger"><code>@​vorburger</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1132">actions/cache#1132</a></li>
<li><a
href="https://github.com/jorendorff"><code>@​jorendorff</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1187">actions/cache#1187</a></li>
<li><a href="https://github.com/chkimes"><code>@​chkimes</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1217">actions/cache#1217</a></li>
<li><a
href="https://github.com/bethanyj28"><code>@​bethanyj28</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1236">actions/cache#1236</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v3...v3.3.2">https://github.com/actions/cache/compare/v3...v3.3.2</a></p>
<h2>v3.3.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Reduced download segment size to 128 MB and timeout to 10 minutes by
<a href="https://github.com/kotewar"><code>@​kotewar</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1129">actions/cache#1129</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v3...v3.3.1">https://github.com/actions/cache/compare/v3...v3.3.1</a></p>
<h2>v3.3.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Bug: Permission is missing in cache delete example by <a
href="https://github.com/kotokaze"><code>@​kotokaze</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1123">actions/cache#1123</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/actions/cache/blob/main/RELEASES.md">actions/cache's
changelog</a>.</em></p>
<blockquote>
<h1>Releases</h1>
<h3>3.0.0</h3>
<ul>
<li>Updated minimum runner version support from node 12 -&gt; node
16</li>
</ul>
<h3>3.0.1</h3>
<ul>
<li>Added support for caching from GHES 3.5.</li>
<li>Fixed download issue for files &gt; 2GB during restore.</li>
</ul>
<h3>3.0.2</h3>
<ul>
<li>Added support for dynamic cache size cap on GHES.</li>
</ul>
<h3>3.0.3</h3>
<ul>
<li>Fixed avoiding empty cache save when no files are available for
caching. (<a
href="https://redirect.github.com/actions/cache/issues/624">issue</a>)</li>
</ul>
<h3>3.0.4</h3>
<ul>
<li>Fixed tar creation error while trying to create tar with path as
<code>~/</code> home folder on <code>ubuntu-latest</code>. (<a
href="https://redirect.github.com/actions/cache/issues/689">issue</a>)</li>
</ul>
<h3>3.0.5</h3>
<ul>
<li>Removed error handling by consuming actions/cache 3.0 toolkit, Now
cache server error handling will be done by toolkit. (<a
href="https://redirect.github.com/actions/cache/pull/834">PR</a>)</li>
</ul>
<h3>3.0.6</h3>
<ul>
<li>Fixed <a
href="https://redirect.github.com/actions/cache/issues/809">#809</a> -
zstd -d: no such file or directory error</li>
<li>Fixed <a
href="https://redirect.github.com/actions/cache/issues/833">#833</a> -
cache doesn't work with github workspace directory</li>
</ul>
<h3>3.0.7</h3>
<ul>
<li>Fixed <a
href="https://redirect.github.com/actions/cache/issues/810">#810</a> -
download stuck issue. A new timeout is introduced in the download
process to abort the download if it gets stuck and doesn't finish within
an hour.</li>
</ul>
<h3>3.0.8</h3>
<ul>
<li>Fix zstd not working for windows on gnu tar in issues <a
href="https://redirect.github.com/actions/cache/issues/888">#888</a> and
<a
href="https://redirect.github.com/actions/cache/issues/891">#891</a>.</li>
<li>Allowing users to provide a custom timeout as input for aborting
download of a cache segment using an environment variable
<code>SEGMENT_DOWNLOAD_TIMEOUT_MINS</code>. Default is 60 minutes.</li>
</ul>
<h3>3.0.9</h3>
<ul>
<li>Enhanced the warning message for cache unavailablity in case of
GHES.</li>
</ul>
<h3>3.0.10</h3>
<ul>
<li>Fix a bug with sorting inputs.</li>
<li>Update definition for restore-keys in README.md</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="13aacd865c"><code>13aacd8</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/cache/issues/1242">#1242</a>
from to-s/main</li>
<li><a
href="53b35c5439"><code>53b35c5</code></a>
Merge branch 'main' into main</li>
<li><a
href="65b8989fab"><code>65b8989</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/cache/issues/1284">#1284</a>
from takost/update-to-node-20</li>
<li><a
href="d0be34d544"><code>d0be34d</code></a>
Fix dist</li>
<li><a
href="66cf064d47"><code>66cf064</code></a>
Merge branch 'main' into update-to-node-20</li>
<li><a
href="1326563738"><code>1326563</code></a>
Merge branch 'main' into main</li>
<li><a
href="e71876755e"><code>e718767</code></a>
Fix format</li>
<li><a
href="01229828ff"><code>0122982</code></a>
Apply workaround for earlyExit</li>
<li><a
href="3185ecfd61"><code>3185ecf</code></a>
Update &quot;only-&quot; actions to node20</li>
<li><a
href="25618a0a67"><code>25618a0</code></a>
Bump version</li>
<li>Additional commits viewable in <a
href="https://github.com/actions/cache/compare/v3...v4">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/cache&package-manager=github_actions&previous-version=3&new-version=4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-31 08:33:02 +01:00
Jakob Borg
f16817632f lib/api: Improve folder summary event, verbose service (#9370)
This makes a couple of small improvements to the folder summary
mechanism:

- The folder summary includes the local and remote sequence numbers in
clear text, rather than some odd sum that I'm not sure what it was
intended to represent.
- The folder summary event is generated when appropriate, regardless of
whether there is an event listener. We did this before because
generating it was expensive, and we wanted to avoid doing it
unnecessarily. Nowadays, however, it's mostly just reading out
pre-calculated metadata, and anyway, it's nice if it shows up reliably
when running with -verbose.

The point of all this is to make it easier to use these events to judge
when devices are, in fact, in sync. As-is, if I'm looking at two
devices, it's very difficult to reliably determine if they are in sync
or not. The reason is that while we can ask device A if it thinks it's
in sync, we can't see if the answer is "yes" because it has processed
all changes from B, or if it just doesn't know about the changes from B
yet. With proper sequence numbers in the event we can compare the two
and determine the truth. This makes testing a lot easier.
2024-01-31 08:24:39 +01:00
Jakob Borg
bda4016109 lib/protocol: Refactor interface (#9375)
This is a refactor of the protocol/model interface to take the actual
message as the parameter, instead of the broken-out fields:

```diff
type Model interface {
        // An index was received from the peer device
-       Index(conn Connection, folder string, files []FileInfo) error
+       Index(conn Connection, idx *Index) error
        // An index update was received from the peer device
-       IndexUpdate(conn Connection, folder string, files []FileInfo) error
+       IndexUpdate(conn Connection, idxUp *IndexUpdate) error
        // A request was made by the peer device
-       Request(conn Connection, folder, name string, blockNo, size int32, offset int64, hash []byte, weakHash uint32, fromTemporary bool) (RequestResponse, error)
+       Request(conn Connection, req *Request) (RequestResponse, error)
        // A cluster configuration message was received
-       ClusterConfig(conn Connection, config ClusterConfig) error
+       ClusterConfig(conn Connection, config *ClusterConfig) error
        // The peer device closed the connection or an error occurred
        Closed(conn Connection, err error)
        // The peer device sent progress updates for the files it is currently downloading
-       DownloadProgress(conn Connection, folder string, updates []FileDownloadProgressUpdate) error
+       DownloadProgress(conn Connection, p *DownloadProgress) error
 }
```

(and changing the `ClusterConfig` to `*ClusterConfig` for symmetry;
we'll be forced to use all pointers everywhere at some point anyway...)

The reason for this is that I have another thing cooking which is a
small troubleshooting change to check index consistency during transfer.
This required adding a field or two to the index/indexupdate messages,
and plumbing the extra parameters in umpteen changes is almost as big a
diff as this is. I figured let's do it once and avoid having to do that
in the future again...

The rest of the diff falls out of the change above, much of it being in
test code where we run these methods manually...
2024-01-31 08:18:27 +01:00
Syncthing Release Automation
8f5d07bd09 gui, man, authors: Update docs, translations, and contributors 2024-01-29 03:45:37 +00:00
kylosus
302b352d78 lib/fs: Add invalid UTF-8 guards to watcher (fixes #9369) (#9372)
Add invalid UTF-8 guards to fix #9369. Probably not a permanent fix, but
putting it up here in case someone else encounters the same panic.
2024-01-28 19:50:26 +01:00
Jakob Borg
45beb28fa5 lib/api: Remove remnants of CSRF tokens file mentions (ref #9284) 2024-01-23 12:07:58 +01:00
Syncthing Release Automation
ee9b20e47a gui, man, authors: Update docs, translations, and contributors 2024-01-22 03:45:30 +00:00
tomasz1986
0f55d5fc3e gui: Remove non-functional HTML from External Versioning tooltip (ref #8923) (#9358)
gui: Remove non-functional HTML from External Versioning tooltip (ref
#8923)

Since [1], it is no longer possible to use HTML in tooltips. This was
addressed in [2], however the commit missed one instance of HTML that
was used to change the font type of the External versioning command
tooltip. This remaining HTML is removed in this commit.

[1] f5e5af391a
[2] 73c52eafb6

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>

### Screenshots


![image](https://github.com/syncthing/syncthing/assets/5626656/d5f6c553-35cb-48c2-b654-809d8bbe93b8)

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-01-20 22:11:42 +01:00
bt90
35e153625c cmd/ursrv: Add FreeBSD detection (#9351)
### Purpose

Classify `ports@freebsd` as `FreeBSD (3rd party)`
2024-01-16 17:14:12 +01:00
bt90
d5e1b99e6c cmd/ursrv: Fix Arch detection (#9350)
### Purpose

Classify `syncthing@archlinux` as `Arch (3rd party)`
2024-01-16 17:13:34 +01:00
Jakob Borg
3297624037 lib/ignore: Optimise ignoring directories for filesystem watcher (fixes #9339) (#9340)
This improves the ignore handling so that directories can be fully
ignored (skipped in the watcher) in more cases. Specifically, where the
previous rule was that any complex `!`-pattern would disable skipping
directories, the new rule is that only matches on patterns *after* such
a `!`-pattern disable skipping. That is, the following now does the
intuitive thing:

```
/foo
/bar
!whatever
*
```

- `/foo/**` and `/bar/**` are completely skipped, since there is no
chance anything underneath them could ever be not-ignored
- `!whatever` toggles the "can't skip directories any more" flag
- Anything that matches `*` can't skip directories, because it's
possible we can have `whatever` match something deeper.

To enable this, some refactoring was necessary:

- The "can skip dirs" flag is now a property of the match result, not of
the pattern set as a whole.
- That meant returning a boolean is not good enough, we need to actually
return the entire `Result` (or, like, two booleans but that seemed
uglier and more annoying to use)
- `ShouldIgnore(string) boolean` went away with
`Match(string).IsIgnored()` being the obvious replacement (API
simplification!)
- The watcher then needed to import the `ignore` package (for the
`Result` type), but `fs` imports the watcher and `ignore` imports `fs`.
That's a cycle, so I broke out `Result` into a package of its own so
that it can be safely imported everywhere in things like `type Matcher
interface { Match(string) result.Result }`. There's a fair amount of
stuttering in `result.Result` and maybe we should go with something like
`ignoreresult.R` or so, leaving this open for discussion.

Tests refactored to suit, I think this change is in fact quite well
covered by the existing ones...

Also some noise because a few of the changed files were quite old and
got the `gofumpt` treatment by my editor. Sorry not sorry.

---------

Co-authored-by: Simon Frei <freisim93@gmail.com>
2024-01-15 10:13:22 +00:00
Syncthing Release Automation
445e8cc532 gui, man, authors: Update docs, translations, and contributors 2024-01-15 03:45:19 +00:00
Jakob Borg
e041877488 lib/ignore: Refactor out result type (#9343) 2024-01-13 18:58:23 +01:00
Jakob Borg
36e08f8eee build: Testing infra images for infra-* branches 2024-01-13 11:24:59 +01:00
nf
8b321387c0 lib/versioner: Expand tildes in version directory (fixes #9241) (#9327)
### Purpose

Fix #9241 by expanding tildes in version paths.

When creating the versioner file system, first try to expand any leading
tildes to the user's home directory before handling relative paths. This
makes a version path `"~/p"` expand to `"$HOME/p"` instead of
`"/folder/~/p"`.

### Testing

Added a test to lib/versioner that exercises this code path. Also
manually tested with local syncthing instances.
2024-01-12 10:46:18 +01:00
Julian Lehrhuber
8edd67a569 lib/scanner: Prevent sync-conflict for receive-only local modifications (#9323)
### Purpose

This PR changes behaviour of syncthing related to `receive-only`
folders, which I believe to be a bug since I wouldn't expect the current
behaviour. With the current syncthing codebase, a file of a
`receive-only` folder that is only modified locally can cause the
creation of a `.sync-conflict` file.

### Testing

Consider this szenario: Setup two paired clients that sync a folder with
a given file (e.g. `Test.txt`). One of the clients configures the folder
to be `receive-only`. Now, change the contents of the file for the
receive-only client **_twice_**.

With the current syncthing codebase, this leads to the creation of a
`.sync-conflict` file that contains the modified contents, while the
regular `Test.txt` file is reset to the cluster's provided contents.
This is due to a `protocol.FileInfo#ShouldConflict` check, that is
succeeding on the locally modified file.

This PR changes this behaviour to not reset the file and not cause the
creation of a `.sync-conflict`. Instead, the second content update is
treated the same as the first content update.

This PR also contains a test that fails on the current codebase and
succeeds with the changes introduced in this PR.

### Screenshots

This is not a GUI change

### Documentation

This is not a user visible change.

## Authorship

Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.

#### Thanks to all the syncthing folks for this awesome piece of
software!
2024-01-08 10:29:20 +01:00
Syncthing Release Automation
e829a63295 gui, man, authors: Update docs, translations, and contributors 2024-01-08 03:45:23 +00:00
diemade
6881a6897a Fix website security link in README.md (#9325)
https://syncthing.net/security/
html file does not exist

### Purpose
Fixing link that is supposed to link to https://syncthing.net/security/
2024-01-06 23:58:56 +01:00
Daniel Padrta
9387e107b3 cmd/syncthing: Add CLI completion functionality (fixes #8616) (#9226)
### Purpose

This implements CLI completion using the Kongplete module. As a side
effect a CLI structure for syncthing/cli was created for kongplete to be
able to parse and implement CLI completion.

### Testing

I've tested the autocompletion manually, and it had worked, but I hadn't
added any tests so as to test it automatically. Additionally, I ran `go
run build.go test` with all tests passing.
2024-01-04 10:20:53 +00:00
Jakob Borg
aa901790b9 lib/api: Save session & CSRF tokens to database, add option to stay logged in (fixes #9151) (#9284)
This adds a "token manager" which handles storing and checking expired
tokens, used for both sessions and CSRF tokens. It removes the old,
corresponding functionality for CSRFs which saved things in a file. The
result is less crap in the state directory, and active login sessions
now survive a Syncthing restart (this really annoyed me).

It also adds a boolean on login to create a longer-lived session cookie,
which is now possible and useful. Thus we can remain logged in over
browser restarts, which was also annoying... :)

<img width="1001" alt="Screenshot 2023-12-12 at 09 56 34"
src="https://github.com/syncthing/syncthing/assets/125426/55cb20c8-78fc-453e-825d-655b94c8623b">

Best viewed with whitespace-insensitive diff, as a bunch of the auth
functions became methods instead of closures which changed indentation.
2024-01-04 10:07:12 +00:00
Jakob Borg
17df4b8634 Update dependencies (#9321)
```
% export GOTOOLCHAIN=go1.20.7
% go list -m all | cut -d ' ' -f 1 | xargs go get -u
% go mod tidy
```

Except:

- github.com/jackpal/gateway now requires Go 1.21
- github.com/shirou/gopsutil breaks linux-mips in latest version
2024-01-04 10:56:11 +01:00
tomasz1986
34ef30dd5c gui: Always inform about loading data in Restore Versions modal (#9317)
Currently, with a large number of versioned files, there is a delay
between the Restore Versions modal showing up on the screen and
initialisation of the actual versions tree. This leads to a situation,
where the modal is initially empty, which confuses the user, making
them think that something is not working correctly.

To avoid the above, always show the loading data information. The string
is displayed using static HTML first, and then replaced with the exact
same content once the tree has been initialised. Both elements use the
same style and position, so there is no visual shift between the two.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-01-03 12:37:51 +01:00
Peter Badida
fc1c7a3c49 lib/build: Allow semver build in version regex (fixes #9267) (#9316) 2024-01-02 20:43:22 +01:00
Peter Badida
2abfefc18c gui: Keep short deviceID length consistent + xrefs (fixes #9313) (#9314)
Making short deviceID length consistent and referencing to protocol file
for future-proof edits. Closes #9313.
2024-01-02 17:31:57 +01:00
dependabot[bot]
86a08eb87d build(deps): bump actions/download-artifact from 3 to 4 (#9294)
Bumps
[actions/download-artifact](https://github.com/actions/download-artifact)
from 3 to 4.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/download-artifact/releases">actions/download-artifact's
releases</a>.</em></p>
<blockquote>
<h2>v4.0.0</h2>
<h2>What's Changed</h2>
<p>The release of upload-artifact@v4 and download-artifact@v4 are major
changes to the backend architecture of Artifacts. They have numerous
performance and behavioral improvements.</p>
<p>For more information, see the <a
href="https://github.com/actions/toolkit/tree/main/packages/artifact"><code>@​actions/artifact</code></a>
documentation.</p>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/bflad"><code>@​bflad</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/download-artifact/pull/194">actions/download-artifact#194</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/download-artifact/compare/v3...v4.0.0">https://github.com/actions/download-artifact/compare/v3...v4.0.0</a></p>
<h2>v3.0.2</h2>
<ul>
<li>Bump <code>@actions/artifact</code> to v1.1.1 - <a
href="https://redirect.github.com/actions/download-artifact/pull/195">actions/download-artifact#195</a></li>
<li>Fixed a bug in Node16 where if an HTTP download finished too quickly
(&lt;1ms, e.g. when it's mocked) we attempt to delete a temp file that
has not been created yet <a
href="hhttps://redirect.github.com/actions/toolkit/pull/1278">actions/toolkit#1278</a></li>
</ul>
<h2>v3.0.1</h2>
<ul>
<li><a
href="https://redirect.github.com/actions/download-artifact/pull/178">Bump
<code>@​actions/core</code> to 1.10.0</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="7a1cd3216c"><code>7a1cd32</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/246">#246</a>
from actions/v4-beta</li>
<li><a
href="8f32874a49"><code>8f32874</code></a>
licensed cache</li>
<li><a
href="b5ff8444b1"><code>b5ff844</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/245">#245</a>
from actions/robherley/v4-documentation</li>
<li><a
href="f07a0f73f5"><code>f07a0f7</code></a>
Update README.md</li>
<li><a
href="7226129829"><code>7226129</code></a>
update test workflow to use different artifact names for matrix</li>
<li><a
href="ada9446619"><code>ada9446</code></a>
update docs and bump <code>@​actions/artifact</code></li>
<li><a
href="7eafc8b729"><code>7eafc8b</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/244">#244</a>
from actions/robherley/bump-toolkit</li>
<li><a
href="3132d12662"><code>3132d12</code></a>
consume latest toolkit</li>
<li><a
href="5be1d38671"><code>5be1d38</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/243">#243</a>
from actions/robherley/v4-beta-updates</li>
<li><a
href="465b526e63"><code>465b526</code></a>
consume latest <code>@​actions/toolkit</code></li>
<li>Additional commits viewable in <a
href="https://github.com/actions/download-artifact/compare/v3...v4">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/download-artifact&package-manager=github_actions&previous-version=3&new-version=4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-01 11:09:23 +01:00
dependabot[bot]
d330d65859 build(deps): bump actions/upload-artifact from 3 to 4 (#9293)
Bumps
[actions/upload-artifact](https://github.com/actions/upload-artifact)
from 3 to 4.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/upload-artifact/releases">actions/upload-artifact's
releases</a>.</em></p>
<blockquote>
<h2>v4.0.0</h2>
<h2>What's Changed</h2>
<p>The release of upload-artifact@v4 and download-artifact@v4 are major
changes to the backend architecture of Artifacts. They have numerous
performance and behavioral improvements.</p>
<p>For more information, see the <a
href="https://github.com/actions/toolkit/tree/main/packages/artifact"><code>@​actions/artifact</code></a>
documentation.</p>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/vmjoseph"><code>@​vmjoseph</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/upload-artifact/pull/464">actions/upload-artifact#464</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/upload-artifact/compare/v3...v4.0.0">https://github.com/actions/upload-artifact/compare/v3...v4.0.0</a></p>
<h2>v3.1.3</h2>
<h2>What's Changed</h2>
<ul>
<li>chore(github): remove trailing whitespaces by <a
href="https://github.com/ljmf00"><code>@​ljmf00</code></a> in <a
href="https://redirect.github.com/actions/upload-artifact/pull/313">actions/upload-artifact#313</a></li>
<li>Bump <code>@​actions/artifact</code> version to v1.1.2 by <a
href="https://github.com/bethanyj28"><code>@​bethanyj28</code></a> in <a
href="https://redirect.github.com/actions/upload-artifact/pull/436">actions/upload-artifact#436</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/upload-artifact/compare/v3...v3.1.3">https://github.com/actions/upload-artifact/compare/v3...v3.1.3</a></p>
<h2>v3.1.2</h2>
<ul>
<li>Update all <code>@actions/*</code> NPM packages to their latest
versions- <a
href="https://redirect.github.com/actions/upload-artifact/issues/374">#374</a></li>
<li>Update all dev dependencies to their most recent versions - <a
href="https://redirect.github.com/actions/upload-artifact/issues/375">#375</a></li>
</ul>
<h2>v3.1.1</h2>
<ul>
<li>Update actions/core package to latest version to remove
<code>set-output</code> deprecation warning <a
href="https://redirect.github.com/actions/upload-artifact/issues/351">#351</a></li>
</ul>
<h2>v3.1.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Bump <code>@​actions/artifact</code> to v1.1.0 (<a
href="https://redirect.github.com/actions/upload-artifact/pull/327">actions/upload-artifact#327</a>)
<ul>
<li>Adds checksum headers on artifact upload (<a
href="https://redirect.github.com/actions/toolkit/pull/1095">actions/toolkit#1095</a>)
(<a
href="https://redirect.github.com/actions/toolkit/pull/1063">actions/toolkit#1063</a>)</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="c7d193f32e"><code>c7d193f</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/upload-artifact/issues/466">#466</a>
from actions/v4-beta</li>
<li><a
href="13131bb095"><code>13131bb</code></a>
licensed cache</li>
<li><a
href="4a6c273b98"><code>4a6c273</code></a>
Merge branch 'main' into v4-beta</li>
<li><a
href="f391bb91a3"><code>f391bb9</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/upload-artifact/issues/465">#465</a>
from actions/robherley/v4-documentation</li>
<li><a
href="9653d03c4b"><code>9653d03</code></a>
Apply suggestions from code review</li>
<li><a
href="875b630764"><code>875b630</code></a>
add limitations section</li>
<li><a
href="ecb21463e9"><code>ecb2146</code></a>
add compression example</li>
<li><a
href="5e7604f84a"><code>5e7604f</code></a>
trim some repeated info</li>
<li><a
href="d6437d0758"><code>d6437d0</code></a>
naming</li>
<li><a
href="1b56155703"><code>1b56155</code></a>
s/v4-beta/v4/g</li>
<li>Additional commits viewable in <a
href="https://github.com/actions/upload-artifact/compare/v3...v4">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/upload-artifact&package-manager=github_actions&previous-version=3&new-version=4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-01 11:08:55 +01:00
Syncthing Release Automation
4e1831fa3a gui, man, authors: Update docs, translations, and contributors 2024-01-01 03:45:30 +00:00
Simon Frei
2f3eacdb6c gui, lib/scanner: Improve scan progress indication (ref #8331) (#9308) 2023-12-31 23:01:16 +01:00
Sven Bachmann
1ce2af1238 lib/protocol: handle empty names in unixOwnershipEqual (fixes #9039) (#9306)
If syncOwnership is enabled and the remote uses for example a dockerized
Syncthing it can't fetch the ownername and groupname of the local
instance. Without this patch this led to an endless cycle of detected
changes on the remote and failing re-sync attempts.

This patch skips comparing the ownername and groupname if they zare empty
on one side.

See https://github.com/syncthing/syncthing/issues/9039 for details.

### Testing

Proposed by @calmh in
https://github.com/syncthing/syncthing/issues/9039#issuecomment-1870584783
and tested locally in my setup,

Setup PC 1:
- Syncthing is run in Docker as user `root` and has none of the users
configured that synchronize their files

Setup PC 2:
  - this PC has all users locally setup
- Syncthing runs as `systemd` service as user `syncthing` and has
multiple capabilities set to set the correct owner and permissions

Setup PC 3:
  - same as PC 2

Handling:
- `PC 1` is send & receive and uses just the `UID` and `GID` identifiers
to store the files
- `PC 2` and `PC 3` synchronize their files over `PC 1` but not directly
to each other

Outcome:
- `PC 2` and `PC 3` should send and receive their files with the correct
ownership and groups from `PC 1`
2023-12-29 09:16:33 +01:00
Syncthing Release Automation
683b48182c gui, man, authors: Update docs, translations, and contributors 2023-12-25 03:45:16 +00:00
Sertonix
795aed306c etc/linux-desktop: use double dash for long options (#9301)
Use a style for options that is consistent with the documentation and
other uses.
2023-12-23 11:16:47 +01:00
greatroar
cdefa535ed lib/connections: Skip allocation in check for missing port (#9297)
Micro-optimization. Already has unit tests.
2023-12-20 11:59:11 +01:00
gudvinr
91084b83b4 lib/upgrade: Extract signing key to embedded file (fixes #9247) (#9296)
### Purpose

Instead of hardcoding `SigningKey` as text use `go:embed`. Fixes #9247.

### Testing

* Building syncthing
* Trying to upgrade (signature verification)
2023-12-18 19:47:57 +00:00
Syncthing Release Automation
5360e7153b gui, man, authors: Update docs, translations, and contributors 2023-12-18 03:45:20 +00:00
Jakob Borg
5d0ca19350 build: Update quic-go (fixes #9287) 2023-12-13 12:35:59 +01:00
Eric P
e8d3529fed lib/model: Only handle relevant folder summaries (kqueue) (fixes #9183) (#9288)
On kqueue-systems, folders listen for folder summaries to (be able to)
warn for potential high resource usage. However, it listened for any
folder summary and not for the summary which matches the folder it's
about. This could cause that an unwatched folder causes a folder summary
containing more files than the threshold (10k), and the listening folder
(with the watcher enabled) triggers the warning.

This makes sure that only the folder summaries which are relevant to the
specific folder are being handled.

### Testing

- Fire up some kqueue-system (freebsd, I used).
- add folder A, disable the watcher, add 10001 files
- add folder B with the watcher enabled, no files are needed here

Before the change:
- add an item to folder A, trigger a rescan to speed up the process
- wait some seconds...warning triggered by folder B's
summarySubscription

After the change:
- Only a warning is triggered if the received folder summary matches the
folder which listens for the summaries
2023-12-13 12:34:24 +01:00
Jakob Borg
935a28c961 lib/model: Use a single lock (phase two: cleanup) (#9276)
Cleanup after #9275.

This renames `fmut` -> `mut`, removes the deadlock detector and
associated plumbing, renames some things from `...PRLocked` to
`...RLocked` and similar, and updates comments.

Apart from the removal of the deadlock detection machinery, no
functional code changes... i.e. almost 100% diff noise, have fun
reviewing.
2023-12-11 22:06:45 +01:00
dependabot[bot]
d21a2de055 build(deps): bump actions/setup-go from 4 to 5 (#9279)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to
5.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/setup-go/releases">actions/setup-go's
releases</a>.</em></p>
<blockquote>
<h2>v5.0.0</h2>
<h2>What's Changed</h2>
<p>In scope of this release, we change Nodejs runtime from node16 to
node20 (<a
href="https://redirect.github.com/actions/setup-go/pull/421">actions/setup-go#421</a>).
Moreover, we update some dependencies to the latest versions (<a
href="https://redirect.github.com/actions/setup-go/pull/445">actions/setup-go#445</a>).</p>
<p>Besides, this release contains such changes as:</p>
<ul>
<li>Fix hosted tool cache usage on windows by <a
href="https://github.com/galargh"><code>@​galargh</code></a> in <a
href="https://redirect.github.com/actions/setup-go/pull/411">actions/setup-go#411</a></li>
<li>Improve documentation regarding dependencies caching by <a
href="https://github.com/artemgavrilov"><code>@​artemgavrilov</code></a>
in <a
href="https://redirect.github.com/actions/setup-go/pull/417">actions/setup-go#417</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/galargh"><code>@​galargh</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/setup-go/pull/411">actions/setup-go#411</a></li>
<li><a
href="https://github.com/artemgavrilov"><code>@​artemgavrilov</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/setup-go/pull/417">actions/setup-go#417</a></li>
<li><a
href="https://github.com/chenrui333"><code>@​chenrui333</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/setup-go/pull/421">actions/setup-go#421</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/setup-go/compare/v4...v5.0.0">https://github.com/actions/setup-go/compare/v4...v5.0.0</a></p>
<h2>v4.1.0</h2>
<h2>What's Changed</h2>
<p>In scope of this release, slow installation on Windows was fixed by
<a href="https://github.com/dsame"><code>@​dsame</code></a> in <a
href="https://redirect.github.com/actions/setup-go/pull/393">actions/setup-go#393</a>
and OS version was added to <code>primaryKey</code> for Ubuntu runners
to avoid conflicts (<a
href="https://redirect.github.com/actions/setup-go/pull/383">actions/setup-go#383</a>)</p>
<p>This release also includes the following changes:</p>
<ul>
<li>Remove implicit dependencies by <a
href="https://github.com/nikolai-laevskii"><code>@​nikolai-laevskii</code></a>
in <a
href="https://redirect.github.com/actions/setup-go/pull/378">actions/setup-go#378</a></li>
<li>Update action.yml by <a
href="https://github.com/mkelly"><code>@​mkelly</code></a> in <a
href="https://redirect.github.com/actions/setup-go/pull/379">actions/setup-go#379</a></li>
<li>Added a description that go-version should be specified as a string
type by <a href="https://github.com/n3xem"><code>@​n3xem</code></a> in
<a
href="https://redirect.github.com/actions/setup-go/pull/367">actions/setup-go#367</a></li>
<li>Add note about YAML parsing versions by <a
href="https://github.com/dmitry-shibanov"><code>@​dmitry-shibanov</code></a>
in <a
href="https://redirect.github.com/actions/setup-go/pull/382">actions/setup-go#382</a></li>
<li>Automatic update of configuration files from 05/23/2023 by <a
href="https://github.com/github-actions"><code>@​github-actions</code></a>
in <a
href="https://redirect.github.com/actions/setup-go/pull/377">actions/setup-go#377</a></li>
<li>Bump tough-cookie and <code>@​azure/ms-rest-js</code> by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a
href="https://redirect.github.com/actions/setup-go/pull/392">actions/setup-go#392</a></li>
<li>Bump word-wrap from 1.2.3 to 1.2.4 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a
href="https://redirect.github.com/actions/setup-go/pull/397">actions/setup-go#397</a></li>
<li>Bump semver from 6.3.0 to 6.3.1 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a
href="https://redirect.github.com/actions/setup-go/pull/396">actions/setup-go#396</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/mkelly"><code>@​mkelly</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/setup-go/pull/379">actions/setup-go#379</a></li>
<li><a href="https://github.com/n3xem"><code>@​n3xem</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/setup-go/pull/367">actions/setup-go#367</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/setup-go/compare/v4...v4.1.0">https://github.com/actions/setup-go/compare/v4...v4.1.0</a></p>
<h2>v4.0.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Update documentation for <code>v4</code> by <a
href="https://github.com/dsame"><code>@​dsame</code></a> in <a
href="https://redirect.github.com/actions/setup-go/pull/354">actions/setup-go#354</a></li>
<li>Fix glob bug in the package.json scripts section by <a
href="https://github.com/IvanZosimov"><code>@​IvanZosimov</code></a> in
<a
href="https://redirect.github.com/actions/setup-go/pull/359">actions/setup-go#359</a></li>
<li>Bump <code>xml2js</code> dependency by <a
href="https://github.com/dmitry-shibanov"><code>@​dmitry-shibanov</code></a>
in <a
href="https://redirect.github.com/actions/setup-go/pull/370">actions/setup-go#370</a></li>
<li>Bump <code>@actions/cache</code> dependency to v3.2.1 by <a
href="https://github.com/nikolai-laevskii"><code>@​nikolai-laevskii</code></a>
in <a
href="https://redirect.github.com/actions/setup-go/pull/374">actions/setup-go#374</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/nikolai-laevskii"><code>@​nikolai-laevskii</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/setup-go/pull/374">actions/setup-go#374</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/setup-go/compare/v4...v4.0.1">https://github.com/actions/setup-go/compare/v4...v4.0.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="0c52d547c9"><code>0c52d54</code></a>
Update dependencies for node20 (<a
href="https://redirect.github.com/actions/setup-go/issues/445">#445</a>)</li>
<li><a
href="bfd2fb341f"><code>bfd2fb3</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/setup-go/issues/421">#421</a>
from chenrui333/node20-runtime</li>
<li><a
href="3d65fa57fc"><code>3d65fa5</code></a>
feat: bump to use actions/checkout@v4</li>
<li><a
href="8a505c9cf2"><code>8a505c9</code></a>
feat: bump to use node20 runtime</li>
<li><a
href="883490dfd0"><code>883490d</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/setup-go/issues/417">#417</a>
from artemgavrilov/main</li>
<li><a
href="d45ebba0ce"><code>d45ebba</code></a>
Rephrase sentence</li>
<li><a
href="317c6617fa"><code>317c661</code></a>
Replace <code>wildcards</code> term with <code>globs</code>.</li>
<li><a
href="f90673ad64"><code>f90673a</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/setup-go/issues/1">#1</a> from
artemgavrilov/caching-docs-improvement</li>
<li><a
href="8018234347"><code>8018234</code></a>
Improve documentation regarding dependencies cachin</li>
<li><a
href="d085b4fe57"><code>d085b4f</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/setup-go/issues/411">#411</a>
from galargh/fix/windows-hostedtoolcache</li>
<li>Additional commits viewable in <a
href="https://github.com/actions/setup-go/compare/v4...v5">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/setup-go&package-manager=github_actions&previous-version=4&new-version=5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-11 21:38:46 +01:00
Jakob Borg
6f1023665c lib/model: Use a single lock (#9275)
I'm tired of the fmut/pmut shenanigans. This consolidates both under one
lock; I'm not convinced there are any significant performance
differences with this approach since we're literally just protecting map
juggling...

- The locking goes away when we were already under an appropriate fmut
lock.
- Where we had fmut.RLock()+pmut.Lock() it gets upgraded to an
fmut.Lock().
- Otherwise s/pmut/fmut/.

In order to avoid diff noise for an important change I did not do the
following cleanups, which will be filed in a PR after this one, if
accepted:

- Renaming fmut to just mut
- Renaming methods that refer to being "PRLocked" and stuff like that
- Removing the no longer relevant deadlock detector
- Comments referring to pmut and locking sequences...
2023-12-11 21:26:23 +01:00
Jakob Borg
c53a1f210c cmd/syncthing: Better cli stdin handling (ref #9166) (#9281)
Seems to work for me, @AudriusButkevicius.
2023-12-11 21:15:52 +01:00
cjc7373
b71a930bfc cmd/syncthing: Mostly replace urfave/cli command line parser with alecthomas/kong (#9166)
`syncthing cli` subcommand was using urfave/cli as the command parser.
This PR replace it with kong, which the main command uses.

Some help texts and error message format are changed. Other than that,
all the command usage and logic remains unchanged.

There's only one place which still uses urfave/cli, which is `syncthing
cli config`, because it uses some magic to dynamically build commands
from struct reflects. I used kong's `passthrough:""` tag to pass any
argument following `syncthing cli config` to urfave/cli parser.

This PR also fixes #9041

---------

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2023-12-11 11:35:57 +01:00
Jakob Borg
a2cbc62521 lib/nat: Fix test build failure (ref #9010) 2023-12-11 10:13:14 +01:00
Jakob Borg
768fd6bff8 Merge branch 'release'
* release:
  lib/model: Add pmut locking for DeviceStatistics (fixes #9274)
  lib/model: Remove spurious "replacing service" failure event (ref #9271)
2023-12-11 08:13:28 +01:00
Jakob Borg
48883e0e32 lib/model: Add pmut locking for DeviceStatistics (fixes #9274)
Looking at deviceConnIDs requires this. Added in #9256.
2023-12-11 08:06:03 +01:00
Jakob Borg
30fe2cf514 lib/model: Add pmut locking for DeviceStatistics (fixes #9274)
Looking at deviceConnIDs requires this. Added in #9256.
2023-12-11 08:04:47 +01:00
Jakob Borg
3850a08252 lib/model: Remove spurious "replacing service" failure event (ref #9271)
This is no longer a notable condition, as we do this pretty much all the
time.
2023-12-11 07:44:04 +01:00
Jakob Borg
d0e407f3c3 lib/model: Remove spurious "replacing service" failure event (ref #9271)
This is no longer a notable condition, as we do this pretty much all the
time.
2023-12-11 07:43:40 +01:00
Maximilian
16db6fcf3d lib/nat, lib/upnp: IPv6 UPnP support (#9010)
This pull request allows syncthing to request an IPv6
[pinhole](https://en.wikipedia.org/wiki/Firewall_pinhole), addressing
issue #7406. This helps users who prefer to use IPv6 for hosting their
services or are forced to do so because of
[CGNAT](https://en.wikipedia.org/wiki/Carrier-grade_NAT). Otherwise,
such users would have to configure their firewall manually to allow
syncthing traffic to pass through while IPv4 users can use UPnP to take
care of network configuration already.

### Testing

I have tested this in a virtual machine setup with miniupnpd running on
the virtualized router. It successfully added an IPv6 pinhole when used
with IPv6 only, an IPv4 port mapping when used with IPv4 only and both
when dual-stack (IPv4 and IPv6) is used.

Automated tests could be added for SOAP responses from the router but
automatically testing this with a real network is likely infeasible.

### Documentation

https://docs.syncthing.net/users/firewall.html could be updated to
mention the fact that UPnP now works with IPv6, although this change is
more "behind the scenes".

---------

Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
Co-authored-by: André Colomb <github.com@andre.colomb.de>
2023-12-11 07:36:18 +01:00
Syncthing Release Automation
4c5528bd0e gui, man, authors: Update docs, translations, and contributors 2023-12-11 03:45:22 +00:00
tomasz1986
d42fff1016 gui: Show folder/device status on small screens (#8643)
gui: Show folder/device status on small screens

On larger screens, folder and device status is shown in a textual form
directly next to folder and device titles. However, on small screens,
only icons are currently shown, which may be ambiguous to new users, who
cannot possibly know what a specific icon means (see [1]). Thus, on
small screens only, display a new entry in folder/device info that
contains the same textual information that is shown in the title on
larger screens.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: André Colomb <src@andre.colomb.de>
2023-12-10 15:14:10 +01:00
Simon Frei
a28de73031 lib/model: Remove runner during folder cleanup (fixes #9269) (#9271)
Before introducing the service map and using it for folder runners, the
entries in folderCfgs and folderRunners for the same key/folder were
removed under a single lock. Stopping the folder happens separately
before that with just the read lock. Now with the service map stopping
the folder and removing it from the map is a single operation. And that
still happens with just a read-lock. However even with a full lock it’s
still problematic: After the folder stopped, the runner isn’t present
anymore while the folder-config still is and sais the folder isn't
paused.

The index handler in turn looks at the folder config that is not paused,
thus assumes the runner has to be present -> nil deref on the runner.

A better solution might be to push most of these fmut maps into the
folder - they anyway are needed in there. Then there's just a single
map/source of info that's necessarily consistent. That's quite a bit of
work though, and probably/likely there will be corner cases there too.
2023-12-08 07:13:09 +01:00
Jakob Borg
75310b58a0 build: Update dependencies (#9265) 2023-12-06 10:53:45 +01:00
Jakob Borg
8064957270 build: Revert specifics for Go 1.21.4, build using Go 1.21.5 (#9264)
This reverts commit e477777f49.

In principle, we could have stayed with `~1.21.1`, but `check-latest:
true` apparently checks some cache/manifest/something that is only
periodically refreshed and isn't aware of 1.21.5 yet. So update the
constraints to force an upgrade.

Also the infrastructure images weren't actually using the constraint
since there was no `setup-go` action...
2023-12-06 09:02:12 +01:00
Jakob Borg
c1ec9a8826 lib/fs: Reduce memory usage in xattrs handling (#9251)
This reduces allocations, in number and in size, while getting extended
attributes. This is mostly noticable when there is a large number of new
files to scan and we're running with the default scanProgressInterval --
then a queue of files is built in-memory, and this queue includes
extended attributes as part of file metadata. (Arguable it shouldn't,
but that's a more difficult and involved change.)

With 1M files to scan, each with one extended attribute, current peak
memory usage looks like this:

	Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total
	Dropped 1435 nodes (cum <= 7.26MB)
	Showing top 10 nodes out of 54
	      flat  flat%   sum%        cum   cum%
976.56MB 67.27% 67.27% 976.56MB 67.27%
github.com/syncthing/syncthing/lib/fs.getXattr
305.44MB 21.04% 88.31% 305.44MB 21.04%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.78MB 3.15% 91.47% 1045.23MB 72.00%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
22.89MB 1.58% 93.04% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 1.58% 94.62% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
16MB 1.10% 95.72% 16.01MB 1.10%
github.com/syndtr/goleveldb/leveldb/memdb.New

After the change, it's this:

	Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total
	Dropped 1400 nodes (cum <= 2.62MB)
	Showing top 10 nodes out of 91
	      flat  flat%   sum%        cum   cum%
305.43MB 58.19% 58.19% 305.43MB 58.19%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.79MB 8.72% 66.91% 68.68MB 13.09%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
32MB 6.10% 73.01% 32.01MB 6.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
22.89MB 4.36% 77.37% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 4.36% 81.73% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
15.35MB 2.92% 84.66% 15.36MB 2.93%
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
	   15.28MB  2.91% 87.57%    15.28MB  2.91%  strings.(*Builder).grow

(The usage for xattrs is reduced from 976 MB to 68 MB)
2023-12-04 12:48:17 +01:00
Jakob Borg
1625b44892 lib/model: Improve LastSeen handling (#9256)
LastSeen for a device was only updated when they connected. This now
updates it when they disconnect, so that we remember the last time we
actually saw them. When asking the API for current stats, currently
connected devices get a last seen value of the current time.
2023-12-04 09:24:10 +01:00
Jakob Borg
d51760f410 lib/scanner: Record inode change time for directories and symlinks (#9250) 2023-12-04 07:48:24 +01:00
Jakob Borg
7b1932d64e lib/api: Improve ignore loading error handling (fixes #9253) (#9254) 2023-12-04 07:11:35 +01:00
Syncthing Release Automation
5bfc540c88 gui, man, authors: Update docs, translations, and contributors 2023-12-04 03:45:22 +00:00
Jakob Borg
4cba99fcd4 lib/fs: Better equality comparison in mtimefs 2023-12-03 16:01:46 +01:00
Jakob Borg
2ae15aa454 cmd/stcrashreceiver: Add metrics for diskstore inventory 2023-11-27 08:24:59 +01:00
Jakob Borg
47bcf4f8f4 cmd/stcrashreceiver: Minor cleanup, stricter file permissions 2023-11-27 08:24:59 +01:00
Jakob Borg
a8b9096353 cmd/stcrashreceiver: Add metrics for incoming reports 2023-11-27 08:24:59 +01:00
Jakob Borg
5328380691 cmd/ursrv: Add metrics for incoming reports 2023-11-27 08:24:59 +01:00
Syncthing Release Automation
6069cf39e5 gui, man, authors: Update docs, translations, and contributors 2023-11-27 03:45:21 +00:00
Emil Lundberg
1f7d236742 gui: Specialize a special-purpose checkbox style (#9236)
### Purpose

Discovered while working on the WebAuthn credentials table in #9175:
there's a style on `td input[type="checkbox"]` that modifies margins for
all checkboxes in `<table>`s. It looks like this style is specially
tailored to the particular table that added it (PR #8734), so it should
have a correspondingly special-purpose class to not accidentally apply
it to other tables.

As best as I could tell there are only 2 instances of `<input
type="checkbox">` in `<td>`s, shown in the screenshots below.

### Testing

- Open "Actions > Logging > Debugging Facilities" and observe the
vertical spacing of the checkboxes.
- Open "Edit Folder > Advanced", check "Sync Extended Attributes" or
"Send Extended Attributes", click "Add filter entry" and observe the
vertical spacing of the checkbox that appears.

### Screenshots

#### Before

![Logs > Debugging
Facilities](https://github.com/syncthing/syncthing/assets/1367758/998fc66d-a0ad-41d9-a476-7a2b3da622d1)
![Add filter
entry](https://github.com/syncthing/syncthing/assets/1367758/647cb565-fcd0-4a81-a6ca-1f75137039b0)

#### After

Logs > Debugging Facilities now more compact:
![Logs > Debugging Facilities now
](https://github.com/syncthing/syncthing/assets/1367758/7cf8fc77-610e-4b4a-be21-c50d30be7bb9)

Add filter entry unchanged:
![Add filter
entry](https://github.com/syncthing/syncthing/assets/1367758/0ba710d6-cee1-49b4-92bc-acfc0c22c2bd)
2023-11-21 10:00:18 +01:00
Syncthing Release Automation
8e9ee3fbe8 gui, man, authors: Update docs, translations, and contributors 2023-11-20 08:49:38 +00:00
Jakob Borg
35b0afc131 build: Support new nested namespaces in Weblate downloads 2023-11-20 09:47:08 +01:00
Simon Frei
958ff67ccc lib/model: Acquire fmut lock in ensureIndexHandler (fixes #9234) (#9235)
Towards the end of the function f.folderCfgs and f.folderRunners are
read without the lock.
2023-11-20 08:12:25 +01:00
tomasz1986
13d9317a38 gui: Allow to translate "unknown device" (#9229)
The string is currently hard-coded in English, so allow to translate it
into other languages. It appears mainly in the browser title bar before
logging in into the GUI or when the GUI is still loading.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-11-16 22:07:15 +01:00
Anatoli Babenia
b184d46d8a cmd/ursrv: Add link to source code (#9224)
To see that https://data.syncthing.net is open source, study the code
and change it.
2023-11-15 08:51:23 +00:00
Jakob Borg
3f32c5cb4b cmd/ursrv: Anchor distribution expressions to avoid mismatches (ref #9141) 2023-11-15 09:32:46 +01:00
Jakob Borg
5c65a1bc83 build: Ursrv image for infrastructure 2023-11-15 08:48:00 +01:00
Jakob Borg
d0a6dc5b13 cmd/ursv: Report on copy range method 2023-11-15 08:48:00 +01:00
Jakob Borg
439c6c5b7c lib/api: Add cache busting for basic auth (ref #9208) (#9215)
This adds our short device ID to the basic auth realm. This has at least
two consequences:

- It is different from what's presented by another device on the same
address (e.g., if I use SSH forwards to different dives on the same
local address), preventing credentials for one from being sent to
another.

- It is different from what we did previously, meaning we avoid cached
credentials from old versions interfering with the new login flow.

I don't *think* there should be things that depend on our precise realm
string, so this shouldn't break any existing setups...

Sneakily this also changes the session cookie and CSRF name, because I
think `id.Short().String()` is nicer than `id.String()[:5]` and the
short ID is two characters longer. That's also not a problem...
2023-11-14 11:57:39 +01:00
Jakob Borg
aaee0c126b cmd/stdiscorv: Expose build info in metrics 2023-11-14 09:31:53 +01:00
André Colomb
f3bd4d71de gui: Fix Weblate merge conflict (#9222)
The manual translation updates in the recently merged PR #9220 caused
conflicts with the existing (test) entries on Weblate. This will fix it.
2023-11-14 08:21:40 +01:00
Jakob Borg
876d056705 build: Fixup Docker changes from previous (#9223) 2023-11-14 08:17:34 +01:00
DerRockWolf
e988978fa1 Add org.opencontainers.image.source to Dockerfiles (#9211)
### Purpose

The OCI image spec specifies well-defined
[annotations](https://github.com/opencontainers/image-spec/blob/main/annotations.md)
that can be added to images.
Theses annotations can then be used by other tools to gather more
information of an image.

This PR adds the `org.opencontainers.image.source` to allow tools such
as [renovate](https://github.com/renovatebot/renovate) to find the
release notes of a give version.

~~I've only done this change for `Dockerfile`. Should I also add the
label to the other dockerfiles?~~
I've now added the source annotations to all `Dockerfile`s & action
workflows.

### Testing

None, change was done by following the [renovate
documentation](https://docs.renovatebot.com/modules/datasource/docker/).
2023-11-14 07:46:14 +01:00
Jakob Borg
d5deede7a1 build: Update dependencies 2023-11-14 07:41:53 +01:00
André Colomb
4f70f5c280 gui: Use nested namespace for theme name translation keys (#9220)
Following up on #9192, this makes use of the new mechanism for the theme
names.

The dummy string added for testing is removed again here. All
translations are updated to the new nested syntax, except Chinese
(zh-HK) where the string weren't actually translated.
2023-11-14 07:22:52 +01:00
Emil Lundberg
a1ad020b63 Support explicit translation ID and dotted namespaces in translation extraction (#9192)
Some translations, especially single words or other short
labels for buttons and the like, may not be transferable between
contexts even if they happen to be equal in English. In these cases,
setting an explicit translation ID is important for context separation.
Angular Translate also supports nested JSON in translation tables,
addressed using `.` as namespace separator; this enhancement makes use
of this when extracting translation with an explicit translation ID.
2023-11-13 21:04:24 +01:00
Jakob Borg
8f1b0df74b lib/api: Improve cookie handling (fixes #9208) (#9214) 2023-11-13 20:37:29 +01:00
Jakob Borg
0f8dc6c1d3 test: Update testing configs
Somewhere along the way I snuck in a change to the test configs that is
quite annoying. This reverts that back to the more usual setup it was
before.
2023-11-13 13:51:17 +01:00
Jakob Borg
8ae9db3b2d build: Use actual Go version as cache key (#9216)
We use `env.GO_VERSION` as cache key for the build cache, but this is
nowadays typically something like `~1.21.1` which doesn't change when
1.21.2, 1.21.3 etc are released, making the cache fairly useless as
everything gets rebuilt. This re-sets the `GO_VERSION` variable after
installing Go so that it contains the actual installed version.
2023-11-13 12:20:40 +01:00
Jakob Borg
e477777f49 build: Version constraint to avoid Go 1.21.4 on Windows (ref #9207) (#9213) 2023-11-13 09:52:59 +00:00
Syncthing Release Automation
7a132bdf24 gui, man, authors: Update docs, translations, and contributors 2023-11-13 03:45:25 +00:00
Jakob Borg
5e2b7825dc cmd/stdiscosrv: Metric for returned retry-after 2023-11-08 12:18:59 +01:00
Jakob Borg
6d30c109e4 build: Push to Docker :edge tag for infrastructure builds 2023-11-08 12:18:59 +01:00
Jakob Borg
58bd931d90 cmd/stdiscosrv: Account IPv4 & IPv6 2023-11-08 12:18:59 +01:00
vapatel2
854499382e cmd/stdiscosrv: Prevent nil IPs from X-Forwarded-For (fixes #9189) (#9190)
### Purpose

Treat X-Forwarded-For as a comma-separated string to prevent nil IP being returned by the Discovery Server

### Testing

Unit Tests implemented

Testing with a Discovery Client can be done as follows:
```
A simple example to replicate this entails running Discovery with HTTP, use Nginx as a reverse proxy and hardcode (as an example) a list of IPs in the X-Forwarded-For header.
1. Send an Announcement with tcp://0.0.0.0:<some-port>
2. Query the DeviceID
3. Observe the returned IP Address is no longer nil; i.e.  `tcp://<nil>:<some-port>`
```
2023-11-08 11:10:23 +00:00
Jakob Borg
cb4c1f9ad2 build: Update dependencies (#9202) 2023-11-06 16:43:11 -08:00
Catfriend1
b452fb3ad2 gui: Add id attribute to login button, allows form filling tools to be used (fixes #9200) (#9201)
Add an id attribute to the submit button shown on the login form. This
allows my password manager's form filling function to interact with the
button after filling in username and password (which already have the id
attribute in place).
2023-11-06 16:30:19 -08:00
Syncthing Release Automation
c17a1fea77 gui, man, authors: Update docs, translations, and contributors 2023-11-06 03:45:25 +00:00
Syncthing Release Automation
d50511c5c6 gui, man, authors: Update docs, translations, and contributors 2023-10-30 03:45:44 +00:00
Jakob Borg
bae6d5f375 lib/location: Fix regression of timestamp handling (ref #9180) (#9185) 2023-10-26 07:41:02 +02:00
Jakob Borg
b5082f6af8 lib/locations: Change default config/data location to new XDG recommendation (fixes #9178, fixes #9179) (#9180)
This makes the new default $XDG_STATE_HOME/syncthing or
~/.local/state/syncthing, while still looking in legacy locations first
for existing installs.

Note that this does not *move* existing installs, and nor should we.
Existing paths will continue to be used as-is, but the user can move the
dir into the new place if they want to use it (as they could prior to
this change as well, for that matter).

### Documentation

Needs update to the config docs about our default locations.
2023-10-25 11:16:24 +02:00
dependabot[bot]
9666e9701b build(deps): bump github.com/quic-go/quic-go from 0.39.0 to 0.39.1 (#9181) 2023-10-24 07:23:38 +02:00
Syncthing Release Automation
86e1c5ff18 gui, man, authors: Update docs, translations, and contributors 2023-10-23 03:45:41 +00:00
tomasz1986
16ae1fbe5e lib/fs: Ignore inode change time on Android (#9177)
lib/fs: Fix conflicts on Android due to fluctuating inode change time

[1] added inode change time to file info in order to support syncing
extended attributes. However, in the case of Android, this inode change
time fluctuates, leading to unexpected conflicts even when the user has
not even touched the files on the Android device itself. Thus, in order
to prevent those conflicts from happening, do not write inode change
time on Android.

[1] 6cac308bcd

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-10-21 08:24:29 +02:00
Jakob Borg
11f508d9be build: Post build logs to Syncthing Loki 2023-10-16 11:07:40 +02:00
Jakob Borg
9ce6a73f42 Revert "cmd/stcrashreceiver: Aggregate slice out of bounds errors"
This reverts commit dc6a10dff4.
2023-10-16 08:08:23 +02:00
Syncthing Release Automation
c5a991cf0a gui, man, authors: Update docs, translations, and contributors 2023-10-16 03:45:30 +00:00
Emil Lundberg
14569f12d3 Hide log out button when auth is not enabled (#9158)
This was an oversight in #8757: the new "Log out" button is always shown
in the "Actions" menu, even when authentication is not enabled.
2023-10-15 14:10:41 +02:00
Jakob Borg
a405c21ebb cmd/stdiscosrv: Only attempt unescaping when there are %-encodings in the header (fixes #9143) 2023-10-14 12:30:29 +02:00
Jakob Borg
dc6a10dff4 cmd/stcrashreceiver: Aggregate slice out of bounds errors 2023-10-14 12:19:55 +02:00
Jakob Borg
d4c2acf6f6 cmd/stcrashreceiver: Propagate synthetic user ID for crashes 2023-10-14 12:19:55 +02:00
Jakob Borg
483ecada80 build: Update dependencies 2023-10-14 12:18:36 +02:00
Eric P
9553365d31 lib/fs: Properly handle Windows deduplicated files (fixes #9120) (#9168)
### Purpose

Deduplicated files are apparently considered 'irregular' under the hood,
this causes them to simply be ignored by Syncthing. This change is more
of a workaround than a proper fix, as the fix should probably happen in
the underlying libraries? - which may take some time. In the meanwhile,
this change should make deduplicated files be treated as regular files
and be indexed and synced as they should.

### Testing

Create some volume where deduplication is turned on (see the relevant
issue for details, including a proper description of how to reproduce
it). Prior to this change, the deduplicated files were simply ignored
(even by the indexer). After this change, the deduplicated files are
being index and synced properly.
2023-10-11 14:40:55 +02:00
orangekame3
5eb20580b1 cmd/ursrv: Replace "2006-01-02" with time.DateOnly (#9157)
This commit replaces "2006-01-02" to time.DateOnly. time.DateOnly is
introduced since Go1.20
2023-10-11 10:32:19 +00:00
Emil Lundberg
ea1ea366d2 lib/api: Check basic auth (and set session cookie) before noauth exceptions (#9159)
This is motivated by the Android app:
https://github.com/syncthing/syncthing-android/pull/1982#issuecomment-1752042554

The planned fix in response to basic auth behaviour changing in #8757
was to add the `Authorization` header when opening the WebView, but it
turns out the function used only applies the header to the initial page
load, not any subsequent script loads or AJAX calls. The
`basicAuthAndSessionMiddleware` checks for no-auth exceptions before
checking the `Authorization` header, so the header has no effect on the
initial page load since the `/` path is a no-auth exception. Thus the
Android app fails to log in when opening the WebView.

This changes the order of checks in `basicAuthAndSessionMiddleware` so
that the `Authorization` header is always checked if present, and a
session cookie is set if it is valid. Only after that does the
middleware fall back to checking for no-auth exceptions.

`api_test.go` has been expanded with additional checks:
- Check that a session cookie is set whenever correct basic auth is
provided.
- Check that a session cookie is not set when basic auth is incorrect.
- Check that a session cookie is not set when authenticating with an API
token (either via `X-Api-Key` or `Authorization: Bearer`).

And an additional test case:
- Check that requests to `/` always succeed, but receive a session
cookie when correct basic auth is provided.

I have manually verified that
- The new assertions fail if the `createSession` call is removed in
`basicAuthAndSessionMiddleware`.
- The new test cases in e6e4df4d70 fail
before the change in 0e47d37e73 is
applied.
2023-10-10 07:48:55 +02:00
Syncthing Release Automation
6e4574a9f7 gui, man, authors: Update docs, translations, and contributors 2023-10-09 03:45:35 +00:00
Jakob Borg
3d0da5ac60 lib/api: Better handle %s templates in LDAP strings (fixes #9072) (#9155)
Also add some escaping for good measure.
2023-10-07 02:29:53 +00:00
Jakob Borg
9f8e6966d8 docker: Allow start even if chown fails (fixes #9133) (#9152) 2023-10-07 02:12:07 +00:00
Jakob Borg
a64ae36bcc lib/model: Verify versioning on configuration reload (fixes #9106) (#9154) 2023-10-07 04:09:51 +02:00
Jakob Borg
690b55360f cmd/stdiscosrv: Handle unescaped cert header from Traefik (fixes #9143) (#9153) 2023-10-07 04:09:07 +02:00
DeflateAwning
2f6187dc0e Add oxford comma (#9137)
Co-authored-by: André Colomb <src@andre.colomb.de>
2023-10-06 17:25:28 +02:00
Emil Lundberg
8294870ffc Add HTML login form (fixes #4137) (#8757) 2023-10-06 13:00:58 +02:00
bt90
ac2e444a97 gui: Fix favicon status (fixes #9149) (#9150) 2023-10-06 12:27:13 +02:00
Jakob Borg
4f6b86a1c0 cmd/stdiscosrv: Slightly tweak replication settings 2023-10-04 14:15:00 +02:00
Jakob Borg
516c057d43 build: Update deps 2023-10-03 10:00:16 +02:00
Jakob Borg
d644dce4e7 build: Run release steps for workflow_dispatch as well 2023-10-03 09:33:52 +02:00
bt90
7c579880eb cmd/ursrv: Add linuxserver.io detection (#9145)
Detect linuxserver
2023-10-02 12:48:04 +02:00
Jakob Borg
296db314f5 lib/config: Improve parsing of gui-address overrides (#9144)
improve parsing of gui-address overrides

make checks for whether the gui-address is overridden consistent by
checking whether the environment variable is set and not an empty
string. the `Network()` function however checked for the inclusion of
a slash instead of the presence of any characters. If the config file's
gui address was set to a unix socket and the gui override to a tcp
address, then the function would have wrongly returned "unix".

the `URL()` function always returned the config file's gui address if a
unix socket was configured, even if an override was specified.

the `URL()` function wrongly formatted unix addresses. the http(s)
protocol was used as the sheme and the path was percent escaped. because
of the previous bug, this could only be triggered if the config file's
gui address was tcp and an unix socket override was given.

simplify the `useTLS()` function's codepath for overrides.

Co-authored-by: digital <didev@dinid.net>
2023-10-02 08:40:03 +02:00
Syncthing Release Automation
a8486b0468 gui, man, authors: Update docs, translations, and contributors 2023-10-02 03:45:41 +00:00
bt90
f8a7a034a7 cmd/ursrv: Fix f-droid detection (#9142)
Fix f-droid detection
2023-09-29 17:42:44 +02:00
bt90
ceae56a860 cmd/ursrv: Support new android build user (#9141)
Support new android build user
2023-09-29 16:34:28 +02:00
DeflateAwning
dcafd6ec72 readme: Style fixes, add security note (#9136) 2023-09-28 11:55:48 +02:00
Jakob Borg
8619a03f01 build: Update Actions 2023-09-25 21:50:17 +02:00
Jakob Borg
b91d7711aa Update dependencies (#9129)
And some QUIC API changes, of course.
2023-09-25 21:45:57 +02:00
d-volution
9940c91ebf gui: Scroll to bottom by clicking message in log viewer (#9128) 2023-09-25 19:42:27 +00:00
tomasz1986
80a577b025 gui: Show if device is untrusted in the main GUI (#9116)
Add a new entry to the unfolded device info to inform the user that the
device has been marked as "untrusted" and all folders shared with it
have to be password-protected or already Receive Encrypted.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-25 21:34:19 +02:00
tomasz1986
d672175ce4 gui: Show if device has Auto Accept enabled in the main GUI (#9118)
Add a new entry to the unfolded device info to inform the user that the
device has Auto Accept enabled.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-25 21:33:16 +02:00
tomasz1986
a44b31d173 gui: Fix body padding infinitely increasing due to overlapping modals (ref #9063) (#9078)
Opening and hiding multiple modals at the same time as well as opening a
modal before fully hiding the previous one can lead to the body padding
infinitely increasing by the scrollbar width each time, with the only
way to fix it being refreshing the GUI.

Therefore, always try to ensure to open and hide multiple modals one by
one, and also that the previous modal has fully been hidden before
proceeding to open the next one. The most common case when this problem
happens is when saving config changes which displays a GUI blocking
modal that overlaps, e.g. with folder or device modals that have not
been hidden yet.

Ref: https://github.com/twbs/bootstrap/issues/3902#issuecomment-1547187799

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-25 21:17:57 +02:00
Martin Polehla
70065e6b13 gitignore: All exe files, no editor configs (#9126) 2023-09-25 14:17:01 +00:00
Syncthing Release Automation
adbb3ed2e9 gui, man, authors: Update docs, translations, and contributors 2023-09-25 03:45:38 +00:00
Jakob Borg
6ed9c0c34c lib/config: Accept pre-hashed password (fixes #9123) (#9124) 2023-09-24 19:23:49 +02:00
tomasz1986
19bbf4f6bf gui: Add missing $scope in editDeviceUntrustedChanged function (#9117)
Because $scope is missing, there are JavaScript errors when ticking and
unticking the "Untrusted" checkbox in the Advanced tab of the Edit
Device modal.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-22 07:39:16 +02:00
bt90
cf46bf0297 lib/connections: Fix transport type detection for QUIC (fixes #8274) (#9114)
Check remote address
2023-09-20 11:23:48 +02:00
Jakob Borg
051cbdc713 lib/fs, lib/model: Be careful about potentially negative durations (fixes #9112) (#9113)
I don't really understand under what circumstances, but sometimes these
calls panic with a "panic: counter cannot decrease in value" because the
value passed to Add() was negative.
2023-09-20 09:04:47 +02:00
Syncthing Release Automation
58d1f3a471 gui, man, authors: Update docs, translations, and contributors 2023-09-18 03:45:31 +00:00
tomasz1986
c9dfd75d8e gui: Block GUI when saving changes only if necessary (ref #9063) (#9079)
Currently, the UI is always blocked from modifications when changes are
being saved, even if the save process takes very little time. This leads
to a situation where showing and closing the blocking modal can take
more time than is actually required to perform the whole operation. The
modal opening and closing very quickly can also cause the screen to
flash for a brief moment, leading to visual discomfort.

Because of this, wait for at least 200 ms and only show the blocking
modal if the changes have not been saved until then yet. The value of
200 ms is loosely based on [1] which states that 'a delay of 0.2–1.0
seconds does mean that users notice the delay and thus feel the computer
is "working" on the command, as opposed to having the command be a
direct effect of the users' actions.' Additionally, the delay must not
be too long, because the main purpose of the blocking modal is to
prevent the user from making further changes, and a longer delay would
possibly allow to do so in that brief amount of time as long as the user
is quick enough with their input.

[1] https://nngroup.com/articles/response-times-3-important-limits

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-12 15:02:37 +02:00
Jakob Borg
f47de83914 lib/protocol: Ensure starting & closing a connection are exclusive (fixes #9102) (#9103)
In principle a connection can close while it's in progress with
starting, and then it's undefined if we wait for goroutines to exit etc.
With this change, we will wait for start to complete before starting to
stop everything.
2023-09-12 14:48:15 +02:00
tomasz1986
caedb19307 gui: Remove unused hard-coded styles from Recent Changes modal (#9101)
gui: Remove unused hard-coded styles from globalChangesModalView modal

Currently, the globalChangesModalView modal has hardcoded th and td
styles. However, they are not even used in the modal itself, because
Bootstrap overrides them with its own styles for these elements in the
same modal. Yet, when hard-coded like that, these styles can conflict
with other table elements in the GUI. Thus, remove them completely.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-12 14:47:31 +02:00
bt90
e860d3b974 lib/connections: Make assumptions about isLAN when interface address listing fails (#9093) 2023-09-12 12:34:30 +00:00
bt90
ed66fba42b lib/beacon, lib/discover: Send IPv4 limited broadcast when address listing fails (fixes #1628) (#9087) 2023-09-12 14:28:17 +02:00
Jakob Borg
415f320005 build: Update dependencies 2023-09-12 14:08:59 +02:00
Jakob Borg
4812600098 lib/versioner: Don't complain when folder is stopping (#9097) 2023-09-11 23:10:18 +02:00
Jakob Borg
5ff11ce142 gui: Add help link for numConnections (#9082) 2023-09-11 14:59:48 +02:00
tomasz1986
541572781b gui: Add missing translation related to Number of Connections (ref #8918) (#9095)
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-11 05:50:23 +02:00
Syncthing Release Automation
e38679d9bf gui, man, authors: Update docs, translations, and contributors 2023-09-11 03:45:45 +00:00
Jakob Borg
f25a169c4c build: Go 1.21.1 or higher 2023-09-06 21:11:19 +02:00
bt90
06ac10ee37 cmd/stdiscosrv: Deduplicate addresses (fixes #8482) (#9080) 2023-09-06 14:36:00 +02:00
Jakob Borg
7c0223bd06 lib/build: Next version is the Gold Grasshopper 2023-09-06 13:13:39 +02:00
Jakob Borg
c6334e61aa all: Support multiple device connections (fixes #141) (#8918)
This adds the ability to have multiple concurrent connections to a single device. This is primarily useful when the network has multiple physical links for aggregated bandwidth. A single connection will never see a higher rate than a single link can give, but multiple connections are load-balanced over multiple links.

It is also incidentally useful for older multi-core CPUs, where bandwidth could be limited by the TLS performance of a single CPU core -- using multiple connections achieves concurrency in the required crypto calculations...

Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: tomasz1986 <twilczynski@naver.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
2023-09-06 12:52:01 +02:00
Jakob Borg
38bbdebffa build: Use actions/checkout@v4 2023-09-05 09:52:19 +02:00
Jakob Borg
e80d04845e build: Minor dependency update 2023-09-05 09:47:51 +02:00
Syncthing Release Automation
4138e22898 gui, man, authors: Update docs, translations, and contributors 2023-09-04 03:45:39 +00:00
Maximilian
c42c0e7ceb lib/connections: Fix WANAddresses returning only unspecified IPs (ref #9010) (#9073)
Avoids taking the address of the same variable twice.
2023-09-03 15:03:27 +00:00
Jakob Borg
5118538179 lib/model: Refactor folderRunners to use a serviceMap (#9071)
Instead of separately tracking the token.

Also changes serviceMap to have a channel version of RemoveAndWait, so
that it's possible to do the removal under a lock but wait outside of
the lock. And changed where we do that in connection close, reversing
the change that happened when I added the serviceMap in 40b3b9ad1.
2023-09-02 16:42:46 +02:00
tomasz1986
4d93648f75 gui: Don't hide default values for folders and devices (#8987)
Currently, some of the information for folders and devices displayed in
the GUI relies on arbitrary values that come pre-set as defaults on a
fresh Syncthing installation, i.e. if the value matches the default, it
is hidden, and if does not, then it is displayed.

With this change, the GUI always displays all information regardless
of their value, making the overall experience more consistent and
predictable.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-02 12:19:18 +02:00
tomasz1986
29f100c162 gui: Fix File Versioning icon to match in all places (#9070)
Currently, different icons are used for File Versioning when displayed
in the unfolded folder info in the main part of the GUI, and the icon
used in the Edit Folder modal. This changes the main GUI icon to match
the icon used in the modal.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-01 11:15:20 +02:00
tomasz1986
cd98a43b80 gui: Fix Logs modal icon to match header icon (ref #9067) (#9069)
The Logs icon was changed in [1] in the header, however the icon used in
the modal was left out. This changes it, so that the header and the
modal icons match.

[1] 2abba1dfb0

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-01 11:14:28 +02:00
Jakob Borg
4bf982376e build: Be more subtle about cross compilation errors
Summarize platforms that fail to build, without overloading the build
log with errors that we anyway ignore. (Currently freebsd/riscv64 fails
to build.)
2023-09-01 09:16:14 +02:00
Jakob Borg
29056d5873 build: Update dependencies (#9068) 2023-09-01 08:39:15 +02:00
tomasz1986
2abba1dfb0 gui: Remove footer and move links to header (fixes #5607) (#9067)
* gui: Remove footer and move links to header (fixes #5607)

Currently, the footer is always present and takes space at the bottom of
the GUI. However, the links listed there are not part of everyday user
interaction, and as such, they unnecessarily clutter the page, reducing
the usable screen space. Thus, transform the current Help link in the
header into a Help dropdown menu, and move the links from the footer
into it.

Also apply the following tweaks:

1. Move the About dialog from Actions to Help.
2. Add an Introduction (to the GUI) link to Help.
3. Change the Support icon from a question mark to a group of people.
4. Change the Changelog and About icons to a filled version to match the
   other icons better.
5. Use a source code icon for Source Code instead of a wrench icon, and
   move the wrench icon to Logs. This is done to prevent Changelog and
   Logs from using the same icon.
6. Update all dropdown icons' Fork Awesome styles to "fa fa-fw <icon>".

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>

* a few more Fork Awesome style updates

---------

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-09-01 08:18:30 +02:00
tomasz1986
325b3b114f gui: Fix lastSeenDays error due to undefined deviceStats when adding new devices (ref #8730) (#9066) 2023-09-01 07:22:04 +02:00
tomasz1986
03590e5ac7 gui: Automatically select device ID on click (ref #8544) (#9065)
The CSS method to select device IDs on click was added in [1]. However,
it was later mistakenly overwritten by [2]. This commit fixes the
regression and also applies the same behaviour to the Edit Device modal
which was omitted in the original commit.

[1] 5baf5fedb5
[2] 5e384c9185

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-08-31 22:16:59 +02:00
tomasz1986
95b3c26da7 gui: Prevent modifications when saving changes (fixes #9019) (#9063) 2023-08-31 17:11:03 +02:00
tomasz1986
3e5f0b1d0e gui: Show in GUI if limitBandwidthInLan is enabled (#9062) 2023-08-31 07:22:24 +02:00
Jakob Borg
3130af3773 lib/upgrade: Enable HTTP/2 for upgrade checks (#9060) 2023-08-30 21:58:34 +02:00
Jakob Borg
abd89f15f7 lib/discover: Enable HTTP/2 for global discovery requests (#9059)
By creating the http.Transport and tls.Configuration ourselves we
override some default behavior and end up with a client that speaks only
HTTP/1.1.

This adds a call to http.ConfigureTransport to do the relevant magic to
enable HTTP/2.

Also tweaks the keepalive settings to be a little kinder to the
server(s).
2023-08-30 21:58:05 +02:00
Jakob Borg
a80e6be353 cmd/stdiscosrv: Streamline context handling 2023-08-30 09:36:27 +02:00
Jakob Borg
acc532fc60 cmd/stdiscosrv: Explicitly enable HTTP/2
The server supports it, but it's not negotiated unless explicitly
allowed in the TLS config NextProtos.
2023-08-30 09:09:52 +02:00
Syncthing Release Automation
3cc3fb7504 gui, man, authors: Update docs, translations, and contributors 2023-08-28 03:45:57 +00:00
Jakob Borg
a04cc95005 cmd/stdiscosrv: Separate HTTPS and replication certificates 2023-08-23 13:43:54 +02:00
Jakob Borg
480fa4b915 cmd/stdiscosrv: Use larger database settings 2023-08-23 13:43:14 +02:00
Jakob Borg
92a4931850 cmd/stdiscosrv: Modernise TLS settings, remove excessive HTTP logging 2023-08-23 13:39:52 +02:00
Jakob Borg
bdfef9010f cmd/stdiscosrv: Serve compressed responses 2023-08-23 13:39:14 +02:00
bt90
467522d04d lib/connections: Allow IPv6 ULA in discovery announcements (fixes #7456) (#9048)
The allowed IPv4 ranges are the same as before. But we now also accept IPv6 addresses in the ULA range FC00::/7. These addresses don't require an interface identifier and are roughly equivalent to the IPv4 private ranges.

Typical usecases:

VPN interface IPs: Wireguard, OpenVPN, Tailscale, ...
fixed IPv6 LAN addressing while the provider assigns a dynamic prefix. e.g used by pihole
https://cs.opensource.google/go/go/+/refs/tags/go1.21.0:src/net/ip.go;l=146
2023-08-23 12:28:48 +02:00
bt90
3147285c60 lib/beacon: Check FlagRunning (#9051) 2023-08-22 11:27:43 +02:00
Jakob Borg
acd767b30b all: Remove lib/util package (#9049)
Grab-bag packages are nasty, this cleans it up a little by splitting it
into topical packages sempahore, netutil, stringutil, structutil.
2023-08-21 19:44:33 +02:00
Jakob Borg
40b3b9ad15 lib/model: Clean up index handler life cycle (fixes #9021) (#9038)
Co-authored-by: Simon Frei <freisim93@gmail.com>
2023-08-21 18:39:13 +02:00
bt90
c2c6133aa5 lib/osutil, lib/upnp: Check FlagRunning (fixes #8767) (#9047) 2023-08-21 14:49:28 +00:00
Jakob Borg
ccec8a4cdb build: Update dependencies (#9046) 2023-08-21 15:56:02 +02:00
Jakob Borg
cbf0e31f69 all: Use Go 1.21, new QUIC API (#9040) 2023-08-21 15:25:52 +02:00
Syncthing Release Automation
c40dae315b gui, man, authors: Update docs, translations, and contributors 2023-08-21 03:45:38 +00:00
Jakob Borg
ac0ce1c38f script: Remove find-metrics which belongs in docs 2023-08-17 12:27:56 +02:00
Jakob Borg
72c683aaca gui: Fix inadvertently always-false comparison (ref #7726) 2023-08-16 11:51:45 +02:00
Syncthing Release Automation
8042bd1a54 gui, man, authors: Update docs, translations, and contributors 2023-08-14 03:45:48 +00:00
Jakob Borg
462389934b cmd/stupgrades: Serve friendlier URLs for upgrade assets (fixes #9033) 2023-08-09 21:01:15 +02:00
Jakob Borg
b347c14bd1 build: Use correct range specification for Go version
The old `^1.20.7` means `1.x.y, >= 1.20.7` which allows 1.21.0, which
was not intended. The new `~1.20.7` means `1.20.x, >= 1.20.7`, which is
safer.
2023-08-09 16:05:11 +02:00
Jakob Borg
8dfec6983b build: WASM is not a thing we need to try to compile for 2023-08-09 11:02:43 +02:00
Jakob Borg
9ebf2dae7b build: Ability to manually trigger Actions builds 2023-08-09 10:50:07 +02:00
André Colomb
a8cacdca94 lib/versioner: Minor fixes in comments and error message (#9031)
* lib/versioner: Factor out DefaultPath constant.

Replace several instances where .stversions is named literally to all
use the same definition in the versioner package.  Exceptions are the
packages where a cyclic dependency on versioner is impossible, or some
tests which combine the versions base path with other components.

* lib/versioner: Fix comment about trash can in simple versioner.

* lib/versioner: Fix wrong versioning type string in error message.

The error message shows the folder type instead of the versioning
type, although the correct field is used in the comparison.
2023-08-09 07:10:06 +00:00
Jakob Borg
8b87cd5229 lib/model: Reinstate setting folder idle state (#9029) 2023-08-08 07:24:02 +02:00
Syncthing Release Automation
e09146ee03 gui, man, authors: Update docs, translations, and contributors 2023-08-07 03:45:35 +00:00
Jakob Borg
b9c08d3814 all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) (#9003) 2023-08-04 19:57:30 +02:00
Jakob Borg
58042b3129 build: Increase Go version to 1.20.7 2023-08-03 08:11:16 +02:00
Keith Harrison
eed12f3ec5 lib/config: Allow sharing already encrypted folder with untrusted devices (fixes #8965) (#9012)
Safety check added in v1.23.6 introduced bug. Bug unshares folders with untrusted devices if folder does not have an encryption password set, regardless of whether the folder is shared with the untrusted device as encrypted or not. Prevents sharing with untrusted devices in some cases where sharing would be encrypted.

Patch preserves safety check but permits sharing folders with untrusted devices if they are shared as encrypted.

Signed-off-by: kewiha <keithh@protonmail.com>
2023-08-02 07:14:53 +00:00
tomasz1986
5323928159 gui: Use case-insensive and backslash-agnostic versions filter (fixes #7973) (#8995)
Currently, the versions filter is case-sensitive regardless of the
underlying OS. With this change, the filter becomes case-insensitive
everywhere, which is more user-friendly and makes it easier to search
for files whose exact case the user may not remember.

In addition, forward and backslashes are no longer distinguished,
whether used as path separators or as part of a file / directory
name (which is unlikely but possible on some platforms).

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-08-01 14:20:01 +02:00
Syncthing Release Automation
97625ccc26 gui, man, authors: Update docs, translations, and contributors 2023-07-31 03:45:37 +00:00
Jakob Borg
4fe746d9aa build: Run govulncheck (fixes #8983) 2023-07-30 14:38:36 +02:00
Jakob Borg
4f8cdd41ee build: Run build & tests on main branch nightly 2023-07-30 14:24:17 +02:00
Jakob Borg
406e3646e5 build: Send test logs to Grafana Loki for statistics 2023-07-30 13:40:26 +02:00
Jakob Borg
9d21b91124 all: Refactor the protocol/model interface a bit (ref #8981) (#9007) 2023-07-29 10:24:44 +02:00
Chih-Hsuan Yen
b806026990 lib/connections: Fix building with -tags noquic (#9009) 2023-07-28 10:08:50 +00:00
tomasz1986
341b79814e gui: Fix tooltips on buttons inside button groups (ref #7984) (#9008)
As per Bootstrap recommendation, buttons with tooltips inside button
groups require to have container: 'body' set. This prevents tooltips
from causing the buttons to jump on hover and also allows the tooltips
to be wider instead of wrapping on every space.

Ref: https://getbootstrap.com/docs/3.3/components/#btn-groups

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-07-27 14:38:48 +02:00
Jakob Borg
319916124b cmd/strelaysrv: Handle accept error with debug set (fixes #9001) (#9004) 2023-07-26 23:55:48 +01:00
Emil Lundberg
b08b99e284 lib/api: Fix data race in TestCSRFRequired (#9006) 2023-07-26 21:33:45 +00:00
Jakob Borg
f565df628c gui: Show full error for failed items (#9005)
Also closes #8992.
2023-07-26 23:20:17 +02:00
Jakob Borg
855c6dc67b lib/api: Allow Bearer authentication style with API key (#9002)
Currently, historically, we look for the `X-API-Key` header to
authenticate with an API key. There's nothing wrong with this, but in
some scenarios it's easier to produce an `Authorization` header with a
`Bearer $token` content, which is nowadays more common. This change adds
support for both, so that we will accept an API key either in our custom
header or as a bearer token.
2023-07-26 13:13:06 +02:00
tomasz1986
dc5e10fa2c gui: Remove Twitter link from footer (#9000) 2023-07-25 00:53:57 +02:00
Syncthing Release Automation
b857e57a35 gui, man, authors: Update docs, translations, and contributors 2023-07-24 03:45:42 +00:00
tomasz1986
f42f041f53 lib/ur: Don't report uptime if start time is in the past (fixes #7698) (#8996)
Currently, because of devices with unset RTC clock, the 100% percentile
for Uptime on [1] is calculated since the Unix epoch which is useless as
far as usage statistics are concerned. Thus, if the Syncthing start time
is set to a past date, assume that the clock is wrong and do not even
try to report the uptime.

[1] https://data.syncthing.net

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
2023-07-22 21:25:03 +00:00
Christian Kujau
6b6b2c6194 lib/model: use WARN for "Unexpected folder" messages (#8998) 2023-07-22 21:17:32 +00:00
tomasz1986
d70eb569f2 lib/osutil: Skip setLowPriority in Windows if already lower (fixes #6597) (#8993) 2023-07-21 04:38:15 +00:00
deepsource-autofix[bot]
21c074cc2c all: replace empty slice literal with var (#8990)
refactor: replace empty slice literal with `var`

An empty slice can be represented by `nil` or an empty slice literal. They are
functionally equivalent — their `len` and `cap` are both zero — but the `nil`
slice is the preferred style. For more information about empty slices,
see [Declaring Empty Slices](https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices).

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2023-07-18 14:44:37 +00:00
deepsource-autofix[bot]
f23c41221b all: fix unused method receiver (#8988)
refactor: fix unused method receiver

Methods with unused receivers can be a symptom of unfinished refactoring or a bug. To keep 
the same method signature, omit the receiver name or '_' as it is unused.

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2023-07-18 14:34:50 +00:00
deepsource-autofix[bot]
24e230d455 all: unused parameter should be replaced by underscore (#8989)
refactor: unused parameter should be replaced by underscore

Unused parameters in functions or methods should be replaced with `_`
(underscore) or removed.

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2023-07-18 14:33:13 +00:00
Syncthing Release Automation
31daa20367 gui, man, authors: Update docs, translations, and contributors 2023-07-17 03:46:46 +00:00
Jakob Borg
e4d0f9dd6c cmd/syncthing: Mention STVERSIONEXTRA in --help output (ref #8980) 2023-07-16 17:48:24 +02:00
Jakob Borg
df2ac7aaeb gui, lib/api: Add possibility to feed through extra version information (#8980)
This adds an environment variable STVERSIONEXTRA that, when set, gets
added to the version information in the API and GUI.

The purpose of all this is to be able to communicate something about the
bundling or packaging, through the log & GUI and the end user, to the
potential person supporting it -- i.e., us. :) A wrapper can set this
variable to indicate that Syncthing is being run via `SyncTrayzor`,
`Syncthing-macOS`, etc., and thus indicate to the end user that the GUI
they are looking at is perhaps not the only source of truth and
management for this instance.
2023-07-16 17:43:10 +02:00
Jakob Borg
b96b23957b cmd/ursrv: Update map tile URL 2023-07-16 17:36:05 +02:00
bt90
265ce139c5 cmd/strelaypoolsrv: Update map tile URL (#8985) 2023-07-16 17:20:40 +02:00
Jakob Borg
48c95eb41d cmd/stcrashreceiver: Correct parsing of current version string 2023-07-12 09:27:34 +02:00
Jakob Borg
937895be69 build: Update dependencies 2023-07-11 09:14:21 +02:00
Jakob Borg
a3886f778d cmd/ursrv: Remove old, unused user movement code 2023-07-10 09:21:40 +02:00
Jakob Borg
6aecc2622c cmd/ursrv: Merge ursrv and uraggregate as subcommands 2023-07-10 09:00:57 +02:00
Jakob Borg
c55b205a0b cmd/ursrv: Remove useless static TLS cert handling 2023-07-10 08:39:30 +02:00
Jakob Borg
2fcf7006e6 cmd/ursrv: Embed static assets 2023-07-10 08:33:09 +02:00
Jakob Borg
bf61e485a6 cmd/ursrv: Refactor to use CLI options, fewer global vars 2023-07-10 08:27:16 +02:00
Jakob Borg
b2886f11b1 gui: Show xattr filter editor when send xattrs checked (fixes #8958) (#8959) 2023-07-10 08:01:08 +02:00
Syncthing Release Automation
11ece5d89e gui, man, authors: Update docs, translations, and contributors 2023-07-10 03:46:49 +00:00
Jakob Borg
cf68dfac43 build: Build Docker image for plain 32 bit arm (fixes #8973) 2023-07-08 10:55:09 +02:00
Jakob Borg
c44de2cd58 lib/fs: Clarify errors for Windows filenames (fixes #8968) (#8969)
With this change, error messages include the offending characters or
name parts. Examples:

    nul.txt: name is invalid, contains Windows reserved name: "nul"
    foo>bar.txt: name is invalid, contains Windows reserved character: ">"
    foo \bar.txt: name is invalid, must not end in space or period on Windows
2023-07-07 11:00:40 +00:00
Jakob Borg
6ff5ed6d23 gui: Avoid spurious comma in shared-with device list (fixes #8967) (#8970) 2023-07-07 07:25:24 +02:00
Jakob Borg
25ec2b63ab cmd/ursrv: Summarize tiny fraction items into Other 2023-07-05 08:22:10 +02:00
Jakob Borg
c5ab71d7a5 cmd/ursrv: Update distributions list 2023-07-05 08:08:29 +02:00
Jakob Borg
1fc4c9d9c5 build: Only push releases to cloud storage, and also use latest 2023-07-04 10:27:26 +02:00
Jakob Borg
bebf2c259c readme: Remove outdated build badges/links 2023-07-03 13:03:09 +02:00
Jakob Borg
b7d526903e build: Push to correct Docker images 2023-07-03 12:08:53 +02:00
Jakob Borg
92917c9f62 Merge branch 'release'
* release:
  build: Build and publish Docker images for relay and discovery server (fixes #8691, fixes #8960)
2023-07-03 11:52:11 +02:00
Jakob Borg
fdde05cf12 build: Build and publish Docker images for relay and discovery server (fixes #8691, fixes #8960)
This adds builds for the discovery and relay servers in the same manner
as Syncthing, so that Docker images are kept up to date with releases.

Inspired by and closes #8834.

Co-authored-by: Migelo <miha@filetki.si>
2023-07-03 11:51:12 +02:00
Syncthing Release Automation
a7d7325e9b gui, man, authors: Update docs, translations, and contributors 2023-07-03 03:45:49 +00:00
Jakob Borg
465823237f build: Build and publish Docker images for relay and discovery server (fixes #8691, fixes #8960)
This adds builds for the discovery and relay servers in the same manner
as Syncthing, so that Docker images are kept up to date with releases.

Inspired by and closes #8834.

Co-authored-by: Migelo <miha@filetki.si>
2023-07-01 08:48:12 +02:00
Jakob Borg
5a1f996e56 build: Build infrastructure images 2023-07-01 08:23:55 +02:00
Jakob Borg
229b6a292c cmd/stcrashreceiver: Add /ping endpoint 2023-07-01 07:53:50 +02:00
Jakob Borg
c84e47a7b2 build: Update dependencies 2023-06-28 14:00:30 +02:00
otbutz
daf7baaeff etc: Update sysctl configs (#8955) 2023-06-28 07:05:12 +02:00
Jakob Borg
a4a1231e92 gui: Remove tilde auto-expansions (fixes #8890) (#8954)
This removes the tilde expansion we had in the GUI. I'm not sure why we
had it, but there are valid reasons to have a folder path with tilde in
it in the config, and it was previously impossible to enter one of
those.
2023-06-28 07:03:53 +02:00
Jakob Borg
b99dee3ac3 cmd/syncthing: Add environment variables for --home, --conf, and --data (fixes #8957) (#8952)
This allows environment overrides for our directories. This is
advantageous because, apart from the obvious, it means we can set it in
the Docker file and not add command line options there. Having the
command line option as we did meant that it was impossible to use the
Docker image for other commands than `serve` (because that is implied
when we see other options on the command line).
2023-06-28 07:03:36 +02:00
Syncthing Release Automation
89fc69249b gui, man, authors: Update docs, translations, and contributors 2023-06-26 03:47:58 +00:00
Jakob Borg
d421d66a3f build: Docker images should have auto upgrade disabled... 2023-06-22 14:38:34 +02:00
guangwu
27aba3567b all: Minor staticcheck fixes (#8939) 2023-06-19 06:50:53 +00:00
Syncthing Release Automation
5532532db9 gui, man, authors: Update docs, translations, and contributors 2023-06-19 03:46:01 +00:00
Felix
c369f8abb2 gui: Have static link to favicon, for bookmarks etc (fixes #7638) (#8850) 2023-06-14 09:59:56 +02:00
Jakob Borg
9e52f6cf2f build: Update some dependencies 2023-06-14 09:43:45 +02:00
Jakob Borg
d22a38d947 build: Make sure we get the latest matching Go version
Also, disable caching in setup-go when we do manual cache setup with a
better cache key. It became default-true in the latest action version.
2023-06-14 09:37:59 +02:00
Jakob Borg
439fa6c848 build: Multi arch Docker images with GitHub actions (ref #8834)
This builds and publishes our Docker images from GitHub.
2023-06-14 09:28:04 +02:00
Jakob Borg
6b475bdb78 lib/config, gui: Disallow some options in combination with "untrusted" (fixes #8920) (#8921)
This prevents combining untrusted with introducer and auto-accept, and
also verifies that folders shared with untrusted devices have passwords
at config loading time.

Co-authored-by: Simon Frei <freisim93@gmail.com>
2023-06-14 09:24:31 +02:00
Syncthing Release Automation
7d56fba321 gui, man, authors: Update docs, translations, and contributors 2023-06-12 03:45:39 +00:00
Simon Frei
bf6ffbbd67 Don't add empty device to config on init (#8933)
We usually want to ensure that our own device is present. However if the
given device ID is the empty ID, we shouldn't do that. This is a
legimate (though way too non-obvious) use-case when opening the config
without knowing/caring about the device ID.
2023-06-10 20:33:39 +00:00
Jakob Borg
a972811f54 build: Push release files to cloud storage 2023-06-06 17:53:05 +02:00
Jakob Borg
88da67d7c3 build: Generate .asc files for release packages (fixes #8897) 2023-06-06 13:58:44 +02:00
Jakob Borg
1f07e05470 build: Properly build all Debian archs (fixes #8898) 2023-06-06 12:52:30 +02:00
Jakob Borg
5cab08a36a Merge branch 'release'
* release:
  gui: Avoid code generating HTML (#8923)
  gui: Remove HTML support in tooltips
2023-06-06 09:09:30 +02:00
Jakob Borg
73c52eafb6 gui: Avoid code generating HTML (#8923) 2023-06-06 09:06:51 +02:00
Jakob Borg
be5961f59b gui: Remove HTML support in tooltips 2023-06-06 09:06:36 +02:00
Jakob Borg
4e2bb58e2d gui: Avoid code generating HTML (#8923) 2023-06-05 13:16:02 +02:00
Jakob Borg
ae176ea9cd build: Tests should run with Go 1.20 on Windows (#8924)
Tests should run with Go 1.20 on Windows
2023-06-05 10:19:47 +02:00
Syncthing Release Automation
2b17db8aa3 gui, man, authors: Update docs, translations, and contributors 2023-06-05 03:45:37 +00:00
Jakob Borg
81a4b22d43 lib/model: Improve test for unignored parent directories (#8926) 2023-06-04 15:32:03 +02:00
Jakob Borg
f7da96fb82 build: Update dependencies (#8925)
The easy ones, there is also a quic-go update but it requires API
refactoring.
2023-06-04 15:31:20 +02:00
Jakob Borg
f5e5af391a gui: Remove HTML support in tooltips 2023-06-03 09:58:08 +02:00
Jakob Borg
5a3ac86c3f cmd/syncthing: Use correct binary when restarting monitor (#8919) 2023-06-02 07:54:31 +02:00
dependabot[bot]
3d78ff9f68 build(deps): bump github.com/minio/sha256-simd from 1.0.0 to 1.0.1 (#8916)
Bumps [github.com/minio/sha256-simd](https://github.com/minio/sha256-simd) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/minio/sha256-simd/releases)
- [Commits](https://github.com/minio/sha256-simd/compare/v1.0.0...v1.0.1)

---
updated-dependencies:
- dependency-name: github.com/minio/sha256-simd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-01 12:16:46 +02:00
Jakob Borg
f3127a66ee build: Increase concurrency, add "basic checks" gatekeeper 2023-06-01 12:15:31 +02:00
Syncthing Release Automation
90b4711ad2 gui, man, authors: Update docs, translations, and contributors 2023-05-29 03:45:27 +00:00
Syncthing Release Automation
716b42103a gui, man, authors: Update docs, translations, and contributors 2023-05-22 03:45:33 +00:00
Anthony Goeckner
405cdedcd3 lib/model: Set platform data for new folders w/ ignorePerms (ref #8883) (#8907)
* Platform data (ownership, xattrs, etc.) is now set correctly for newly-received folders, even if the received folder has the NoPermissions flag.

* Call setPlatformData on receivers that have ignorePerms set to true.
2023-05-17 09:06:50 +02:00
Syncthing Release Automation
0b3a101ccd gui, man, authors: Update docs, translations, and contributors 2023-05-15 03:45:55 +00:00
Eng Zer Jun
089320aadc lib: replace Readdir(-1) with os.ReadDir (#8901) 2023-05-11 15:35:52 +00:00
Will Rouesnel
b2fb2ef276 lib/api: Allow BindDN to exclude any username formatting (fixes #8899) (#8900)
This allows a syncthing instance to be locked to exactly 1 user without
needing search capability on the LDAP instance.
2023-05-10 07:52:02 +02:00
Jakob Borg
e136d11dce build: Attempt cross compilation for ~all targets, allow it to fail 2023-05-09 10:01:57 +00:00
Ross Smith II
3adfe2f91f lib/fs: Fix root path handling for Windows (fixes #8778)
Co-authored-by: Jakob Borg <jakob@kastelo.net>
2023-05-09 10:01:57 +00:00
Jakob Borg
1103a27337 all: Grand test refactor (fixes #8779, fixes #8799)
This fixes various test issues with Go 1.20.

- Most tests rewritten to use fakefs where possible
- Some tests that were already skipped, or dubious (invasive,
  unmaintainable, unclear what they even tested) have been removed
- Some actual code rewritten to better support testing in fakefs

Co-authored-by: Eric P <eric@kastelo.net>
2023-05-09 10:01:57 +00:00
Alexander Seiler
ddce692f72 all: Correct various typos (#8870) 2023-05-09 08:54:02 +02:00
Syncthing Release Automation
66faea7712 gui, man, authors: Update docs, translations, and contributors 2023-05-08 03:45:35 +00:00
Anthony Goeckner
7e31ec5417 lib/model: Set platform data, incl. copying ownership, for new folders w/ NoPermissions flag (#8883)
Platform data (ownership, xattrs, etc.) is now set correctly for newly-received folders, even if the received folder has the NoPermissions flag.
2023-05-02 11:11:39 +02:00
André Colomb
a4fa764b7d gui: Add Thai (th) translation template. (#8887) 2023-05-02 06:38:29 +00:00
Jakob Borg
7226b8456b build: Produce nightly release builds 2023-05-01 09:42:44 +02:00
Syncthing Release Automation
dae5eab787 gui, man, authors: Update docs, translations, and contributors 2023-05-01 03:45:35 +00:00
K.B.Dharun Krishna
f38e9628a1 build: Bump actions version; fix Node 12 deprecation warning (#8881) 2023-04-29 15:21:46 +02:00
Jakob Borg
43e3b12e29 build: Build Debian packages 2023-04-28 13:22:25 +02:00
Jakob Borg
aa01ff5d50 build: Sign for upgrades 2023-04-28 13:03:25 +02:00
Jakob Borg
63503e0c98 build: Notarize mac builds 2023-04-28 13:03:25 +02:00
dependabot[bot]
947dd0db09 build(deps): bump github.com/quic-go/quic-go from 0.33.0 to 0.34.0 (#8877)
Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.33.0 to 0.34.0.
- [Release notes](https://github.com/quic-go/quic-go/releases)
- [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md)
- [Commits](https://github.com/quic-go/quic-go/compare/v0.33.0...v0.34.0)

---
updated-dependencies:
- dependency-name: github.com/quic-go/quic-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-24 19:04:27 +02:00
Syncthing Release Automation
218b6e5193 gui, man, authors: Update docs, translations, and contributors 2023-04-24 03:45:39 +00:00
Eric P
9f131eee6b lib/ignore: Properly handle non-existing included ignore-files (fixes #8764) (#8874)
In the sequence of loading ignores, the error File Does Not Exist is not being considered a fatal  error, since the .stignore file is allowed to not exist. However, included ignore files also tossed that same error in case those do not exist while in those cases it's considered an error and it should lead to the folder stopping. Changing the error when opening an included ignore file to something other than the regular does fix this issue, as in it now works again as described in the Documentation.
2023-04-20 15:00:55 +02:00
Jakob Borg
09efe03e1d lib/connections: Avoid using nil lanChecker
Otherwise it panics when someone calls Priority() on it...
2023-04-19 10:42:25 +02:00
Syncthing Release Automation
0f87607cd5 gui, man, authors: Update docs, translations, and contributors 2023-04-17 03:45:33 +00:00
Jakob Borg
9b660c1959 lib/config, lib/connections: Configurable protocol priority (ref #8626) (#8868)
This makes the various protocol priorities configurable among the other
options. With this, it's possible to prefer QUIC over TCP for WAN
connections, for example. Both sides need to be similarly configured for
this to work properly.

The default priority order remains the same as previously (TCP, QUIC,
Relay, with LAN better than WAN).

To make this happen I made each dialer & listener more priority aware,
and moved the check for whether a connection is LAN or not into the
dialer / listener -- this is the new "lanChecker" type that's passed
around.
2023-04-16 14:54:28 +02:00
Simon Frei
c867a5f5b3 build: Upgrade recli (fixes #8503) (#8871) 2023-04-14 18:46:50 +02:00
Jakob Borg
1886b47031 build: Update dependencies (#8869) 2023-04-11 20:01:42 +02:00
Jakob Borg
f59ffc8ddd lib/model: Improve path generation for auto accepted folders (fixes #8859) (#8860)
- Make sure we don't try to use empty last path components
- Create the directory to "reserve" it once we've decided to use it
2023-04-11 13:07:22 +02:00
Evgeny Kuznetsov
61444960bc docs: fix typo (#8857) 2023-04-10 10:30:11 +00:00
Syncthing Release Automation
30bb8f2116 gui, man, authors: Update docs, translations, and contributors 2023-04-10 03:45:37 +00:00
Jakob Borg
4a8c691aef lib/syncthing: Handle successful global migration (fixes #8851) (#8852)
lib/syncthing: Handle successfull global migration (fixes #8851)
2023-04-05 15:25:55 +02:00
Syncthing Release Automation
e8dafb979c gui, man, authors: Update docs, translations, and contributors 2023-04-03 03:45:47 +00:00
Simon Frei
3a41d4afa5 lib/model: Set enc. trailer size on pull (ref #8563, #8556) (#8839)
In the original fix in #8563 I simply forgot this. Which meant #8556
wasn't actually fixed, as the trialer size would have been 0 (default),
and thus we would have still sent the inflated size to encrypted peers.
2023-03-28 22:47:24 +02:00
Simon Frei
6a66aee489 lib/model: Fix file size inconsistency due to enc. trailer (#8840)
lib/model: Fix file size inconsisency due to enc. trailer

Fixes a regression due to PR #8563, while arguable the bug was actually
introduced in a much older PR #7155, but didn't have any bad effects so
far:
We account for the encryption trailer in the db updater routine,
calculating the file-info size there. However there's no guarantee that
the file-info at this point is still the exact same as when it was
written. It was before, but isn't anymore since introducing the new
EncryptedTrailerSize field.
Fix: Adjust the size in the info at the same place where the trailer is
written, i.e. we definitely have the actual size on disk.
2023-03-28 22:02:59 +02:00
Syncthing Release Automation
51e85d5162 gui, man, authors: Update docs, translations, and contributors 2023-03-27 03:45:37 +00:00
Jakob Borg
b49f535834 cmd/stdiscorv: Fix database test (fixes #8828)
The problem was that a statistics/cleanup run is triggered when the
database started and runs concurrently with the test. That cleanup run
removes old entries without valid addresses, and one of the test objects
matched this. The test object would thus randomly be removed in the
middle of the test, causing a failure. This fixes it so the object looks
recent when the cleaner-upper looks, and also uses a RAM database
(faster).
2023-03-24 09:15:12 +01:00
Jakob Borg
34b312b85b lib/ur: Fix custom releases URL comparison 2023-03-24 09:15:12 +01:00
André Colomb
be3751ab5f gui: Remove untranslated strings from JSON files (#8836)
* gui: Import latest state from Weblate.

Downloaded 2023-03-23 at 20:38:53.

* gui: Remove untranslated strings from JSON files.

The GUI code will fall back to the English string translation anyway,
so keeping it identical inside the other language files does not
help.  It makes the filter handling on Weblate easier though.

Current state retreived with these filter settings:

    state:>=translated OR (state:needs-editing AND NOT check:same)
2023-03-23 21:12:31 +01:00
Dimitri Papadopoulos Orfanos
526e21ae26 all: Fix typos found by codespell (#8833) 2023-03-21 08:07:28 +01:00
Jakob Borg
bc371eacf6 gui: Hide download progress legend when download progress is disabled 2023-03-20 15:34:18 +01:00
Syncthing Release Automation
5caf9bfc18 gui, man, authors: Update docs, translations, and contributors 2023-03-20 03:45:38 +00:00
Jakob Borg
f378e63147 lib/protocol: Handle encrypted requests without encrypted hash (fixes #8277) (#8827)
The layout of the request differs based on whether it comes from an
untrusted device or a trusted device with encrypted enabled. Handle
both.

Closes #8819.
2023-03-18 10:22:18 +01:00
dependabot[bot]
fb89898462 build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.1 to 2.0.2 (#8826)
Bumps [github.com/hashicorp/golang-lru/v2](https://github.com/hashicorp/golang-lru) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/hashicorp/golang-lru/releases)
- [Commits](https://github.com/hashicorp/golang-lru/compare/v2.0.1...v2.0.2)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/golang-lru/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-18 10:17:16 +01:00
tomasz1986
358cf25cff lib/config: Allow sub-second watcher delay (fixes #7859) (#7864)
Allow the watcher delay to take fractional values, effectively allowing
for much shorter delays. The minimum value is limited at 0.01, which
effectively translates to 10ms. This is required in order to guarantee
that there is still enough time to aggregate multiple single change
events.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-03-18 08:50:38 +01:00
Syncthing Release Automation
59de7048bd gui, man, authors: Update docs, translations, and contributors 2023-03-13 03:45:45 +00:00
Jakob Borg
466b56ded1 lib/protocol: Cache expensive key operations (fixes #8599) (#8820)
This adds a cache to the expensive key generation operations. It's fixes
size LRU/MRU stuff to keep memory usage bounded under absurd conditions.

Also closes #8600.
2023-03-12 20:06:59 +01:00
André Colomb
3ffe859fe8 gui: Add Persian (fa) translation template (#8822)
gui: Add Persian (fa) translation template.

Based on user request from Weblate, user @a-kbd.
2023-03-10 16:27:48 +01:00
Simon Frei
da72df6ffc lib: Correctly handle encrypted trailer size (fixes #8556) (#8563) 2023-03-10 14:14:14 +01:00
tomasz1986
b53c1b9a04 gui: Disable Restore Versions filters when no versioned files exist (fixes #5408) (#8539)
Currently, the name and date filters in the Restore Versions modal are
always enabled, even if there are no versioned files present. With this
change, they are enabled only when there are no errors and versioned
files actually exist.

In addition to disabling the filters, also completely skip date picker
generation, as it serves no function while still displaying a non-sense
date range, which starts today and ends in the past.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-03-10 13:41:15 +01:00
dependabot[bot]
b00976781c build(deps): bump github.com/chmduquesne/rollinghash from 0.0.0-20180912150627-a60f8e7142b5 to 4.0.0+incompatible (#8804)
build(deps): bump github.com/chmduquesne/rollinghash

Bumps [github.com/chmduquesne/rollinghash](https://github.com/chmduquesne/rollinghash) from 0.0.0-20180912150627-a60f8e7142b5 to 4.0.0+incompatible.
- [Release notes](https://github.com/chmduquesne/rollinghash/releases)
- [Commits](https://github.com/chmduquesne/rollinghash/commits/v4.0.0)

---
updated-dependencies:
- dependency-name: github.com/chmduquesne/rollinghash
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-10 13:36:25 +01:00
Jakob Borg
58f9dcca31 build: Update dependencies (#8821) 2023-03-10 13:04:13 +01:00
Jakob Borg
ab8e6a82ab lib/api: Expose blocksHash in file info (#8810)
This adds the BlocksHash field from the FileInfo to our API output. It
can be useful for debugging, or for external tools. I'm intentionally
leaving it as an opaque base64 string because no meaning should be
derived from it: it's just a string.
2023-03-06 15:37:15 +01:00
Syncthing Release Automation
3cdcb7dd72 gui, man, authors: Update docs, translations, and contributors 2023-03-06 03:45:45 +00:00
entity0xfe
4558eef446 lib/discover: Don't leak relay-tokens to discovery (#8762)
Use an allowlist to send only the `id` query param to the discovery server.
2023-03-04 12:16:57 +01:00
Syncthing Release Automation
8bbf2ba9ac gui, man, authors: Update docs, translations, and contributors 2023-02-27 03:45:16 +00:00
André Colomb
49c56b8caa gui: Add Croatian (hr) translation template (#8801) 2023-02-24 17:54:30 +01:00
dependabot[bot]
0b8aa7974b build(deps): bump github.com/quic-go/quic-go from 0.32.0 to 0.33.0 (#8800)
Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.32.0 to 0.33.0.
- [Release notes](https://github.com/quic-go/quic-go/releases)
- [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md)
- [Commits](https://github.com/quic-go/quic-go/compare/v0.32.0...v0.33.0)

---
updated-dependencies:
- dependency-name: github.com/quic-go/quic-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-22 13:01:08 +01:00
Jakob Borg
cbec697e5f cmd/stupgrades: Cache should apply to HEAD as well as GET 2023-02-22 12:22:52 +01:00
Jakob Borg
8991ecf444 build: Add more GitHub Actions 2023-02-22 10:56:55 +01:00
tomasz1986
17887ce0b1 gui: Remove non-existent customicons.css file reference (fixes #8797) (#8798)
The reference comes from fd0a6225aa,
but the file itself was removed in the process of working on the pull
request, yet the reference to it was still left in the index.html.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-02-21 19:05:15 +01:00
Andreas Sommer
2f88dafa56 Only fail after chmod error if permissions differ (e.g. on config file) (#8771) 2023-02-20 15:41:10 +01:00
Syncthing Release Automation
2321d0db08 gui, man, authors: Update docs, translations, and contributors 2023-02-20 03:45:34 +00:00
Jakob Borg
7047e00d1e build: Use Go 1.19.6 for Windows 2023-02-15 13:20:47 +01:00
Jakob Borg
a8533b269d build: Update dependencies 2023-02-14 19:08:25 +01:00
Syncthing Release Automation
ee30647bf7 gui, man, authors: Update docs, translations, and contributors 2023-02-13 03:48:08 +00:00
André Colomb
750accb25f gui: Remove duplicate Spanish (Spain) translation (fixes #8781) (#8782)
gui: Remove duplicate Spanish (Spain) translation.

Currently not among the set of "valid" (sufficiently complete)
languages and no longer synced with the tranlation platform, Weblate.
2023-02-09 10:10:40 +01:00
Eric P
0530f0edbf gui: Add xattr filter editor (fixes #8660) (#8734) 2023-02-09 09:14:36 +01:00
André Colomb
784129e1cf gui: Switch to Weblate for translations (#8777)
* Normalize EOL before EOF.

* Import current Transifex state, only translated.

Skip duplicated and completely empty languages.

* Re-add untranslated strings to JSON files.

These are now treated correctly as untranslated, but present with the
original (English) translation key.

* Update valid language codes and names.

* Refer to Weblate instead of Transifex for translations.

* lang-es: Copy untranslated strings from es-ES.

* Integrate translation changes from Weblate.

Translated using Weblate (Polish)

Currently translated at 100.0% (512 of 512 strings)

Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/pl/

Translated using Weblate (Danish)

Currently translated at 98.6% (505 of 512 strings)

Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/da/

Translated using Weblate (Lithuanian)

Currently translated at 85.4% (441 of 516 strings)

Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/lt/

Translated using Weblate (French)

Currently translated at 100.0% (516 of 516 strings)

Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/fr/

Translated using Weblate (Danish)

Currently translated at 98.6% (509 of 516 strings)

Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/da/

Translated using Weblate (Spanish)

Currently translated at 100.0% (516 of 516 strings)

Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/es/

* translation update uses weblate

---------

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2023-02-07 12:22:08 +01:00
greatroar
38f2b34d29 all: Use new Go 1.19 atomic types (#8772) 2023-02-07 12:07:34 +01:00
Syncthing Release Automation
882b711958 gui, man, authors: Update docs, translations, and contributors 2023-02-06 03:45:18 +00:00
Jakob Borg
99595ce3d9 build: Update quic-go and pfilter for Go 1.20 (fixes #8768) (#8769) 2023-02-02 22:00:50 +01:00
Jakob Borg
d5e4ef249f Add forgotten copyright notices 2023-02-01 22:59:41 +01:00
Jakob Borg
65cfefaa3c cmd, docker: Updates for infrastructure
These are some changes to the relay pool server, upgrade server, and
crash receiver to run under Kubernetes. It's been in production for a
while.
2023-01-31 11:17:52 +01:00
Jakob Borg
a6c2a5a0ce cmd/ursrv: The driver import is important, though 2023-01-31 09:57:16 +01:00
Jakob Borg
a5b6199507 cmd/ursrv: Remove old, unused migration code 2023-01-31 09:12:48 +01:00
Jakob Borg
966e9f9a68 cmd/ursrv: Harmonize timespan of charts 2023-01-31 09:11:15 +01:00
Jakob Borg
b10e11abb7 cmd/ursrv: Remove broken and unsustainable join/leave chart 2023-01-31 09:10:52 +01:00
Jakob Borg
5a50de1154 cmd/ursrv: Fix broken block transfer chart 2023-01-31 09:09:36 +01:00
Syncthing Release Automation
dc6d695c00 gui, man, authors: Update docs, translations, and contributors 2023-01-30 03:45:28 +00:00
Vik
b120278b3a gui: Fix broken link to Transifex in lang/README.txt (#8761)
Updated URL
2023-01-29 21:52:09 +01:00
Syncthing Release Automation
c25f5b40d9 gui, man, authors: Update docs, translations, and contributors 2023-01-23 03:45:17 +00:00
Jakob Borg
abdac2caa2 Handle relay connect timeout (fixes #8749) (#8755)
This makes sure the service manager doesn't interpret timeout errors, or any other error, as a signal to stop the service instead of restarting it.

I added it directly to our service utility function, as it may help catch other instances of the same problem... We would typically want timeouts etc to be a retryable error, unless it is the top level context that has timed out and we check for that specifically.
2023-01-19 11:15:18 +01:00
Syncthing Release Automation
5f1e27bb7f gui, man, authors: Update docs, translations, and contributors 2023-01-16 03:48:02 +00:00
André Colomb
a85b4ab263 build: Go 1.19.5 2023-01-12 07:30:17 +04:00
Syncthing Release Automation
cf962a4fe8 gui, man, authors: Update docs, translations, and contributors 2023-01-09 03:45:24 +00:00
André Colomb
39c0bfa05a script: Add weblatedl.go for downloading updated translations (#8723)
Based on the transifexdl.go script.  Map Weblate language codes to the
existing ones during import.  Do not delete incomplete languages from
repo.
2023-01-08 15:25:00 +01:00
tomasz1986
458d6cff2a gui: Allow to translate action and type in Recent Changes modal (#8548)
Currently, action and type are both displayed only in English. This
commit makes it possible to translate both of them.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-01-05 14:33:03 +01:00
Syncthing Release Automation
ded881c372 gui, man, authors: Update docs, translations, and contributors 2023-01-02 03:45:30 +00:00
tomasz1986
fb4209e382 gui: Fix undefined lastSeenDays error in disconnected-inactive status check (ref #8530) (#8730)
The current code assumes that lastSeenDays is always set which is not
the case when device reports its lastSeen as equal to the Unix epoch.
Thus, make sure that lastSeenDays is set before proceeding with the
disconnected-inactive status check to avoid JS errors in the browser.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2022-12-26 12:59:04 +01:00
Syncthing Release Automation
473ca68dc4 gui, man, authors: Update docs, translations, and contributors 2022-12-26 03:45:20 +00:00
Jakob Borg
c4e69cd66c gui, api: Indicate running under container (#8728)
This adds a word to the version string when running containerized. The
purpose is mostly to facilitate troubleshooting via screenshot by
"leaking" this rather important aspect of the setup. Additionally, the
version row gets "no-overflow-ellipsis" treatment so that the whole
thing is actually visible in the GUI and the (now useless) tooltip is
removed. In production releases this won't make a difference as the
whole thing will typically fit, but in odd setups it provides more info
up front.
2022-12-25 08:08:41 +01:00
Simon Frei
634a3d0e3b lib/fs: Use io/fs errors as recommended in std lib (#8726) 2022-12-21 23:42:22 +01:00
Jakob Borg
09f4d865ae build: Handle co-authors (ref #3744) (#8708)
The authorship script didn't pick up people who were only ever
"co-authors" of a commit, such as when they wrote stuff which was later
included in a PR by someone else, or added code during code review.

This modified the script to look closer in the commit bodies for
"Co-authored-by:"-lines and adds those found to the set of authors.
2022-12-21 22:02:44 +01:00
Jakob Borg
ad0044fec8 lib/fs: Watching is unsupported on android/amd64 (fixes #8709) (#8710) 2022-12-21 22:01:00 +01:00
Simon Frei
d157d12037 lib/model: Only log at info level if setting change time fails (#8725) 2022-12-21 21:58:35 +01:00
Aleksey Vasenev
f9d68474ac lib/model: Don't lower rescan interval from default on auto accepted enc folder (fixes #8572) (#8573) 2022-12-21 19:52:22 +01:00
Syncthing Release Automation
f0126fe1ab gui, man, authors: Update docs, translations, and contributors 2022-12-19 03:45:31 +00:00
André Colomb
940536a964 gui: Remove unmaintained language variant nl-BE (#8722)
gui: Remove unmaintained language variant nl-BE.

This has long been removed from Transifex and from the valid-langs.js
list, but somehow the translation file stayed in the repo.  There is
already a generic Dutch variant (nl) available as replacement.
2022-12-18 08:30:26 +01:00
André Colomb
f378c4f191 gui, script: Fix indentation in lang-en.json to match others (#8721)
* Indent lang-en.json with four spaces as all other languages.

* Regenerate lang-en.json.

Removes two no longer used strings.
2022-12-18 08:29:59 +01:00
James O'Beirne
358d2143e2 docker: Ensure entrypoint is executable (#8719)
On systems with safe umasks (`umask 077`), the entrypoint as copied from
the host may not be executable by other users. Ensure that it is set to
be within the Dockerfile.
2022-12-16 08:16:22 +01:00
Jakob Borg
622ade40ad Go 1.19.4 2022-12-13 14:03:06 +01:00
Jakob Borg
323405ea07 Update dependencies (#8717) 2022-12-12 17:20:33 +01:00
Syncthing Release Automation
88c226ef87 gui, man, authors: Update docs, translations, and contributors 2022-12-12 03:48:00 +00:00
Jakob Borg
d4b668caac docker: Update healthcheck cmd for buildx (ref #8640) (#8706) 2022-12-06 21:23:49 +01:00
Aleksey Vasenev
05738001ac lib/model: Fixed adding empty items to device list (fixes #8646) (#8647) 2022-12-06 21:22:35 +01:00
Jakob Borg
d40d9d5698 build: Update dependencies (#8704) 2022-12-06 11:39:43 +01:00
Syncthing Release Automation
648c71cbd1 gui, man, authors: Update docs, translations, and contributors 2022-12-05 03:45:25 +00:00
André Colomb
ab0eb909a2 gui, lib/connections: Let the backend decide whether connection is local (fixes #8686) (#8694)
* lib/connections: Cache isLAN decision for later external access.

The check whether a remote device's address is on a local network
currently happens when handling the Hello message, to configure the
limiters.  Save the result to the ConnectionInfo and pass it out as
part of the model's ConnectionInfo struct in ConnectionStats().

* gui: Use provided connection attribute to distinguish LAN / WAN.

Replace the dumb IP address check which didn't catch common cases and
actually could contradict what the backend decided.  That could have
been confusing if the GUI says WAN, but the limiter is not actually
applied because the backend thinks it's a LAN.

Add strings for QUIC and relay connections to also differentiate
between LAN and WAN.

* gui: Redefine reception level icons for all connection types.

Move the mapping to the JS code, as it is much easier to handle
multiple switch cases by fall-through there.

QUIC is regarded no less than TCP anymore.  LAN and WAN make the
difference between levels 4 / 3 and 2 / 1:

{TCP,QUIC} LAN --> {TCP,QUIC} WAN --> Relay LAN --> Relay WAN -->
Disconnected.
2022-11-28 09:28:33 +01:00
668 changed files with 39695 additions and 46266 deletions

2
.github/CODEOWNERS vendored
View File

@@ -1,2 +0,0 @@
/AUTHORS @calmh
/*.md @calmh

View File

@@ -1,13 +0,0 @@
---
name: Feature request
about: If you're just not sure how to do something, see "ask a question".
labels: enhancement, needs-triage
---
### Include required information
Please be sure to include at least:
- what problem your new feature would solve
- how or why you think it is generally useful (i.e., not just for you)
- what alternatives or workarounds you considered

28
.github/ISSUE_TEMPLATE/01-feature.yml vendored Normal file
View File

@@ -0,0 +1,28 @@
name: Feature request
description: File a new feature request
labels: ["enhancement", "needs-triage"]
body:
- type: textarea
id: feature
attributes:
label: Feature description
description: Please describe the behavior you'd like to see.
validations:
required: true
- type: textarea
id: problem-usecase
attributes:
label: Problem or use case
description: Please explain which problem this would solve, or what the use case is for the feature. Keep in mind that it's more likely to be implemented if it's generally useful for a larger number of users.
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives or workarounds
description: Please describe any alternatives or workarounds you have considered and, possibly, rejected.
validations:
required: true

View File

@@ -1,23 +0,0 @@
---
name: Bug report
about: If you're actually looking for support, see "ask a question".
labels: bug, needs-triage
---
### Does your log mention database corruption?
If your Syncthing log reports panics because of database corruption it is
most likely a fault with your system's storage or memory. Affected log
entries will contain lines starting with `panic: leveldb`. You will need to
delete the index database to clear this, by running `syncthing
-reset-database`.
### Include required information
Please be sure to include at least:
- which version of Syncthing and what operating system you are using
- browser and version, if applicable
- what happened,
- what you expected to happen instead, and
- any steps to reproduce the problem.

51
.github/ISSUE_TEMPLATE/02-bug.yml vendored Normal file
View File

@@ -0,0 +1,51 @@
name: Bug report
description: If you're actually looking for support instead, see "I need help / I have a question".
labels: ["bug", "needs-triage"]
body:
- type: markdown
attributes:
value: |
:no_entry_sign: If you want to report a security issue, please see [our Security Policy](https://syncthing.net/security/) and do not report the issue here.
:interrobang: If you are not sure if there is a bug, but something isn't working right and you need help, please [use the forum](https://forum.syncthing.net/).
- type: textarea
id: what-happened
attributes:
label: What happened?
description: Also tell us, what did you expect to happen, and any steps we might use to reproduce the problem.
placeholder: Tell us what you see!
validations:
required: true
- type: input
id: version
attributes:
label: Syncthing version
description: What version of Syncthing are you running?
placeholder: v1.27.4
validations:
required: true
- type: input
id: platform
attributes:
label: Platform & operating system
description: On what platform(s) are you seeing the problem?
placeholder: Linux arm64
validations:
required: true
- type: input
id: browser
attributes:
label: Browser version
description: If the problem is related to the GUI, describe your browser and version.
placeholder: Safari 17.3.1
- type: textarea
id: logs
attributes:
label: Relevant log output
description: Please copy and paste any relevant log output or crash backtrace. This will be automatically formatted into code, so no need for backticks.
render: shell

View File

@@ -1,13 +1,13 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: weekly
open-pull-requests-limit: 10
- package-ecosystem: "gomod"
directory: "/"
schedule:
interval: weekly
open-pull-requests-limit: 10
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: monthly
open-pull-requests-limit: 10
- package-ecosystem: "gomod"
directory: "/"
schedule:
interval: monthly
open-pull-requests-limit: 10

View File

@@ -0,0 +1,85 @@
name: Build Infrastructure Images
on:
push:
branches:
- infrastructure
- infra-*
env:
GO_VERSION: "~1.24.0"
CGO_ENABLED: "0"
BUILD_USER: docker
BUILD_HOST: github.syncthing.net
permissions:
contents: read
packages: write
jobs:
docker-syncthing:
name: Build and push Docker images
if: github.repository == 'syncthing/syncthing'
runs-on: ubuntu-latest
environment: release
strategy:
matrix:
pkg:
- stcrashreceiver
- strelaypoolsrv
- stupgrades
- ursrv
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}
check-latest: true
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build binaries
run: |
for arch in arm64 amd64; do
go run build.go -goos linux -goarch "$arch" build ${{ matrix.pkg }}
mv ${{ matrix.pkg }} ${{ matrix.pkg }}-linux-"$arch"
done
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Set Docker tags (all branches)
run: |
tags=docker.io/syncthing/${{ matrix.pkg }}:${{ github.sha }},ghcr.io/syncthing/infra/${{ matrix.pkg }}:${{ github.sha }}
echo "TAGS=$tags" >> $GITHUB_ENV
- name: Set Docker tags (latest)
if: github.ref == 'refs/heads/infrastructure'
run: |
tags=docker.io/syncthing/${{ matrix.pkg }}:latest,ghcr.io/syncthing/infra/${{ matrix.pkg }}:latest,${{ env.TAGS }}
echo "TAGS=$tags" >> $GITHUB_ENV
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
file: ./Dockerfile.${{ matrix.pkg }}
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ env.TAGS }}
labels: |
org.opencontainers.image.revision=${{ github.sha }}

18
.github/workflows/build-nightly.yaml vendored Normal file
View File

@@ -0,0 +1,18 @@
name: Build Syncthing (Nightly)
on:
schedule:
# Run nightly build at 05:00 UTC
- cron: '00 05 * * *'
workflow_dispatch:
permissions:
contents: write
packages: write
jobs:
build-syncthing:
uses: ./.github/workflows/build-syncthing.yaml
# if we only want nightlies to run for specific users:
# if: contains(fromJSON('["syncthing", "calmh"]'), github.repository_owner)
secrets: inherit

View File

File diff suppressed because it is too large Load Diff

21
.github/workflows/trigger-nightly.yaml vendored Normal file
View File

@@ -0,0 +1,21 @@
name: Trigger nightly build & release
on:
workflow_dispatch:
schedule:
# Run nightly build at 01:00 UTC
- cron: '00 01 * * *'
jobs:
trigger-nightly:
runs-on: ubuntu-latest
name: Push to release-nightly to trigger build
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.ACTIONS_GITHUB_TOKEN }}
fetch-depth: 0
- run: |
git push origin main:release-nightly

View File

@@ -2,7 +2,7 @@ name: Update translations and documentation
on:
workflow_dispatch:
schedule:
- cron: '42 3 * * 1'
- cron: '42 3 * * 1'
jobs:
@@ -10,13 +10,13 @@ jobs:
runs-on: ubuntu-latest
name: Update translations and documentation
steps:
- uses: actions/checkout@629c2de402a417ea7690ca6ce3f33229e27606a5 # v2
- uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.ACTIONS_GITHUB_TOKEN }}
- uses: actions/setup-go@268d8c0ca0432bb2cf416faae41297df9d262d7f # v2
- uses: actions/setup-go@v5
with:
go-version: ^1.18.4
go-version: stable
- run: |
set -euo pipefail
git config --global user.name 'Syncthing Release Automation'
@@ -25,5 +25,4 @@ jobs:
bash build.sh prerelease
git push
env:
TRANSIFEX_USER: ${{ secrets.TRANSIFEX_USER }}
TRANSIFEX_PASS: ${{ secrets.TRANSIFEX_PASS }}
WEBLATE_TOKEN: ${{ secrets.WEBLATE_TOKEN }}

5
.gitignore vendored
View File

@@ -1,11 +1,10 @@
/syncthing
/stdiscosrv
syncthing.exe
stdiscosrv.exe
*.tar.gz
*.zip
*.asc
*.deb
*.exe
.jshintrc
coverage.out
files/pidx
@@ -19,4 +18,4 @@ deb
/repos
/proto/scripts/protoc-gen-gosyncthing
/gui/next-gen-gui
.idea
/compat.json

View File

@@ -1,26 +1,39 @@
linters-settings:
maligned:
suggest-new: true
linters:
enable-all: true
disable:
- goimports
- cyclop
- depguard
- lll
- gochecknoinits
- gochecknoglobals
- gofmt
- scopelint
- gocyclo
- exhaustive
- exhaustruct
- funlen
- wsl
- gci
- gochecknoglobals
- gochecknoinits
- gocognit
- goconst
- gocyclo
- godox
- gofmt
- goimports
- gomoddirectives
- inamedparam
- interfacebloat
- ireturn
- lll
- maintidx
- nestif
- nonamedreturns
- paralleltest
- protogetter
- scopelint
- tagalign
- tagliatelle
- testpackage
- varnamelen
- wsl
service:
golangci-lint-version: 1.21.x
prepare:
- rm -f go.sum # 1.12 -> 1.13 issues with QUIC-go
- GO111MODULE=on go mod vendor
- go run build.go assets
issues:
exclude-dirs:
- internal/gen
- cmd/dev
- repos

98
.policy.yml Normal file
View File

@@ -0,0 +1,98 @@
# This is the policy-bot configuration for this repository. It controls
# which approvals are required for any given pull request. The format is
# described at https://github.com/palantir/policy-bot. The syntax of the
# policy can be verified by the bot:
# curl https://pb.syncthing.net/api/validate -X PUT -T .policy.yml
# The policy below is what is required for any pull request.
policy:
approval:
- subject is conventional commit
- project metadata requires maintainer approval
- or:
- is approved by a syncthing contributor
- is a translation or dependency update by a contributor
- is a trivial change by a contributor
# Additionally, contributors can disapprove of a PR
disapproval:
requires:
teams:
- syncthing/contributors
# The rules for the policy are described below.
approval_rules:
# All commits (PRs before squashing) should have a valid conventional
# commit type subject.
- name: subject is conventional commit
requires:
conditions:
title:
matches:
- '^(feat|fix|docs|chore|refactor|build): [a-z].+'
- '^(feat|fix|docs|chore|refactor|build)\(\w+(, \w+)*\): [a-z].+'
# Changes to important project metadata and documentation, including this
# policy, require signoff by a maintainer
- name: project metadata requires maintainer approval
if:
changed_files:
paths:
- ^[^/]+\.md
- ^\.policy\.yml
- ^LICENSE
requires:
count: 1
teams:
- syncthing/maintainers
options:
ignore_update_merges: true
allow_contributor: true
# Regular pull requests require approval by an active contributor
- name: is approved by a syncthing contributor
requires:
count: 1
teams:
- syncthing/contributors
options:
ignore_update_merges: true
allow_contributor: true
# Changes to some files (translations, dependencies, compatibility) do not
# require approval if they were proposed by a contributor and have a
# matching commit subject
- name: is a translation or dependency update by a contributor
if:
only_changed_files:
paths:
- ^gui/default/assets/lang/
- ^go\.mod$
- ^go\.sum$
- ^compat\.yaml$
title:
matches:
- '^chore\(gui\):'
- '^build\(deps\):'
- '^build\(compat\):'
has_author_in:
teams:
- syncthing/contributors
# If the change is small and the label "trivial" is added, we accept that
# on trust. These PRs can be audited after the fact as appropriate.
# Features are not trivial.
- name: is a trivial change by a contributor
if:
modified_lines:
total: "< 25"
title:
not_matches:
- '^feat'
has_labels:
- trivial
has_author_in:
teams:
- syncthing/contributors

4
.yamlfmt Normal file
View File

@@ -0,0 +1,4 @@
line_ending: lf
formatter:
type: basic
retain_line_breaks: true

64
AUTHORS
View File

@@ -20,12 +20,17 @@ Alan Pope <alan@popey.com>
Alberto Donato <albertodonato@users.noreply.github.com>
Aleksey Vasenev <margtu-fivt@ya.ru>
Alessandro G. (alessandro.g89) <alessandro.g89@gmail.com>
Alex Ionescu <github@ionescu.sh>
Alex Lindeman <139387+aelindeman@users.noreply.github.com>
Alex Xu <alex.hello71@gmail.com>
Alexander Graf (alex2108) <register-github@alex-graf.de>
Alexander Seiler <seileralex@gmail.com>
Alexandre Alves <alexandrealvesdb.contact@gmail.com>
Alexandre Viau (aviau) <alexandre@alexandreviau.net> <aviau@debian.org>
Aman Gupta <aman@tmm1.net>
Anatoli Babenia <anatoli@rainforce.org>
Anderson Mesquita (andersonvom) <andersonvom@gmail.com>
Andreas Sommer <andreas.sommer87@googlemail.com>
andresvia <andres.via@gmail.com>
Andrew Dunham (andrew-d) <andrew@du.nham.ca>
Andrew Meyer <andrewm.bpi@gmail.com>
@@ -34,6 +39,7 @@ Andrey D (scienmind) <scintertech@cryptolab.net> <scienmind@users.noreply.github
André Colomb (acolomb) <src@andre.colomb.de> <github.com@andre.colomb.de>
andyleap <andyleap@gmail.com>
Anjan Momi <anjan@momi.ca>
Anthony Goeckner <agoeckner@users.noreply.github.com>
Antoine Lamielle (0x010C) <antoine.lamielle@0x010c.fr> <gh@0x010c.fr>
Antony Male (canton7) <antony.male@gmail.com>
Anur <anurnomeru@163.com>
@@ -46,6 +52,7 @@ Audrius Butkevicius (AudriusButkevicius) <audrius.butkevicius@gmail.com> <github
Aurélien Rainone <476650+arl@users.noreply.github.com>
BAHADIR YILMAZ <bahadiryilmaz32@gmail.com>
Bart De Vries (mogwa1) <devriesb@gmail.com>
Beat Reichenbach <44111292+beatreichenbach@users.noreply.github.com>
Ben Curthoys (bencurthoys) <ben@bencurthoys.com>
Ben Schulz (uok) <ueomkail@gmail.com> <uok@users.noreply.github.com>
Ben Shepherd (benshep) <bjashepherd@gmail.com>
@@ -64,50 +71,63 @@ Brian R. Becker (brbecker) <brbecker@gmail.com>
bt90 <btom1990@googlemail.com>
Caleb Callaway (cqcallaw) <enlightened.despot@gmail.com>
Carsten Hagemann (carstenhag) <moter8@gmail.com> <carsten@chagemann.de>
Catfriend1 <16361913+Catfriend1@users.noreply.github.com>
Cathryne Linenweaver (Cathryne) <cathryne.linenweaver@gmail.com> <Cathryne@users.noreply.github.com> <katrinleinweber@MAC.local>
Cedric Staniewski (xduugu) <cedric@gmx.ca>
chenrui <rui@meetup.com>
Chih-Hsuan Yen <yan12125@gmail.com>
Chih-Hsuan Yen <yan12125@gmail.com> <1937689+yan12125@users.noreply.github.com>
Choongkyu <choongkyu.kim+gh@gmail.com> <vapidlyrapid+gh@gmail.com>
Chris Howie (cdhowie) <me@chrishowie.com>
Chris Joel (cdata) <chris@scriptolo.gy>
Chris Tonkinson <chris@masterbran.ch>
Christian Kujau <ckujau@users.noreply.github.com>
Christian Prescott <me@christianprescott.com>
chucic <chucic@seznam.cz>
cjc7373 <niuchangcun@gmail.com>
Colin Kennedy (moshen) <moshen.colin@gmail.com>
Cromefire_ <tim.l@nghorst.net> <26320625+cromefire@users.noreply.github.com>
cui fliter <imcusg@gmail.com>
Cyprien Devillez <cypx@users.noreply.github.com>
d-volution <49024624+d-volution@users.noreply.github.com>
Dale Visser <dale.visser@live.com>
Dan <benda.daniel@gmail.com>
Daniel Barczyk <46358936+DanielBarczyk@users.noreply.github.com>
Daniel Bergmann (brgmnn) <dan.arne.bergmann@gmail.com> <brgmnn@users.noreply.github.com>
Daniel Harte (norgeous) <daniel@harte.me> <daniel@danielharte.co.uk> <norgeous@users.noreply.github.com>
Daniel Martí (mvdan) <mvdan@mvdan.cc>
Daniel Padrta <64928366+danpadcz@users.noreply.github.com>
Darshil Chanpura (dtchanpura) <dtchanpura@gmail.com> <dcprime314@gmail.com>
dashangcun <907225865@qq.com>
David Rimmer (dinosore) <dinosore@dbrsoftware.co.uk>
deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
DeflateAwning <11021263+DeflateAwning@users.noreply.github.com>
Denis A. (dva) <denisva@gmail.com>
Dennis Wilson (snnd) <dw@risu.io>
dependabot-preview[bot] <dependabot-preview[bot]@users.noreply.github.com> <27856297+dependabot-preview[bot]@users.noreply.github.com>
dependabot[bot] <dependabot[bot]@users.noreply.github.com> <49699333+dependabot[bot]@users.noreply.github.com>
derekriemer <derek.riemer@colorado.edu>
DerRockWolf <50499906+DerRockWolf@users.noreply.github.com>
desbma <desbma@users.noreply.github.com>
Devon G. Redekopp <devon@redekopp.com>
diemade <spamkill@posteo.ch>
digital <didev@dinid.net>
Dimitri Papadopoulos Orfanos <3234522+DimitriPapadopoulos@users.noreply.github.com>
Dmitry Saveliev (dsaveliev) <d.e.saveliev@gmail.com>
Domenic Horner <domenic@tgxn.net>
Dominik Heidler (asdil12) <dominik@heidler.eu>
Elias Jarlebring (jarlebring) <jarlebring@gmail.com>
Elliot Huffman <thelich2@gmail.com>
Emil Hessman (ceh) <emil@hessman.se>
Emil Lundberg <emil@emlun.se>
Eng Zer Jun <engzerjun@gmail.com>
entity0xfe <109791748+entity0xfe@users.noreply.github.com>
entity0xfe <109791748+entity0xfe@users.noreply.github.com> <entity0xfe@my.domain>
Eric Lesiuta <elesiuta@gmail.com>
Eric P <eric@kastelo.net>
Erik Meitner (WSGCSysadmin) <e.meitner@willystreet.coop>
Evan Spensley <94762716+0evan@users.noreply.github.com>
Evgeny Kuznetsov <evgeny@kuznetsov.md>
Federico Castagnini (facastagnini) <federico.castagnini@gmail.com>
Felix <53702818+f-eliks@users.noreply.github.com>
Felix Ableitner (Nutomic) <me@nutomic.com>
Felix Lampe <mail@flampe.de>
Felix Unterpaintner (bigbear2nd) <bigbear2nd@gmail.com>
@@ -121,10 +141,14 @@ Gleb Sinyavskiy <zhulik.gleb@gmail.com>
Graham Miln (grahammiln) <graham.miln@dssw.co.uk> <graham.miln@miln.eu>
greatroar <61184462+greatroar@users.noreply.github.com>
Greg <gco@jazzhaiku.com>
guangwu <guoguangwu@magic-shield.com>
gudvinr <gudvinr@gmail.com>
Gusted <postmaster@gusted.xyz> <williamzijl7@hotmail.com>
Han Boetes <han@boetes.org>
HansK-p <42314815+HansK-p@users.noreply.github.com>
Harrison Jones (harrisonhjones) <harrisonhjones@users.noreply.github.com>
Heiko Zuerker (Smiley73) <heiko@zuerker.org>
Hireworks <129852174+hireworksltd@users.noreply.github.com>
Hugo Locurcio <hugo.locurcio@hugo.pro>
Iain Barnett <iainspeed@gmail.com>
Ian Johnson (anonymouse64) <ian.johnson@canonical.com> <person.uwsome@gmail.com>
@@ -137,14 +161,17 @@ Jacek Szafarkiewicz (hadogenes) <szafar@linux.pl>
Jack Croft <jccroft1@users.noreply.github.com>
Jacob <jyundt@gmail.com>
Jake Peterson (acogdev) <jake@acogdev.com>
Jakob Borg (calmh) <jakob@nym.se> <jakob@kastelo.net>
Jakob Borg (calmh) <jakob@nym.se> <jakob@kastelo.net> <jborg@coreweave.com>
James O'Beirne <wild-github@au92.org>
James Patterson (jpjp) <jamespatterson@operamail.com> <jpjp@users.noreply.github.com>
janost <janost@tuta.io>
Jaroslav Lichtblau <svetlemodry@users.noreply.github.com>
Jaroslav Malec (dzarda) <dzardacz@gmail.com>
jaseg <githubaccount@jaseg.net>
Jaspitta <ste.scarpitta@gmail.com>
Jauder Ho <jauderho@users.noreply.github.com>
Jaya Chithra (jayachithra) <s.k.jayachithra@gmail.com>
Jaya Kumar <jaya.kumar@ict.nl>
Jeffery To <jeffery.to@gmail.com>
jelle van der Waa <jelle@vdwaa.nl>
Jens Diemer (jedie) <github.com@jensdiemer.de> <git@jensdiemer.de>
@@ -160,11 +187,15 @@ Jonathan Cross <jcross@gmail.com>
Jonta <359397+Jonta@users.noreply.github.com>
Jose Manuel Delicado (jmdaweb) <jmdaweb@hotmail.com> <jmdaweb@users.noreply.github.com>
jtagcat <git-514635f7@jtag.cat> <git-12dbd862@jtag.cat>
Julian Lehrhuber <jul13579@users.noreply.github.com>
Jörg Thalheim <Mic92@users.noreply.github.com>
Jędrzej Kula <kula.jedrek@gmail.com>
K.B.Dharun Krishna <kbdharunkrishna@gmail.com>
Kalle Laine <pahakalle@protonmail.com>
Kapil Sareen <kapilsareen584@gmail.com>
Karol Różycki (krozycki) <rozycki.karol@gmail.com>
Kebin Liu <lkebin@gmail.com>
Keith Harrison <keithh@protonmail.com>
Keith Turner <kturner@apache.org>
Kelong Cong (kc1212) <kc04bc@gmx.com> <kc1212@users.noreply.github.com>
Ken'ichi Kamada (kamadak) <kamada@nanohz.org>
@@ -173,6 +204,7 @@ Kevin Bushiri (keevBush) <keevbush@gmail.com> <36192217+keevBush@users.noreply.g
Kevin White, Jr. (kwhite17) <kevinwhite1710@gmail.com>
klemens <ka7@github.com>
Kurt Fitzner (Kudalufi) <kurt@va1der.ca> <kurt.fitzner@gmail.com>
kylosus <33132401+kylosus@users.noreply.github.com>
Lars K.W. Gohlke (lkwg82) <lkwg82@gmx.de>
Lars Lehtonen <lars.lehtonen@gmail.com>
Laurent Arnoud <laurent@spkdev.net>
@@ -182,7 +214,9 @@ Liu Siyuan (liusy182) <liusy182@gmail.com> <liusy182@hotmail.com>
Lode Hoste (Zillode) <zillode@zillode.be>
Lord Landon Agahnim (LordLandon) <lordlandon@gmail.com>
LSmithx2 <42276854+lsmithx2@users.noreply.github.com>
luchenhan <168071714+luchenhan@users.noreply.github.com>
Lukas Lihotzki <lukas@lihotzki.de>
Luke Hamburg <1992842+luckman212@users.noreply.github.com>
luzpaz <luzpaz@users.noreply.github.com>
Majed Abdulaziz (majedev) <majed.alhajry@gmail.com>
Marc Laporte (marclaporte) <marc@marclaporte.com> <marc@laporte.name>
@@ -193,8 +227,10 @@ Marcus Legendre <marcus.legendre@gmail.com>
Mario Majila <mariustshipichik@gmail.com>
Mark Pulford (mpx) <mark@kyne.com.au>
Martchus <martchus@gmx.net>
Martin Polehla <p0l0us@users.noreply.github.com>
Mateusz Naściszewski (mateon1) <matin1111@wp.pl>
Mateusz Ż <thedead4fun@live.com>
mathias4833 <67101597+mathias4833@users.noreply.github.com>
Matic Potočnik <hairyfotr@gmail.com>
Matt Burke (burkemw3) <mburke@amplify.com> <burkemw3@gmail.com>
Matt Robenolt <matt@ydekproductions.com>
@@ -202,14 +238,17 @@ Matteo Ruina <matteo.ruina@gmail.com>
Maurizio Tomasi <ziotom78@gmail.com>
Max <github@germancoding.com>
Max Schulze (kralo) <max.schulze@online.de> <kralo@users.noreply.github.com>
maxice8 <30738253+maxice8@users.noreply.github.com>
MaximAL <almaximal@ya.ru>
Maxime Thirouin <m@moox.io>
Maximilian <maxi.rostock@outlook.de> <public@complexvector.space>
mclang <1721600+mclang@users.noreply.github.com>
Michael Jephcote (Rewt0r) <rewt0r@gmx.com> <Rewt0r@users.noreply.github.com>
Michael Ploujnikov (plouj) <ploujj@gmail.com>
Michael Rienstra <mrienstra@gmail.com>
Michael Tilli (pyfisch) <pyfisch@gmail.com>
MichaIng <micha@dietpi.com>
Migelo <miha@filetki.si>
Mike Boone <mike@boonedocks.net>
MikeLund <MikeLund@users.noreply.github.com>
MikolajTwarog <43782609+MikolajTwarog@users.noreply.github.com>
@@ -217,6 +256,7 @@ Mingxuan Lin <gdlmx@users.noreply.github.com>
mv1005 <49659413+mv1005@users.noreply.github.com>
Nate Morrison (nrm21) <natemorrison@gmail.com>
Naveen <172697+naveensrinivasan@users.noreply.github.com>
nf <nf@wh3rd.net>
Nicholas Rishel (PrototypeNM1) <rishel.nick@gmail.com> <PrototypeNM1@users.noreply.github.com>
Nick Busey <NickBusey@users.noreply.github.com>
Nico Stapelbroek <3368018+nstapelbroek@users.noreply.github.com>
@@ -228,6 +268,7 @@ NinoM4ster <ninom4ster@gmail.com>
Nitroretro <43112364+Nitroretro@users.noreply.github.com>
NoLooseEnds <jon.koslung@gmail.com>
Oliver Freyermuth <o.freyermuth@googlemail.com>
orangekame3 <miya.org.0309@gmail.com>
otbutz <tbutz@optitool.de>
Otiel <Otiel@users.noreply.github.com>
overkill <22098433+0verk1ll@users.noreply.github.com>
@@ -248,6 +289,7 @@ Philippe Schommers (filoozoom) <philippe@schommers.be>
Phill Luby (pluby) <phill.luby@newredo.com>
Pier Paolo Ramon <ramonpierre@gmail.com>
Piotr Bejda (piobpl) <piotrb10@gmail.com>
polyfloyd <polyfloyd@users.noreply.github.com>
Pramodh KP (pramodhkp) <pramodh.p@directi.com> <1507241+pramodhkp@users.noreply.github.com>
Quentin Hibon <qh.public@yahoo.com>
Rahmi Pruitt <rjpruitt16@gmail.com>
@@ -266,36 +308,50 @@ Sacheendra Talluri (sacheendra) <sacheendra.t@gmail.com>
Scott Klupfel (kluppy) <kluppy@going2blue.com>
sec65 <106604020+sec65@users.noreply.github.com>
Sergey Mishin (ralder) <ralder@yandex.ru>
Shaarad Dalvi <60266155+shaaraddalvi@users.noreply.github.com>
Sertonix <83883937+Sertonix@users.noreply.github.com>
Severin von Wnuck-Lipinski <ss7@live.de>
Shaarad Dalvi <60266155+shaaraddalvi@users.noreply.github.com> <shdalv@microsoft.com>
Simon Frei (imsodin) <freisim93@gmail.com>
Simon Mwepu <simonmwepu@gmail.com>
Simon Pickup <simon@pickupinfinity.com>
Sly_tom_cat <slytomcat@mail.ru>
Sonu Kumar Saw <31889738+dev-saw99@users.noreply.github.com>
Stefan Kuntz (Stefan-Code) <stefan.github@gmail.com> <Stefan.github@gmail.com>
Stefan Tatschner (rumpelsepp) <stefan@sevenbyte.org> <rumpelsepp@sevenbyte.org> <stefan@rumpelsepp.org>
Steven Eckhoff <steven.eckhoff.opensource@gmail.com>
Suhas Gundimeda (snugghash) <suhas.gundimeda@gmail.com> <snugghash@gmail.com>
Sven Bachmann <dev@mcbachmann.de>
Syncthing Automation <automation@syncthing.net>
Syncthing Release Automation <release@syncthing.net>
Taylor Khan (nelsonkhan) <nelsonkhan@gmail.com>
Terrance <git@terrance.allofti.me>
Thomas <9749173+uhthomas@users.noreply.github.com>
Thomas Hipp <thomashipp@gmail.com>
Tim Abell (timabell) <tim@timwise.co.uk>
Tim Howes (timhowes) <timhowes@berkeley.edu>
Tim Nordenfur <tim@gurka.se>
Tobias Frölich <40638719+tobifroe@users.noreply.github.com>
Tobias Klauser <tobias.klauser@gmail.com>
Tobias Nygren (tnn2) <tnn@nygren.pp.se>
Tobias Tom (tobiastom) <t.tom@succont.de>
Tom Jakubowski <tom@crystae.net>
Tomasz Wilczyński <5626656+tomasz1986@users.noreply.github.com> <twilczynski@naver.com>
Tommy Thorn <tommy-github-email@thorn.ws>
Tommy van der Vorst <tommy-github@pixelspark.nl> <tommy@pixelspark.nl>
Tully Robinson (tojrobinson) <tully@tojr.org>
Tyler Brazier (tylerbrazier) <tyler@tylerbrazier.com>
Tyler Kropp <kropptyler@gmail.com>
Unrud (Unrud) <unrud@openaliasbox.org> <Unrud@users.noreply.github.com>
vapatel2 <149737089+vapatel2@users.noreply.github.com>
Veeti Paananen (veeti) <veeti.paananen@rojekti.fi>
Victor Buinsky (buinsky) <vix_booja@tut.by>
Vik <63919734+ViktorOn@users.noreply.github.com>
Vil Brekin (Vilbrekin) <vilbrekin@gmail.com>
villekalliomaki <53118179+villekalliomaki@users.noreply.github.com>
Vladimir Rusinov <vrusinov@google.com> <vladimir.rusinov@gmail.com>
wangguoliang <liangcszzu@163.com>
WangXi <xib1102@icloud.com>
Will Rouesnel <wrouesnel@wrouesnel.com>
William A. Kennington III (wkennington) <william@wkennington.com>
wouter bolsterlee <wouter@bolsterl.ee>
Wulf Weich (wweich) <wweich@users.noreply.github.com> <wweich@gmx.de> <wulf@weich-kr.de>

View File

@@ -24,10 +24,15 @@ too much information will never get you yelled at. :)
## Contributing Translations
All translations are done via
[Transifex](https://www.transifex.com/projects/p/syncthing/). If you
wish to contribute to a translation, just head over there and sign up.
[Weblate](https://hosted.weblate.org/projects/syncthing/). If you wish
to contribute to a translation, just head over there and sign up.
Before every release, the language resources are updated from the
latest info on Transifex.
latest info on Weblate.
Note that the previously used service at
[Transifex](https://www.transifex.com/projects/p/syncthing/) is being
retired and we kindly ask you to sign up on Weblate for continued
involvement.
## Contributing Code

View File

@@ -1,15 +1,41 @@
ARG GOVERSION=latest
#
# Maybe build Syncthing. This is a bit ugly as we can't make an entire
# section of the Dockerfile conditional, so we end up always pulling the
# golang image as builder. Then we check if the executable we need already
# exists (pre-built) otherwise we build it.
#
FROM golang:$GOVERSION AS builder
ARG BUILD_USER
ARG BUILD_HOST
ARG TARGETARCH
WORKDIR /src
COPY . .
ENV CGO_ENABLED=0
ENV BUILD_HOST=syncthing.net
ENV BUILD_USER=docker
RUN rm -f syncthing && go run build.go -no-upgrade build syncthing
RUN if [ ! -f syncthing-linux-$TARGETARCH ] ; then \
go run build.go -no-upgrade build syncthing ; \
mv syncthing syncthing-linux-$TARGETARCH ; \
fi
#
# The rest of the Dockerfile uses the binary from the builder, prebuilt or
# not.
#
FROM alpine
ARG TARGETARCH
LABEL org.opencontainers.image.authors="The Syncthing Project" \
org.opencontainers.image.url="https://syncthing.net" \
org.opencontainers.image.documentation="https://docs.syncthing.net" \
org.opencontainers.image.source="https://github.com/syncthing/syncthing" \
org.opencontainers.image.vendor="The Syncthing Project" \
org.opencontainers.image.licenses="MPL-2.0" \
org.opencontainers.image.title="Syncthing"
EXPOSE 8384 22000/tcp 22000/udp 21027/udp
@@ -17,7 +43,7 @@ VOLUME ["/var/syncthing"]
RUN apk add --no-cache ca-certificates curl libcap su-exec tzdata
COPY --from=builder /src/syncthing /bin/syncthing
COPY --from=builder /src/syncthing-linux-$TARGETARCH /bin/syncthing
COPY --from=builder /src/script/docker-entrypoint.sh /bin/entrypoint.sh
ENV PUID=1000 PGID=1000 HOME=/var/syncthing
@@ -26,4 +52,6 @@ HEALTHCHECK --interval=1m --timeout=10s \
CMD curl -fkLsS -m 2 127.0.0.1:8384/rest/noauth/health | grep -o --color=never OK || exit 1
ENV STGUIADDRESS=0.0.0.0:8384
ENTRYPOINT ["/bin/entrypoint.sh", "/bin/syncthing", "-home", "/var/syncthing/config"]
ENV STHOMEDIR=/var/syncthing/config
RUN chmod 755 /bin/entrypoint.sh
ENTRYPOINT ["/bin/entrypoint.sh", "/bin/syncthing"]

View File

@@ -1,6 +1,14 @@
ARG GOVERSION=latest
FROM golang:$GOVERSION
LABEL org.opencontainers.image.authors="The Syncthing Project" \
org.opencontainers.image.url="https://syncthing.net" \
org.opencontainers.image.documentation="https://docs.syncthing.net" \
org.opencontainers.image.source="https://github.com/syncthing/syncthing" \
org.opencontainers.image.vendor="The Syncthing Project" \
org.opencontainers.image.licenses="MPL-2.0" \
org.opencontainers.image.title="Syncthing Builder"
# FPM to build Debian packages
RUN apt-get update && apt-get install -y --no-install-recommends \
locales rubygems ruby-dev build-essential git \

View File

@@ -1,19 +0,0 @@
FROM alpine
ARG TARGETARCH
EXPOSE 8384 22000/tcp 22000/udp 21027/udp
VOLUME ["/var/syncthing"]
RUN apk add --no-cache ca-certificates su-exec tzdata libcap
COPY ./syncthing-linux-$TARGETARCH /bin/syncthing
COPY ./script/docker-entrypoint.sh /bin/entrypoint.sh
ENV PUID=1000 PGID=1000 HOME=/var/syncthing
HEALTHCHECK --interval=1m --timeout=10s \
CMD nc -z 127.0.0.1 8384 || exit 1
ENV STGUIADDRESS=0.0.0.0:8384
ENTRYPOINT ["/bin/entrypoint.sh", "/bin/syncthing", "-home", "/var/syncthing/config"]

View File

@@ -0,0 +1,16 @@
FROM alpine
ARG TARGETARCH
LABEL org.opencontainers.image.authors="The Syncthing Project" \
org.opencontainers.image.url="https://syncthing.net" \
org.opencontainers.image.documentation="https://docs.syncthing.net" \
org.opencontainers.image.source="https://github.com/syncthing/syncthing" \
org.opencontainers.image.vendor="The Syncthing Project" \
org.opencontainers.image.licenses="MPL-2.0" \
org.opencontainers.image.title="Syncthing Crash Receiver"
EXPOSE 8080
COPY stcrashreceiver-linux-${TARGETARCH} /bin/stcrashreceiver
ENTRYPOINT [ "/bin/stcrashreceiver" ]

View File

@@ -1,15 +1,28 @@
ARG GOVERSION=latest
FROM golang:$GOVERSION AS builder
ARG BUILD_USER
ARG BUILD_HOST
ARG TARGETARCH
WORKDIR /src
COPY . .
ENV CGO_ENABLED=0
ENV BUILD_HOST=syncthing.net
ENV BUILD_USER=docker
RUN rm -f stdiscosrv && go run build.go -no-upgrade build stdiscosrv
RUN if [ ! -f stdiscosrv-linux-$TARGETARCH ] ; then \
go run build.go -no-upgrade build stdiscosrv ; \
mv stdiscosrv stdiscosrv-linux-$TARGETARCH ; \
fi
FROM alpine
ARG TARGETARCH
LABEL org.opencontainers.image.authors="The Syncthing Project" \
org.opencontainers.image.url="https://syncthing.net" \
org.opencontainers.image.documentation="https://docs.syncthing.net" \
org.opencontainers.image.source="https://github.com/syncthing/syncthing" \
org.opencontainers.image.vendor="The Syncthing Project" \
org.opencontainers.image.licenses="MPL-2.0" \
org.opencontainers.image.title="Syncthing Discovery Server"
EXPOSE 19200 8443
@@ -17,7 +30,7 @@ VOLUME ["/var/stdiscosrv"]
RUN apk add --no-cache ca-certificates su-exec
COPY --from=builder /src/stdiscosrv /bin/stdiscosrv
COPY --from=builder /src/stdiscosrv-linux-$TARGETARCH /bin/stdiscosrv
COPY --from=builder /src/script/docker-entrypoint.sh /bin/entrypoint.sh
ENV PUID=1000 PGID=1000 HOME=/var/stdiscosrv

16
Dockerfile.strelaypoolsrv Normal file
View File

@@ -0,0 +1,16 @@
FROM alpine
ARG TARGETARCH
LABEL org.opencontainers.image.authors="The Syncthing Project" \
org.opencontainers.image.url="https://syncthing.net" \
org.opencontainers.image.documentation="https://docs.syncthing.net" \
org.opencontainers.image.source="https://github.com/syncthing/syncthing" \
org.opencontainers.image.vendor="The Syncthing Project" \
org.opencontainers.image.licenses="MPL-2.0" \
org.opencontainers.image.title="Syncthing Relay Pool Server"
EXPOSE 8080
COPY strelaypoolsrv-linux-${TARGETARCH} /bin/strelaypoolsrv
ENTRYPOINT ["/bin/strelaypoolsrv", "-listen", ":8080"]

View File

@@ -1,15 +1,28 @@
ARG GOVERSION=latest
FROM golang:$GOVERSION AS builder
ARG BUILD_USER
ARG BUILD_HOST
ARG TARGETARCH
WORKDIR /src
COPY . .
ENV CGO_ENABLED=0
ENV BUILD_HOST=syncthing.net
ENV BUILD_USER=docker
RUN rm -f strelaysrv && go run build.go -no-upgrade build strelaysrv
RUN if [ ! -f strelaysrv-linux-$TARGETARCH ] ; then \
go run build.go -no-upgrade build strelaysrv ; \
mv strelaysrv strelaysrv-linux-$TARGETARCH ; \
fi
FROM alpine
ARG TARGETARCH
LABEL org.opencontainers.image.authors="The Syncthing Project" \
org.opencontainers.image.url="https://syncthing.net" \
org.opencontainers.image.documentation="https://docs.syncthing.net" \
org.opencontainers.image.source="https://github.com/syncthing/syncthing" \
org.opencontainers.image.vendor="The Syncthing Project" \
org.opencontainers.image.licenses="MPL-2.0" \
org.opencontainers.image.title="Syncthing Relay Server"
EXPOSE 22067 22070
@@ -17,7 +30,7 @@ VOLUME ["/var/strelaysrv"]
RUN apk add --no-cache ca-certificates su-exec
COPY --from=builder /src/strelaysrv /bin/strelaysrv
COPY --from=builder /src/strelaysrv-linux-$TARGETARCH /bin/strelaysrv
COPY --from=builder /src/script/docker-entrypoint.sh /bin/entrypoint.sh
ENV PUID=1000 PGID=1000 HOME=/var/strelaysrv

16
Dockerfile.stupgrades Normal file
View File

@@ -0,0 +1,16 @@
FROM alpine
ARG TARGETARCH
LABEL org.opencontainers.image.authors="The Syncthing Project" \
org.opencontainers.image.url="https://syncthing.net" \
org.opencontainers.image.documentation="https://docs.syncthing.net" \
org.opencontainers.image.source="https://github.com/syncthing/syncthing" \
org.opencontainers.image.vendor="The Syncthing Project" \
org.opencontainers.image.licenses="MPL-2.0" \
org.opencontainers.image.title="Syncthing Upgrades"
EXPOSE 8080
COPY stupgrades-linux-${TARGETARCH} /bin/stupgrades
ENTRYPOINT [ "/bin/stupgrades" ]

16
Dockerfile.ursrv Normal file
View File

@@ -0,0 +1,16 @@
FROM alpine
ARG TARGETARCH
LABEL org.opencontainers.image.authors="The Syncthing Project" \
org.opencontainers.image.url="https://syncthing.net" \
org.opencontainers.image.documentation="https://docs.syncthing.net" \
org.opencontainers.image.source="https://github.com/syncthing/syncthing" \
org.opencontainers.image.vendor="The Syncthing Project" \
org.opencontainers.image.licenses="MPL-2.0" \
org.opencontainers.image.title="Syncthing Usage Reporting Server"
EXPOSE 8080
COPY ursrv-linux-${TARGETARCH} /bin/ursrv
ENTRYPOINT [ "/bin/ursrv" ]

View File

@@ -24,17 +24,17 @@ to avoid corrupting the user's files.
### 2. Secure Against Attackers
Again, protecting the user's data is paramount. Regardless of our other
goals we must never allow the user's data to be susceptible to eavesdropping
goals, we must never allow the user's data to be susceptible to eavesdropping
or modification by unauthorized parties.
> This should be understood in context. It is not necessarily reasonable to
> expect Syncthing to be resistant against well equipped state level
> attackers. We will however do our best. Note also that this is different
> attackers. We will, however, do our best. Note also that this is different
> from anonymity which is not, currently, a goal.
### 3. Easy to Use
Syncthing should be approachable, understandable and inclusive.
Syncthing should be approachable, understandable, and inclusive.
> Complex concepts and maths form the base of Syncthing's functionality.
> This should nonetheless be abstracted or hidden to a degree where
@@ -52,18 +52,18 @@ User interaction should be required only when absolutely necessary.
### 5. Universally Available
Syncthing should run on every common computer. We are mindful that the
latest technology is not always available to any given individual.
latest technology is not always available to every individual.
> Computers include desktops, laptops, servers, virtual machines, small
> general purpose computers such as Raspberry Pis and, *where possible*,
> tablets and phones. NAS appliances, toasters, cars, firearms, thermostats
> tablets and phones. NAS appliances, toasters, cars, firearms, thermostats,
> and so on may include computing capabilities but it is not our goal for
> Syncthing to run smoothly on these devices.
### 6. For Individuals
Syncthing is primarily about empowering the individual user with safe,
secure and easy to use file synchronization.
secure, and easy to use file synchronization.
> We acknowledge that it's also useful in an enterprise setting and include
> functionality to support that. If this is in conflict with the

View File

@@ -15,6 +15,9 @@ To grant Syncthing additional capabilities without running as root, use the
`PCAP` environment variable with the same syntax as that for `setcap(8)`.
For example, `PCAP=cap_chown,cap_fowner+ep`.
To set a different umask value, use the `UMASK` environment variable. For
example `UMASK=002`.
## Example Usage
**Docker cli**
@@ -46,6 +49,11 @@ services:
- 22000:22000/udp # QUIC file transfers
- 21027:21027/udp # Receive local discovery broadcasts
restart: unless-stopped
healthcheck:
test: curl -fkLsS -m 2 127.0.0.1:8384/rest/noauth/health | grep -o --color=never OK || exit 1
interval: 1m
timeout: 10s
retries: 3
```
## Discovery
@@ -81,6 +89,11 @@ services:
- /wherever/st-sync:/var/syncthing
network_mode: host
restart: unless-stopped
healthcheck:
test: curl -fkLsS -m 2 127.0.0.1:8384/rest/noauth/health | grep -o --color=never OK || exit 1
interval: 1m
timeout: 10s
retries: 3
```
Be aware that syncthing alone is now in control of what interfaces and ports it

View File

@@ -2,9 +2,6 @@
---
[![Latest Linux & Cross Build](https://img.shields.io/teamcity/https/build.syncthing.net/s/Syncthing_BuildLinuxCross.svg?style=flat-square&label=linux+%26+cross+build)](https://build.syncthing.net/viewType.html?buildTypeId=Syncthing_BuildLinuxCross&guest=1)
[![Latest Windows Build](https://img.shields.io/teamcity/https/build.syncthing.net/s/Syncthing_BuildWindows.svg?style=flat-square&label=windows+build)](https://build.syncthing.net/viewType.html?buildTypeId=Syncthing_BuildWindows&guest=1)
[![Latest Mac Build](https://img.shields.io/teamcity/https/build.syncthing.net/s/Syncthing_BuildMac.svg?style=flat-square&label=mac+build)](https://build.syncthing.net/viewType.html?buildTypeId=Syncthing_BuildMac&guest=1)
[![MPLv2 License](https://img.shields.io/badge/license-MPLv2-blue.svg?style=flat-square)](https://www.mozilla.org/MPL/2.0/)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/88/badge)](https://bestpractices.coreinfrastructure.org/projects/88)
[![Go Report Card](https://goreportcard.com/badge/github.com/syncthing/syncthing)](https://goreportcard.com/report/github.com/syncthing/syncthing)
@@ -13,8 +10,8 @@
Syncthing is a **continuous file synchronization program**. It synchronizes
files between two or more computers. We strive to fulfill the goals below.
The goals are listed in order of importance, the most important one being
the first. This is the summary version of the goal list - for more
The goals are listed in order of importance, the most important ones first.
This is the summary version of the goal list - for more
commentary, see the full [Goals document][13].
Syncthing should be:
@@ -27,12 +24,12 @@ Syncthing should be:
2. **Secure Against Attackers**
Again, protecting the user's data is paramount. Regardless of our other
goals we must never allow the user's data to be susceptible to
goals, we must never allow the user's data to be susceptible to
eavesdropping or modification by unauthorized parties.
3. **Easy to Use**
Syncthing should be approachable, understandable and inclusive.
Syncthing should be approachable, understandable, and inclusive.
4. **Automatic**
@@ -41,12 +38,12 @@ Syncthing should be:
5. **Universally Available**
Syncthing should run on every common computer. We are mindful that the
latest technology is not always available to any given individual.
latest technology is not always available to every individual.
6. **For Individuals**
Syncthing is primarily about empowering the individual user with safe,
secure and easy to use file synchronization.
secure, and easy to use file synchronization.
7. **Everything Else**
@@ -60,23 +57,22 @@ Take a look at the [getting started guide][2].
There are a few examples for keeping Syncthing running in the background
on your system in [the etc directory][3]. There are also several [GUI
implementations][11] for Windows, Mac and Linux.
implementations][11] for Windows, Mac, and Linux.
## Docker
To run Syncthing in Docker, see [the Docker README][16].
## Vote on features/bugs
We'd like to encourage you to [vote][12] on issues that matter to you.
This helps the team understand what are the biggest pain points for our users, and could potentially influence what is being worked on next.
## Getting in Touch
The first and best point of contact is the [Forum][8].
If you've found something that is clearly a
bug, feel free to report it in the [GitHub issue tracker][10].
If you believe that youve found a Syncthing-related security vulnerability,
please report it by emailing security@syncthing.net. Do not report it in the
Forum or issue tracker.
## Building
Building Syncthing from source is easy. After extracting the source bundle from
@@ -86,13 +82,11 @@ build process.
## Signed Releases
As of v0.10.15 and onwards release binaries are GPG signed with the key
D26E6ED000654A3E, available from https://syncthing.net/security.html and
most key servers.
There is also a built in automatic upgrade mechanism (disabled in some
distribution channels) which uses a compiled in ECDSA signature. macOS
binaries are also properly code signed.
Release binaries are GPG signed with the key available from
https://syncthing.net/security/. There is also a built-in automatic
upgrade mechanism (disabled in some distribution channels) which uses a
compiled in ECDSA signature. macOS and Windows binaries are also
code-signed.
## Documentation
@@ -109,7 +103,6 @@ All code is licensed under the [MPLv2 License][7].
[8]: https://forum.syncthing.net/
[10]: https://github.com/syncthing/syncthing/issues
[11]: https://docs.syncthing.net/users/contrib.html#gui-wrappers
[12]: https://www.bountysource.com/teams/syncthing/issues
[13]: https://github.com/syncthing/syncthing/blob/main/GOALS.md
[14]: assets/logo-text-128.png
[15]: https://syncthing.net/

12
buf.gen.yaml Normal file
View File

@@ -0,0 +1,12 @@
version: v2
managed:
enabled: true
override:
- file_option: go_package_prefix
value: github.com/syncthing/syncthing/internal/gen
plugins:
- remote: buf.build/protocolbuffers/go:v1.35.1
out: .
opt: module=github.com/syncthing/syncthing
inputs:
- directory: proto

10
buf.yaml Normal file
View File

@@ -0,0 +1,10 @@
version: v2
modules:
- path: proto
name: github.com/syncthing/syncthing
lint:
use:
- STANDARD
breaking:
use:
- WIRE_JSON

327
build.go
View File

@@ -4,8 +4,8 @@
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
//go:build ignore
// +build ignore
//go:build tools
// +build tools
package main
@@ -15,7 +15,6 @@ import (
"bytes"
"compress/flate"
"compress/gzip"
"encoding/base64"
"encoding/json"
"errors"
"flag"
@@ -34,6 +33,8 @@ import (
"time"
buildpkg "github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/upgrade"
"sigs.k8s.io/yaml"
)
var (
@@ -84,7 +85,6 @@ var targets = map[string]target{
"all": {
// Only valid for the "build" and "install" commands as it lacks all
// the archive creation stuff. buildPkgs gets filled out in init()
tags: []string{"purego"},
},
"syncthing": {
// The default target for "build", "install", "tar", "zip", "deb", etc.
@@ -95,41 +95,40 @@ var targets = map[string]target{
buildPkgs: []string{"github.com/syncthing/syncthing/cmd/syncthing"},
binaryName: "syncthing", // .exe will be added automatically for Windows builds
archiveFiles: []archiveFile{
{src: "{{binary}}", dst: "{{binary}}", perm: 0755},
{src: "README.md", dst: "README.txt", perm: 0644},
{src: "LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
{src: "{{binary}}", dst: "{{binary}}", perm: 0o755},
{src: "README.md", dst: "README.txt", perm: 0o644},
{src: "LICENSE", dst: "LICENSE.txt", perm: 0o644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0o644},
// All files from etc/ and extra/ added automatically in init().
},
systemdService: "syncthing@*.service",
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "README.md", dst: "deb/usr/share/doc/syncthing/README.txt", perm: 0644},
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing/AUTHORS.txt", perm: 0644},
{src: "man/syncthing.1", dst: "deb/usr/share/man/man1/syncthing.1", perm: 0644},
{src: "man/syncthing-config.5", dst: "deb/usr/share/man/man5/syncthing-config.5", perm: 0644},
{src: "man/syncthing-stignore.5", dst: "deb/usr/share/man/man5/syncthing-stignore.5", perm: 0644},
{src: "man/syncthing-device-ids.7", dst: "deb/usr/share/man/man7/syncthing-device-ids.7", perm: 0644},
{src: "man/syncthing-event-api.7", dst: "deb/usr/share/man/man7/syncthing-event-api.7", perm: 0644},
{src: "man/syncthing-faq.7", dst: "deb/usr/share/man/man7/syncthing-faq.7", perm: 0644},
{src: "man/syncthing-networking.7", dst: "deb/usr/share/man/man7/syncthing-networking.7", perm: 0644},
{src: "man/syncthing-rest-api.7", dst: "deb/usr/share/man/man7/syncthing-rest-api.7", perm: 0644},
{src: "man/syncthing-security.7", dst: "deb/usr/share/man/man7/syncthing-security.7", perm: 0644},
{src: "man/syncthing-versioning.7", dst: "deb/usr/share/man/man7/syncthing-versioning.7", perm: 0644},
{src: "etc/linux-systemd/system/syncthing@.service", dst: "deb/lib/systemd/system/syncthing@.service", perm: 0644},
{src: "etc/linux-systemd/system/syncthing-resume.service", dst: "deb/lib/systemd/system/syncthing-resume.service", perm: 0644},
{src: "etc/linux-systemd/user/syncthing.service", dst: "deb/usr/lib/systemd/user/syncthing.service", perm: 0644},
{src: "etc/linux-sysctl/30-syncthing.conf", dst: "deb/usr/lib/sysctl.d/30-syncthing.conf", perm: 0644},
{src: "etc/firewall-ufw/syncthing", dst: "deb/etc/ufw/applications.d/syncthing", perm: 0644},
{src: "etc/linux-desktop/syncthing-start.desktop", dst: "deb/usr/share/applications/syncthing-start.desktop", perm: 0644},
{src: "etc/linux-desktop/syncthing-ui.desktop", dst: "deb/usr/share/applications/syncthing-ui.desktop", perm: 0644},
{src: "assets/logo-32.png", dst: "deb/usr/share/icons/hicolor/32x32/apps/syncthing.png", perm: 0644},
{src: "assets/logo-64.png", dst: "deb/usr/share/icons/hicolor/64x64/apps/syncthing.png", perm: 0644},
{src: "assets/logo-128.png", dst: "deb/usr/share/icons/hicolor/128x128/apps/syncthing.png", perm: 0644},
{src: "assets/logo-256.png", dst: "deb/usr/share/icons/hicolor/256x256/apps/syncthing.png", perm: 0644},
{src: "assets/logo-512.png", dst: "deb/usr/share/icons/hicolor/512x512/apps/syncthing.png", perm: 0644},
{src: "assets/logo-only.svg", dst: "deb/usr/share/icons/hicolor/scalable/apps/syncthing.svg", perm: 0644},
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0o755},
{src: "README.md", dst: "deb/usr/share/doc/syncthing/README.txt", perm: 0o644},
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing/LICENSE.txt", perm: 0o644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing/AUTHORS.txt", perm: 0o644},
{src: "man/syncthing.1", dst: "deb/usr/share/man/man1/syncthing.1", perm: 0o644},
{src: "man/syncthing-config.5", dst: "deb/usr/share/man/man5/syncthing-config.5", perm: 0o644},
{src: "man/syncthing-stignore.5", dst: "deb/usr/share/man/man5/syncthing-stignore.5", perm: 0o644},
{src: "man/syncthing-device-ids.7", dst: "deb/usr/share/man/man7/syncthing-device-ids.7", perm: 0o644},
{src: "man/syncthing-event-api.7", dst: "deb/usr/share/man/man7/syncthing-event-api.7", perm: 0o644},
{src: "man/syncthing-faq.7", dst: "deb/usr/share/man/man7/syncthing-faq.7", perm: 0o644},
{src: "man/syncthing-networking.7", dst: "deb/usr/share/man/man7/syncthing-networking.7", perm: 0o644},
{src: "man/syncthing-rest-api.7", dst: "deb/usr/share/man/man7/syncthing-rest-api.7", perm: 0o644},
{src: "man/syncthing-security.7", dst: "deb/usr/share/man/man7/syncthing-security.7", perm: 0o644},
{src: "man/syncthing-versioning.7", dst: "deb/usr/share/man/man7/syncthing-versioning.7", perm: 0o644},
{src: "etc/linux-systemd/system/syncthing@.service", dst: "deb/lib/systemd/system/syncthing@.service", perm: 0o644},
{src: "etc/linux-systemd/user/syncthing.service", dst: "deb/usr/lib/systemd/user/syncthing.service", perm: 0o644},
{src: "etc/linux-sysctl/30-syncthing.conf", dst: "deb/usr/lib/sysctl.d/30-syncthing.conf", perm: 0o644},
{src: "etc/firewall-ufw/syncthing", dst: "deb/etc/ufw/applications.d/syncthing", perm: 0o644},
{src: "etc/linux-desktop/syncthing-start.desktop", dst: "deb/usr/share/applications/syncthing-start.desktop", perm: 0o644},
{src: "etc/linux-desktop/syncthing-ui.desktop", dst: "deb/usr/share/applications/syncthing-ui.desktop", perm: 0o644},
{src: "assets/logo-32.png", dst: "deb/usr/share/icons/hicolor/32x32/apps/syncthing.png", perm: 0o644},
{src: "assets/logo-64.png", dst: "deb/usr/share/icons/hicolor/64x64/apps/syncthing.png", perm: 0o644},
{src: "assets/logo-128.png", dst: "deb/usr/share/icons/hicolor/128x128/apps/syncthing.png", perm: 0o644},
{src: "assets/logo-256.png", dst: "deb/usr/share/icons/hicolor/256x256/apps/syncthing.png", perm: 0o644},
{src: "assets/logo-512.png", dst: "deb/usr/share/icons/hicolor/512x512/apps/syncthing.png", perm: 0o644},
{src: "assets/logo-only.svg", dst: "deb/usr/share/icons/hicolor/scalable/apps/syncthing.svg", perm: 0o644},
},
},
"stdiscosrv": {
@@ -141,23 +140,22 @@ var targets = map[string]target{
buildPkgs: []string{"github.com/syncthing/syncthing/cmd/stdiscosrv"},
binaryName: "stdiscosrv", // .exe will be added automatically for Windows builds
archiveFiles: []archiveFile{
{src: "{{binary}}", dst: "{{binary}}", perm: 0755},
{src: "cmd/stdiscosrv/README.md", dst: "README.txt", perm: 0644},
{src: "LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
{src: "{{binary}}", dst: "{{binary}}", perm: 0o755},
{src: "cmd/stdiscosrv/README.md", dst: "README.txt", perm: 0o644},
{src: "LICENSE", dst: "LICENSE.txt", perm: 0o644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0o644},
},
systemdService: "stdiscosrv.service",
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "cmd/stdiscosrv/README.md", dst: "deb/usr/share/doc/syncthing-discosrv/README.txt", perm: 0644},
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing-discosrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-discosrv/AUTHORS.txt", perm: 0644},
{src: "man/stdiscosrv.1", dst: "deb/usr/share/man/man1/stdiscosrv.1", perm: 0644},
{src: "cmd/stdiscosrv/etc/linux-systemd/stdiscosrv.service", dst: "deb/lib/systemd/system/stdiscosrv.service", perm: 0644},
{src: "cmd/stdiscosrv/etc/linux-systemd/default", dst: "deb/etc/default/syncthing-discosrv", perm: 0644},
{src: "cmd/stdiscosrv/etc/firewall-ufw/stdiscosrv", dst: "deb/etc/ufw/applications.d/stdiscosrv", perm: 0644},
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0o755},
{src: "cmd/stdiscosrv/README.md", dst: "deb/usr/share/doc/syncthing-discosrv/README.txt", perm: 0o644},
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing-discosrv/LICENSE.txt", perm: 0o644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-discosrv/AUTHORS.txt", perm: 0o644},
{src: "man/stdiscosrv.1", dst: "deb/usr/share/man/man1/stdiscosrv.1", perm: 0o644},
{src: "cmd/stdiscosrv/etc/linux-systemd/stdiscosrv.service", dst: "deb/lib/systemd/system/stdiscosrv.service", perm: 0o644},
{src: "cmd/stdiscosrv/etc/linux-systemd/default", dst: "deb/etc/default/syncthing-discosrv", perm: 0o644},
{src: "cmd/stdiscosrv/etc/firewall-ufw/stdiscosrv", dst: "deb/etc/ufw/applications.d/stdiscosrv", perm: 0o644},
},
tags: []string{"purego"},
},
"strelaysrv": {
name: "strelaysrv",
@@ -168,44 +166,48 @@ var targets = map[string]target{
buildPkgs: []string{"github.com/syncthing/syncthing/cmd/strelaysrv"},
binaryName: "strelaysrv", // .exe will be added automatically for Windows builds
archiveFiles: []archiveFile{
{src: "{{binary}}", dst: "{{binary}}", perm: 0755},
{src: "cmd/strelaysrv/README.md", dst: "README.txt", perm: 0644},
{src: "cmd/strelaysrv/LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
{src: "{{binary}}", dst: "{{binary}}", perm: 0o755},
{src: "cmd/strelaysrv/README.md", dst: "README.txt", perm: 0o644},
{src: "cmd/strelaysrv/LICENSE", dst: "LICENSE.txt", perm: 0o644},
{src: "LICENSE", dst: "LICENSE.txt", perm: 0o644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0o644},
},
systemdService: "strelaysrv.service",
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "cmd/strelaysrv/README.md", dst: "deb/usr/share/doc/syncthing-relaysrv/README.txt", perm: 0644},
{src: "cmd/strelaysrv/LICENSE", dst: "deb/usr/share/doc/syncthing-relaysrv/LICENSE.txt", perm: 0644},
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing-relaysrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-relaysrv/AUTHORS.txt", perm: 0644},
{src: "man/strelaysrv.1", dst: "deb/usr/share/man/man1/strelaysrv.1", perm: 0644},
{src: "cmd/strelaysrv/etc/linux-systemd/strelaysrv.service", dst: "deb/lib/systemd/system/strelaysrv.service", perm: 0644},
{src: "cmd/strelaysrv/etc/linux-systemd/default", dst: "deb/etc/default/syncthing-relaysrv", perm: 0644},
{src: "cmd/strelaysrv/etc/firewall-ufw/strelaysrv", dst: "deb/etc/ufw/applications.d/strelaysrv", perm: 0644},
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0o755},
{src: "cmd/strelaysrv/README.md", dst: "deb/usr/share/doc/syncthing-relaysrv/README.txt", perm: 0o644},
{src: "cmd/strelaysrv/LICENSE", dst: "deb/usr/share/doc/syncthing-relaysrv/LICENSE.txt", perm: 0o644},
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing-relaysrv/LICENSE.txt", perm: 0o644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-relaysrv/AUTHORS.txt", perm: 0o644},
{src: "man/strelaysrv.1", dst: "deb/usr/share/man/man1/strelaysrv.1", perm: 0o644},
{src: "cmd/strelaysrv/etc/linux-systemd/strelaysrv.service", dst: "deb/lib/systemd/system/strelaysrv.service", perm: 0o644},
{src: "cmd/strelaysrv/etc/linux-systemd/default", dst: "deb/etc/default/syncthing-relaysrv", perm: 0o644},
{src: "cmd/strelaysrv/etc/firewall-ufw/strelaysrv", dst: "deb/etc/ufw/applications.d/strelaysrv", perm: 0o644},
},
},
"strelaypoolsrv": {
name: "strelaypoolsrv",
debname: "syncthing-relaypoolsrv",
debdeps: []string{"libc6"},
description: "Syncthing Relay Pool Server",
buildPkgs: []string{"github.com/syncthing/syncthing/cmd/strelaypoolsrv"},
binaryName: "strelaypoolsrv", // .exe will be added automatically for Windows builds
archiveFiles: []archiveFile{
{src: "{{binary}}", dst: "{{binary}}", perm: 0755},
{src: "cmd/strelaypoolsrv/README.md", dst: "README.txt", perm: 0644},
{src: "cmd/strelaypoolsrv/LICENSE", dst: "LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "AUTHORS.txt", perm: 0644},
},
installationFiles: []archiveFile{
{src: "{{binary}}", dst: "deb/usr/bin/{{binary}}", perm: 0755},
{src: "cmd/strelaypoolsrv/README.md", dst: "deb/usr/share/doc/syncthing-relaypoolsrv/README.txt", perm: 0644},
{src: "cmd/strelaypoolsrv/LICENSE", dst: "deb/usr/share/doc/syncthing-relaypoolsrv/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing-relaypoolsrv/AUTHORS.txt", perm: 0644},
},
buildPkgs: []string{"github.com/syncthing/syncthing/cmd/infra/strelaypoolsrv"},
binaryName: "strelaypoolsrv",
},
"stupgrades": {
name: "stupgrades",
description: "Syncthing Upgrade Check Server",
buildPkgs: []string{"github.com/syncthing/syncthing/cmd/infra/stupgrades"},
binaryName: "stupgrades",
},
"stcrashreceiver": {
name: "stcrashreceiver",
description: "Syncthing Crash Server",
buildPkgs: []string{"github.com/syncthing/syncthing/cmd/infra/stcrashreceiver"},
binaryName: "stcrashreceiver",
},
"ursrv": {
name: "ursrv",
description: "Syncthing Usage Reporting Server",
buildPkgs: []string{"github.com/syncthing/syncthing/cmd/infra/ursrv"},
binaryName: "ursrv",
},
}
@@ -213,15 +215,11 @@ func initTargets() {
all := targets["all"]
pkgs, _ := filepath.Glob("cmd/*")
for _, pkg := range pkgs {
pkg = filepath.Base(pkg)
if strings.HasPrefix(pkg, ".") {
// ignore dotfiles
if files, err := filepath.Glob(pkg + "/*.go"); err != nil || len(files) == 0 {
// No go files in the directory
continue
}
if noupgrade && pkg == "stupgrades" {
continue
}
all.buildPkgs = append(all.buildPkgs, fmt.Sprintf("github.com/syncthing/syncthing/cmd/%s", pkg))
all.buildPkgs = append(all.buildPkgs, fmt.Sprintf("github.com/syncthing/syncthing/%s", pkg))
}
targets["all"] = all
@@ -229,13 +227,13 @@ func initTargets() {
// and "extra" dirs.
syncthingPkg := targets["syncthing"]
for _, file := range listFiles("etc") {
syncthingPkg.archiveFiles = append(syncthingPkg.archiveFiles, archiveFile{src: file, dst: file, perm: 0644})
syncthingPkg.archiveFiles = append(syncthingPkg.archiveFiles, archiveFile{src: file, dst: file, perm: 0o644})
}
for _, file := range listFiles("extra") {
syncthingPkg.archiveFiles = append(syncthingPkg.archiveFiles, archiveFile{src: file, dst: file, perm: 0644})
syncthingPkg.archiveFiles = append(syncthingPkg.archiveFiles, archiveFile{src: file, dst: file, perm: 0o644})
}
for _, file := range listFiles("extra") {
syncthingPkg.installationFiles = append(syncthingPkg.installationFiles, archiveFile{src: file, dst: "deb/usr/share/doc/syncthing/" + filepath.Base(file), perm: 0644})
syncthingPkg.installationFiles = append(syncthingPkg.installationFiles, archiveFile{src: file, dst: "deb/usr/share/doc/syncthing/" + filepath.Base(file), perm: 0o644})
}
targets["syncthing"] = syncthingPkg
}
@@ -320,11 +318,16 @@ func runCommand(cmd string, target target) {
case "transifex":
transifex()
case "weblate":
weblate()
case "tar":
buildTar(target, tags)
writeCompatJSON()
case "zip":
buildZip(target, tags)
writeCompatJSON()
case "deb":
buildDeb(target)
@@ -385,7 +388,6 @@ func parseFlags() {
func test(tags []string, pkgs ...string) {
lazyRebuildAssets()
tags = append(tags, "purego")
args := []string{"test", "-tags", strings.Join(tags, " ")}
if long {
timeout = longTimeout
@@ -419,7 +421,7 @@ func bench(tags []string, pkgs ...string) {
func integration(bench bool) {
lazyRebuildAssets()
args := []string{"test", "-v", "-timeout", "60m", "-tags"}
tags := "purego,integration"
tags := "integration"
if bench {
tags += ",benchmark"
}
@@ -730,7 +732,7 @@ func shouldBuildSyso(dir string) (string, error) {
}
jsonPath := filepath.Join(dir, "versioninfo.json")
err = os.WriteFile(jsonPath, bs, 0644)
err = os.WriteFile(jsonPath, bs, 0o644)
if err != nil {
return "", errors.New("failed to create " + jsonPath + ": " + err.Error())
}
@@ -789,7 +791,7 @@ func copyFile(src, dst string, perm os.FileMode) error {
}
copy:
os.MkdirAll(filepath.Dir(dst), 0777)
os.MkdirAll(filepath.Dir(dst), 0o777)
if err := os.WriteFile(dst, in, perm); err != nil {
return err
}
@@ -814,12 +816,12 @@ func listFiles(dir string) []string {
func rebuildAssets() {
os.Setenv("SOURCE_DATE_EPOCH", fmt.Sprint(buildStamp()))
runPrint(goCmd, "generate", "github.com/syncthing/syncthing/lib/api/auto", "github.com/syncthing/syncthing/cmd/strelaypoolsrv/auto")
runPrint(goCmd, "generate", "github.com/syncthing/syncthing/lib/api/auto", "github.com/syncthing/syncthing/cmd/infra/strelaypoolsrv/auto")
}
func lazyRebuildAssets() {
shouldRebuild := shouldRebuildAssets("lib/api/auto/gui.files.go", "gui") ||
shouldRebuildAssets("cmd/strelaypoolsrv/auto/gui.files.go", "cmd/strelaypoolsrv/gui")
shouldRebuildAssets("cmd/infra/strelaypoolsrv/auto/gui.files.go", "cmd/infra/strelaypoolsrv/gui")
if withNextGenGUI {
shouldRebuild = buildNextGenGUI() || shouldRebuild
@@ -849,7 +851,7 @@ func buildNextGenGUI() bool {
for _, src := range listFiles("next-gen-gui/dist") {
rel, _ := filepath.Rel("next-gen-gui/dist", src)
dst := filepath.Join("gui", rel)
if err := copyFile(src, dst, 0644); err != nil {
if err := copyFile(src, dst, 0o644); err != nil {
fmt.Println("copy:", err)
os.Exit(1)
}
@@ -897,7 +899,7 @@ func updateDependencies() {
}
goVersion := string(matches[1])
runPrint(goCmd, "get", "-u", "all")
runPrint(goCmd, "get", "-u", "./...")
runPrint(goCmd, "mod", "tidy", "-go="+goVersion, "-compat="+goVersion)
// We might have updated the protobuf package and should regenerate to match.
@@ -905,22 +907,9 @@ func updateDependencies() {
}
func proto() {
pv := protobufVersion()
repo := "https://github.com/gogo/protobuf.git"
path := filepath.Join("repos", "protobuf")
runPrint(goCmd, "install", fmt.Sprintf("github.com/gogo/protobuf/protoc-gen-gogofast@%v", pv))
os.MkdirAll("repos", 0755)
if _, err := os.Stat(path); err != nil {
runPrint("git", "clone", repo, path)
} else {
runPrintInDir(path, "git", "fetch")
}
runPrintInDir(path, "git", "checkout", pv)
runPrint(goCmd, "generate", "github.com/syncthing/syncthing/cmd/stdiscosrv")
runPrint(goCmd, "generate", "proto/generate.go")
// buf needs to be installed
// https://buf.build/docs/installation/
runPrint("buf", "generate")
}
func testmocks() {
@@ -953,6 +942,11 @@ func transifex() {
runPrint(goCmd, "run", "../../../../script/transifexdl.go")
}
func weblate() {
os.Chdir("gui/default/assets/lang")
runPrint(goCmd, "run", "../../../../script/weblatedl.go")
}
func ldflags(tags []string) string {
b := new(strings.Builder)
b.WriteString("-w")
@@ -1088,10 +1082,14 @@ func getBranchSuffix() string {
branch = parts[len(parts)-1]
switch branch {
case "master", "release", "main":
case "release", "main":
// these are not special
return ""
}
if strings.HasPrefix(branch, "release-") {
// release branches are not special
return ""
}
validBranchRe := regexp.MustCompile(`^[a-zA-Z0-9_.-]+$`)
if !validBranchRe.MatchString(branch) {
@@ -1346,10 +1344,7 @@ func zipFile(out string, files []archiveFile) {
}
func codesign(target target) {
switch goos {
case "windows":
windowsCodesign(target.BinaryName())
case "darwin":
if goos == "darwin" {
macosCodesign(target.BinaryName())
}
}
@@ -1373,70 +1368,6 @@ func macosCodesign(file string) {
}
}
func windowsCodesign(file string) {
st := "signtool.exe"
if path := os.Getenv("CODESIGN_SIGNTOOL"); path != "" {
st = path
}
for i, algo := range []string{"sha1", "sha256"} {
args := []string{"sign", "/fd", algo}
if f := os.Getenv("CODESIGN_CERTIFICATE_FILE"); f != "" {
args = append(args, "/f", f)
} else if b := os.Getenv("CODESIGN_CERTIFICATE_BASE64"); b != "" {
// Decode the PFX certificate from base64.
bs, err := base64.RawStdEncoding.DecodeString(b)
if err != nil {
log.Println("Codesign: signing failed: decoding base64:", err)
return
}
// Write it to a temporary file
f, err := os.CreateTemp("", "codesign-*.pfx")
if err != nil {
log.Println("Codesign: signing failed: creating temp file:", err)
return
}
_ = f.Chmod(0600) // best effort remove other users' access
defer os.Remove(f.Name())
if _, err := f.Write(bs); err != nil {
log.Println("Codesign: signing failed: writing temp file:", err)
return
}
if err := f.Close(); err != nil {
log.Println("Codesign: signing failed: closing temp file:", err)
return
}
// Use that when signing
args = append(args, "/f", f.Name())
}
if p := os.Getenv("CODESIGN_CERTIFICATE_PASSWORD"); p != "" {
args = append(args, "/p", p)
}
if tr := os.Getenv("CODESIGN_TIMESTAMP_SERVER"); tr != "" {
switch algo {
case "sha256":
args = append(args, "/tr", tr, "/td", algo)
default:
args = append(args, "/t", tr)
}
}
if i > 0 {
args = append(args, "/as")
}
args = append(args, file)
bs, err := runError(st, args...)
if err != nil {
log.Printf("Codesign: signing failed: %v: %s", err, string(bs))
return
}
log.Println("Codesign: successfully signed", file, "using", algo)
}
}
func metalint() {
lazyRebuildAssets()
runPrint(goCmd, "test", "-run", "Metalint", "./meta")
@@ -1454,14 +1385,6 @@ func (t target) BinaryName() string {
return t.binaryName
}
func protobufVersion() string {
bs, err := runError(goCmd, "list", "-f", "{{.Version}}", "-m", "github.com/gogo/protobuf")
if err != nil {
log.Fatal("Getting protobuf version:", err)
}
return string(bs)
}
func currentAndLatestVersions(n int) ([]string, error) {
bs, err := runError("git", "tag", "--sort", "taggerdate")
if err != nil {
@@ -1528,3 +1451,29 @@ func nextPatchVersion(ver string) string {
digits[len(digits)-1] = strconv.Itoa(n + 1)
return strings.Join(digits, ".")
}
func writeCompatJSON() {
bs, err := os.ReadFile("compat.yaml")
if err != nil {
log.Fatal("Reading compat.yaml:", err)
}
var entries []upgrade.ReleaseCompatibility
if err := yaml.Unmarshal(bs, &entries); err != nil {
log.Fatal("Parsing compat.yaml:", err)
}
rt := runtime.Version()
for _, e := range entries {
if !strings.HasPrefix(rt, e.Runtime) {
continue
}
bs, _ := json.MarshalIndent(e, "", " ")
if err := os.WriteFile("compat.json", bs, 0o644); err != nil {
log.Fatal("Writing compat.json:", err)
}
return
}
log.Fatalf("runtime %v not found in compat.yaml", rt)
}

View File

@@ -23,10 +23,10 @@ case "${1:-default}" in
prerelease)
script authors
build transifex
build weblate
pushd man ; ./refresh.sh ; popd
git add -A gui man AUTHORS
git commit -m 'gui, man, authors: Update docs, translations, and contributors'
git commit -m 'chore(gui, man, authors): update docs, translations, and contributors'
;;
*)

View File

@@ -7,6 +7,7 @@
package main
import (
"crypto/sha256"
"errors"
"flag"
"fmt"
@@ -15,7 +16,7 @@ import (
"os"
"path/filepath"
"github.com/syncthing/syncthing/lib/sha256"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
)
func main() {

View File

@@ -15,6 +15,10 @@ import (
"strings"
"time"
"google.golang.org/protobuf/proto"
"github.com/syncthing/syncthing/internal/gen/discoproto"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/beacon"
"github.com/syncthing/syncthing/lib/discover"
"github.com/syncthing/syncthing/lib/protocol"
@@ -74,20 +78,21 @@ func recv(bc beacon.Interface) {
continue
}
var ann discover.Announce
ann.Unmarshal(data[4:])
var ann discoproto.Announce
proto.Unmarshal(data[4:], &ann)
if ann.ID == myID {
id, _ := protocol.DeviceIDFromBytes(ann.Id)
if id == myID {
// This is one of our own fake packets, don't print it.
continue
}
// Print announcement details for the first packet from a given
// device ID and source address, or if -all was given.
key := ann.ID.String() + src.String()
key := id.String() + src.String()
if all || !seen[key] {
log.Printf("Announcement from %v\n", src)
log.Printf(" %v at %s\n", ann.ID, strings.Join(ann.Addresses, ", "))
log.Printf(" %v at %s\n", id, strings.Join(ann.Addresses, ", "))
seen[key] = true
}
}
@@ -95,11 +100,11 @@ func recv(bc beacon.Interface) {
// sends fake discovery announcements once every second
func send(bc beacon.Interface) {
ann := discover.Announce{
ID: myID,
ann := &discoproto.Announce{
Id: myID[:],
Addresses: []string{"tcp://fake.example.com:12345"},
}
bs, _ := ann.Marshal()
bs, _ := proto.Marshal(ann)
for {
bc.Send(bs)

View File

@@ -14,6 +14,8 @@ import (
"net/http"
"os"
"time"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
)
type event struct {

View File

@@ -13,6 +13,7 @@ import (
"os"
"path/filepath"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/scanner"
)

View File

@@ -16,6 +16,7 @@ import (
"os"
"time"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/discover"
"github.com/syncthing/syncthing/lib/events"

View File

@@ -12,6 +12,7 @@ import (
"fmt"
"os"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/fs"
"github.com/syncthing/syncthing/lib/ignore"
)

View File

@@ -15,6 +15,8 @@ import (
"os"
"path/filepath"
"time"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
)
func main() {
@@ -43,7 +45,7 @@ func generateFiles(dir string, files, maxexp int, srcname string) error {
}
p0 := filepath.Join(dir, string(n[0]), n[0:2])
err = os.MkdirAll(p0, 0755)
err = os.MkdirAll(p0, 0o755)
if err != nil {
log.Fatal(err)
}
@@ -66,7 +68,7 @@ func generateFiles(dir string, files, maxexp int, srcname string) error {
}
func generateOneFile(fd io.ReadSeeker, p1 string, s int64) error {
src := io.LimitReader(&inifiteReader{fd}, s)
src := io.LimitReader(&infiniteReader{fd}, s)
dst, err := os.Create(p1)
if err != nil {
return err
@@ -82,7 +84,7 @@ func generateOneFile(fd io.ReadSeeker, p1 string, s int64) error {
return err
}
os.Chmod(p1, os.FileMode(rand.Intn(0777)|0400))
os.Chmod(p1, os.FileMode(rand.Intn(0o777)|0o400))
t := time.Now().Add(-time.Duration(rand.Intn(30*86400)) * time.Second)
return os.Chtimes(p1, t, t)
@@ -105,11 +107,11 @@ func readRand(bs []byte) (int, error) {
return len(bs), nil
}
type inifiteReader struct {
type infiniteReader struct {
rd io.ReadSeeker
}
func (i *inifiteReader) Read(bs []byte) (int, error) {
func (i *infiniteReader) Read(bs []byte) (int, error) {
n, err := i.rd.Read(bs)
if err == io.EOF {
err = nil

View File

@@ -12,6 +12,7 @@ import (
"log"
"os"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/signature"
"github.com/syncthing/syncthing/lib/upgrade"
)

View File

@@ -26,6 +26,7 @@ import (
"sync/atomic"
"time"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/protocol"
)
@@ -44,7 +45,7 @@ func main() {
found := make(chan result)
stop := make(chan struct{})
var count int64
var count atomic.Int64
// Print periodic progress reports.
go printProgress(prefix, &count)
@@ -72,7 +73,7 @@ func main() {
// Try certificates until one is found that has the prefix at the start of
// the resulting device ID. Increments count atomically, sends the result to
// found, returns when stop is closed.
func generatePrefixed(prefix string, count *int64, found chan<- result, stop <-chan struct{}) {
func generatePrefixed(prefix string, count *atomic.Int64, found chan<- result, stop <-chan struct{}) {
notBefore := time.Now()
notAfter := time.Date(2049, 12, 31, 23, 59, 59, 0, time.UTC)
@@ -109,7 +110,7 @@ func generatePrefixed(prefix string, count *int64, found chan<- result, stop <-c
}
id := protocol.NewDeviceID(derBytes)
atomic.AddInt64(count, 1)
count.Add(1)
if strings.HasPrefix(id.String(), prefix) {
select {
@@ -121,7 +122,7 @@ func generatePrefixed(prefix string, count *int64, found chan<- result, stop <-c
}
}
func printProgress(prefix string, count *int64) {
func printProgress(prefix string, count *atomic.Int64) {
started := time.Now()
wantBits := 5 * len(prefix)
if wantBits > 63 {
@@ -132,7 +133,7 @@ func printProgress(prefix string, count *int64) {
fmt.Printf("Want %d bits for prefix %q, about %.2g certs to test (statistically speaking)\n", wantBits, prefix, expectedIterations)
for range time.NewTicker(15 * time.Second).C {
tried := atomic.LoadInt64(count)
tried := count.Load()
elapsed := time.Since(started)
rate := float64(tried) / elapsed.Seconds()
expected := timeStr(expectedIterations / rate)
@@ -157,7 +158,7 @@ func saveCert(priv interface{}, derBytes []byte) {
os.Exit(1)
}
keyOut, err := os.OpenFile("key.pem", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
keyOut, err := os.OpenFile("key.pem", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
fmt.Println(err)
os.Exit(1)

View File

@@ -7,13 +7,14 @@
package main
import (
"crypto/sha256"
"flag"
"fmt"
"io"
"os"
"time"
"github.com/syncthing/syncthing/lib/sha256"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
)
func main() {

View File

@@ -0,0 +1,198 @@
// Copyright (C) 2023 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"compress/gzip"
"context"
"io"
"log"
"math"
"os"
"path/filepath"
"sort"
"time"
)
type diskStore struct {
dir string
inbox chan diskEntry
maxBytes int64
maxFiles int
currentFiles []currentFile
currentSize int64
}
type diskEntry struct {
path string
data []byte
}
type currentFile struct {
path string
size int64
mtime int64
}
func (d *diskStore) Serve(ctx context.Context) {
if err := os.MkdirAll(d.dir, 0o700); err != nil {
log.Println("Creating directory:", err)
return
}
if err := d.inventory(); err != nil {
log.Println("Failed to inventory disk store:", err)
}
d.clean()
cleanTimer := time.NewTicker(time.Minute)
inventoryTimer := time.NewTicker(24 * time.Hour)
buf := new(bytes.Buffer)
gw := gzip.NewWriter(buf)
for {
select {
case entry := <-d.inbox:
path := d.fullPath(entry.path)
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
log.Println("Creating directory:", err)
continue
}
buf.Reset()
gw.Reset(buf)
if _, err := gw.Write(entry.data); err != nil {
log.Println("Failed to compress crash report:", err)
continue
}
if err := gw.Close(); err != nil {
log.Println("Failed to compress crash report:", err)
continue
}
if err := os.WriteFile(path, buf.Bytes(), 0o600); err != nil {
log.Printf("Failed to write %s: %v", entry.path, err)
_ = os.Remove(path)
continue
}
d.currentSize += int64(buf.Len())
d.currentFiles = append(d.currentFiles, currentFile{
size: int64(len(entry.data)),
path: path,
})
case <-cleanTimer.C:
d.clean()
case <-inventoryTimer.C:
if err := d.inventory(); err != nil {
log.Println("Failed to inventory disk store:", err)
}
case <-ctx.Done():
return
}
}
}
func (d *diskStore) Put(path string, data []byte) bool {
select {
case d.inbox <- diskEntry{
path: path,
data: data,
}:
return true
default:
return false
}
}
func (d *diskStore) Get(path string) ([]byte, error) {
path = d.fullPath(path)
bs, err := os.ReadFile(path)
if err != nil {
return nil, err
}
gr, err := gzip.NewReader(bytes.NewReader(bs))
if err != nil {
return nil, err
}
defer gr.Close()
return io.ReadAll(gr)
}
func (d *diskStore) Exists(path string) bool {
path = d.fullPath(path)
_, err := os.Lstat(path)
return err == nil
}
func (d *diskStore) clean() {
for len(d.currentFiles) > 0 && (len(d.currentFiles) > d.maxFiles || d.currentSize > d.maxBytes) {
f := d.currentFiles[0]
log.Println("Removing", f.path)
if err := os.Remove(f.path); err != nil {
log.Println("Failed to remove file:", err)
}
d.currentFiles = d.currentFiles[1:]
d.currentSize -= f.size
}
var oldest time.Duration
if len(d.currentFiles) > 0 {
oldest = time.Since(time.Unix(d.currentFiles[0].mtime, 0)).Truncate(time.Minute)
}
metricDiskstoreFilesTotal.Set(float64(len(d.currentFiles)))
metricDiskstoreBytesTotal.Set(float64(d.currentSize))
metricDiskstoreOldestAgeSeconds.Set(math.Round(oldest.Seconds()))
log.Printf("Clean complete: %d files, %d MB, oldest is %v ago", len(d.currentFiles), d.currentSize>>20, oldest)
}
func (d *diskStore) inventory() error {
d.currentFiles = nil
d.currentSize = 0
err := filepath.Walk(d.dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
if filepath.Ext(path) != ".gz" {
return nil
}
d.currentSize += info.Size()
d.currentFiles = append(d.currentFiles, currentFile{
path: path,
size: info.Size(),
mtime: info.ModTime().Unix(),
})
return nil
})
sort.Slice(d.currentFiles, func(i, j int) bool {
return d.currentFiles[i].mtime < d.currentFiles[j].mtime
})
var oldest time.Duration
if len(d.currentFiles) > 0 {
oldest = time.Since(time.Unix(d.currentFiles[0].mtime, 0)).Truncate(time.Minute)
}
metricDiskstoreFilesTotal.Set(float64(len(d.currentFiles)))
metricDiskstoreBytesTotal.Set(float64(d.currentSize))
metricDiskstoreOldestAgeSeconds.Set(math.Round(oldest.Seconds()))
log.Printf("Inventory complete: %d files, %d MB, oldest is %v ago", len(d.currentFiles), d.currentSize>>20, oldest)
return err
}
func (d *diskStore) fullPath(path string) string {
return filepath.Join(d.dir, path[0:2], path[2:]) + ".gz"
}

View File

@@ -0,0 +1,228 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
// Command stcrashreceiver is a trivial HTTP server that allows two things:
//
// - uploading files (crash reports) named like a SHA256 hash using a PUT request
// - checking whether such file exists using a HEAD request
//
// Typically this should be deployed behind something that manages HTTPS.
package main
import (
"context"
"crypto/sha256"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"path/filepath"
"regexp"
"strings"
"github.com/alecthomas/kong"
raven "github.com/getsentry/raven-go"
"github.com/prometheus/client_golang/prometheus/promhttp"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/ur"
)
const maxRequestSize = 1 << 20 // 1 MiB
type cli struct {
Dir string `help:"Parent directory to store crash and failure reports in" env:"REPORTS_DIR" default:"."`
DSN string `help:"Sentry DSN" env:"SENTRY_DSN"`
Listen string `help:"HTTP listen address" default:":8080" env:"LISTEN_ADDRESS"`
MaxDiskFiles int `help:"Maximum number of reports on disk" default:"100000" env:"MAX_DISK_FILES"`
MaxDiskSizeMB int64 `help:"Maximum disk space to use for reports" default:"1024" env:"MAX_DISK_SIZE_MB"`
SentryQueue int `help:"Maximum number of reports to queue for sending to Sentry" default:"64" env:"SENTRY_QUEUE"`
DiskQueue int `help:"Maximum number of reports to queue for writing to disk" default:"64" env:"DISK_QUEUE"`
MetricsListen string `help:"HTTP listen address for metrics" default:":8081" env:"METRICS_LISTEN_ADDRESS"`
IngorePatterns string `help:"File containing ignore patterns (regexp)" env:"IGNORE_PATTERNS" type:"existingfile"`
}
func main() {
var params cli
kong.Parse(&params)
mux := http.NewServeMux()
ds := &diskStore{
dir: filepath.Join(params.Dir, "crash_reports"),
inbox: make(chan diskEntry, params.DiskQueue),
maxFiles: params.MaxDiskFiles,
maxBytes: params.MaxDiskSizeMB << 20,
}
go ds.Serve(context.Background())
ss := &sentryService{
dsn: params.DSN,
inbox: make(chan sentryRequest, params.SentryQueue),
}
go ss.Serve(context.Background())
var ip *ignorePatterns
if params.IngorePatterns != "" {
var err error
ip, err = loadIgnorePatterns(params.IngorePatterns)
if err != nil {
log.Fatalf("Failed to load ignore patterns: %v", err)
}
}
cr := &crashReceiver{
store: ds,
sentry: ss,
ignore: ip,
}
mux.Handle("/", cr)
mux.HandleFunc("/ping", func(w http.ResponseWriter, req *http.Request) {
w.Write([]byte("OK"))
})
if params.MetricsListen != "" {
mmux := http.NewServeMux()
mmux.Handle("/metrics", promhttp.Handler())
go func() {
if err := http.ListenAndServe(params.MetricsListen, mmux); err != nil {
log.Fatalln("HTTP serve metrics:", err)
}
}()
}
if params.DSN != "" {
mux.HandleFunc("/newcrash/failure", handleFailureFn(params.DSN, filepath.Join(params.Dir, "failure_reports"), ip))
}
log.SetOutput(os.Stdout)
if err := http.ListenAndServe(params.Listen, mux); err != nil {
log.Fatalln("HTTP serve:", err)
}
}
func handleFailureFn(dsn, failureDir string, ignore *ignorePatterns) func(w http.ResponseWriter, req *http.Request) {
return func(w http.ResponseWriter, req *http.Request) {
result := "failure"
defer func() {
metricFailureReportsTotal.WithLabelValues(result).Inc()
}()
lr := io.LimitReader(req.Body, maxRequestSize)
bs, err := io.ReadAll(lr)
req.Body.Close()
if err != nil {
http.Error(w, err.Error(), 500)
return
}
if ignore.match(bs) {
result = "ignored"
return
}
var reports []ur.FailureReport
err = json.Unmarshal(bs, &reports)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
if len(reports) == 0 {
// Shouldn't happen
log.Printf("Got zero failure reports")
return
}
version, err := build.ParseVersion(reports[0].Version)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
for _, r := range reports {
pkt := packet(version, "failure")
pkt.Message = r.Description
pkt.Extra = raven.Extra{
"count": r.Count,
}
for k, v := range r.Extra {
pkt.Extra[k] = v
}
if r.Goroutines != "" {
url, err := saveFailureWithGoroutines(r.FailureData, failureDir)
if err != nil {
log.Println("Saving failure report:", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
pkt.Extra["goroutinesURL"] = url
}
message := sanitizeMessageLDB(r.Description)
pkt.Fingerprint = []string{message}
if err := sendReport(dsn, pkt, userIDFor(req)); err != nil {
log.Println("Failed to send failure report:", err)
} else {
log.Println("Sent failure report:", r.Description)
result = "success"
}
}
}
}
func saveFailureWithGoroutines(data ur.FailureData, failureDir string) (string, error) {
bs := make([]byte, len(data.Description)+len(data.Goroutines))
copy(bs, data.Description)
copy(bs[len(data.Description):], data.Goroutines)
id := fmt.Sprintf("%x", sha256.Sum256(bs))
path := fullPathCompressed(failureDir, id)
err := compressAndWrite(bs, path)
if err != nil {
return "", err
}
return reportServer + path, nil
}
type ignorePatterns struct {
patterns []*regexp.Regexp
}
func loadIgnorePatterns(path string) (*ignorePatterns, error) {
bs, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var patterns []*regexp.Regexp
for _, line := range strings.Split(string(bs), "\n") {
line = strings.TrimSpace(line)
if line == "" {
continue
}
re, err := regexp.Compile(line)
if err != nil {
return nil, err
}
patterns = append(patterns, re)
}
log.Printf("Loaded %d ignore patterns", len(patterns))
return &ignorePatterns{patterns: patterns}, nil
}
func (i *ignorePatterns) match(report []byte) bool {
if i == nil {
return false
}
for _, re := range i.patterns {
if re.Match(report) {
return true
}
}
return false
}

View File

@@ -0,0 +1,40 @@
// Copyright (C) 2023 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
metricCrashReportsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "crashreceiver",
Name: "crash_reports_total",
}, []string{"result"})
metricFailureReportsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "crashreceiver",
Name: "failure_reports_total",
}, []string{"result"})
metricDiskstoreFilesTotal = promauto.NewGauge(prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "crashreceiver",
Name: "diskstore_files_total",
})
metricDiskstoreBytesTotal = promauto.NewGauge(prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "crashreceiver",
Name: "diskstore_bytes_total",
})
metricDiskstoreOldestAgeSeconds = promauto.NewGauge(prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "crashreceiver",
Name: "diskstore_oldest_age_seconds",
})
)

View File

@@ -8,14 +8,17 @@ package main
import (
"bytes"
"context"
"errors"
"io"
"log"
"regexp"
"strings"
"sync"
raven "github.com/getsentry/raven-go"
"github.com/maruel/panicparse/v2/stack"
"github.com/syncthing/syncthing/lib/build"
)
const reportServer = "https://crash.syncthing.net/report/"
@@ -31,6 +34,45 @@ var (
clientsMut sync.Mutex
)
type sentryService struct {
dsn string
inbox chan sentryRequest
}
type sentryRequest struct {
reportID string
userID string
data []byte
}
func (s *sentryService) Serve(ctx context.Context) {
for {
select {
case req := <-s.inbox:
pkt, err := parseCrashReport(req.reportID, req.data)
if err != nil {
log.Println("Failed to parse crash report:", err)
continue
}
if err := sendReport(s.dsn, pkt, req.userID); err != nil {
log.Println("Failed to send crash report:", err)
}
case <-ctx.Done():
return
}
}
}
func (s *sentryService) Send(reportID, userID string, data []byte) bool {
select {
case s.inbox <- sentryRequest{reportID, userID, data}:
return true
default:
return false
}
}
func sendReport(dsn string, pkt *raven.Packet, userID string) error {
pkt.Interfaces = append(pkt.Interfaces, &raven.User{ID: userID})
@@ -64,7 +106,7 @@ func parseCrashReport(path string, report []byte) (*raven.Packet, error) {
return nil, errors.New("no first line")
}
version, err := parseVersion(string(parts[0]))
version, err := build.ParseVersion(string(parts[0]))
if err != nil {
return nil, err
}
@@ -102,12 +144,12 @@ func parseCrashReport(path string, report []byte) (*raven.Packet, error) {
}
// Lock the source code loader to the version we are processing here.
if version.commit != "" {
if version.Commit != "" {
// We have a commit hash, so we know exactly which source to use
loader.LockWithVersion(version.commit)
} else if strings.HasPrefix(version.tag, "v") {
loader.LockWithVersion(version.Commit)
} else if strings.HasPrefix(version.Tag, "v") {
// Lets hope the tag is close enough
loader.LockWithVersion(version.tag)
loader.LockWithVersion(version.Tag)
} else {
// Last resort
loader.LockWithVersion("main")
@@ -174,89 +216,26 @@ func crashReportFingerprint(message string) []string {
return []string{"{{ default }}", message}
}
// syncthing v1.1.4-rc.1+30-g6aaae618-dirty-crashrep "Erbium Earthworm" (go1.12.5 darwin-amd64) jb@kvin.kastelo.net 2019-05-23 16:08:14 UTC [foo, bar]
var longVersionRE = regexp.MustCompile(`syncthing\s+(v[^\s]+)\s+"([^"]+)"\s\(([^\s]+)\s+([^-]+)-([^)]+)\)\s+([^\s]+)[^\[]*(?:\[(.+)\])?$`)
type version struct {
version string // "v1.1.4-rc.1+30-g6aaae618-dirty-crashrep"
tag string // "v1.1.4-rc.1"
commit string // "6aaae618", blank when absent
codename string // "Erbium Earthworm"
runtime string // "go1.12.5"
goos string // "darwin"
goarch string // "amd64"
builder string // "jb@kvin.kastelo.net"
extra []string // "foo", "bar"
}
func (v version) environment() string {
if v.commit != "" {
return "Development"
}
if strings.Contains(v.tag, "-rc.") {
return "Candidate"
}
if strings.Contains(v.tag, "-") {
return "Beta"
}
return "Stable"
}
func parseVersion(line string) (version, error) {
m := longVersionRE.FindStringSubmatch(line)
if len(m) == 0 {
return version{}, errors.New("unintelligeble version string")
}
v := version{
version: m[1],
codename: m[2],
runtime: m[3],
goos: m[4],
goarch: m[5],
builder: m[6],
}
parts := strings.Split(v.version, "+")
v.tag = parts[0]
if len(parts) > 1 {
fields := strings.Split(parts[1], "-")
if len(fields) >= 2 && strings.HasPrefix(fields[1], "g") {
v.commit = fields[1][1:]
}
}
if len(m) >= 8 && m[7] != "" {
tags := strings.Split(m[7], ",")
for i := range tags {
tags[i] = strings.TrimSpace(tags[i])
}
v.extra = tags
}
return v, nil
}
func packet(version version, reportType string) *raven.Packet {
func packet(version build.VersionParts, reportType string) *raven.Packet {
pkt := &raven.Packet{
Platform: "go",
Release: version.tag,
Environment: version.environment(),
Release: version.Tag,
Environment: version.Environment(),
Tags: raven.Tags{
raven.Tag{Key: "version", Value: version.version},
raven.Tag{Key: "tag", Value: version.tag},
raven.Tag{Key: "codename", Value: version.codename},
raven.Tag{Key: "runtime", Value: version.runtime},
raven.Tag{Key: "goos", Value: version.goos},
raven.Tag{Key: "goarch", Value: version.goarch},
raven.Tag{Key: "builder", Value: version.builder},
raven.Tag{Key: "version", Value: version.Version},
raven.Tag{Key: "tag", Value: version.Tag},
raven.Tag{Key: "codename", Value: version.Codename},
raven.Tag{Key: "runtime", Value: version.Runtime},
raven.Tag{Key: "goos", Value: version.GOOS},
raven.Tag{Key: "goarch", Value: version.GOARCH},
raven.Tag{Key: "builder", Value: version.Builder},
raven.Tag{Key: "report_type", Value: reportType},
},
}
if version.commit != "" {
pkt.Tags = append(pkt.Tags, raven.Tag{Key: "commit", Value: version.commit})
if version.Commit != "" {
pkt.Tags = append(pkt.Tags, raven.Tag{Key: "commit", Value: version.Commit})
}
for _, tag := range version.extra {
for _, tag := range version.Extra {
pkt.Tags = append(pkt.Tags, raven.Tag{Key: tag, Value: "1"})
}
return pkt

View File

@@ -12,52 +12,6 @@ import (
"testing"
)
func TestParseVersion(t *testing.T) {
cases := []struct {
longVersion string
parsed version
}{
{
longVersion: `syncthing v1.1.4-rc.1+30-g6aaae618-dirty-crashrep "Erbium Earthworm" (go1.12.5 darwin-amd64) jb@kvin.kastelo.net 2019-05-23 16:08:14 UTC`,
parsed: version{
version: "v1.1.4-rc.1+30-g6aaae618-dirty-crashrep",
tag: "v1.1.4-rc.1",
commit: "6aaae618",
codename: "Erbium Earthworm",
runtime: "go1.12.5",
goos: "darwin",
goarch: "amd64",
builder: "jb@kvin.kastelo.net",
},
},
{
longVersion: `syncthing v1.1.4-rc.1+30-g6aaae618-dirty-crashrep "Erbium Earthworm" (go1.12.5 darwin-amd64) jb@kvin.kastelo.net 2019-05-23 16:08:14 UTC [foo, bar]`,
parsed: version{
version: "v1.1.4-rc.1+30-g6aaae618-dirty-crashrep",
tag: "v1.1.4-rc.1",
commit: "6aaae618",
codename: "Erbium Earthworm",
runtime: "go1.12.5",
goos: "darwin",
goarch: "amd64",
builder: "jb@kvin.kastelo.net",
extra: []string{"foo", "bar"},
},
},
}
for _, tc := range cases {
v, err := parseVersion(tc.longVersion)
if err != nil {
t.Errorf("%s\nerror: %v\n", tc.longVersion, err)
continue
}
if fmt.Sprint(v) != fmt.Sprint(tc.parsed) {
t.Errorf("%s\nA: %v\nE: %v\n", tc.longVersion, v, tc.parsed)
}
}
}
func TestParseReport(t *testing.T) {
bs, err := os.ReadFile("_testdata/panic.log")
if err != nil {

View File

@@ -71,7 +71,6 @@ func (l *githubSourceCodeLoader) Load(filename string, line, context int) ([][]b
url := urlPrefix + l.version + filename[idx:]
resp, err := l.client.Get(url)
if err != nil {
fmt.Println("Loading source:", err)
return nil, 0

View File

@@ -0,0 +1,133 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"io"
"log"
"net/http"
"path"
"strings"
"sync"
)
type crashReceiver struct {
store *diskStore
sentry *sentryService
ignore *ignorePatterns
ignoredMut sync.RWMutex
ignored map[string]struct{}
}
func (r *crashReceiver) ServeHTTP(w http.ResponseWriter, req *http.Request) {
// The final path component should be a SHA256 hash in hex, so 64 hex
// characters. We don't care about case on the request but use lower
// case internally.
reportID := strings.ToLower(path.Base(req.URL.Path))
if len(reportID) != 64 {
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
for _, c := range reportID {
if c >= 'a' && c <= 'f' {
continue
}
if c >= '0' && c <= '9' {
continue
}
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
switch req.Method {
case http.MethodGet:
r.serveGet(reportID, w, req)
case http.MethodHead:
r.serveHead(reportID, w, req)
case http.MethodPut:
r.servePut(reportID, w, req)
default:
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
}
}
// serveGet responds to GET requests by serving the uncompressed report.
func (r *crashReceiver) serveGet(reportID string, w http.ResponseWriter, _ *http.Request) {
bs, err := r.store.Get(reportID)
if err != nil {
http.Error(w, "Not found", http.StatusNotFound)
return
}
w.Write(bs)
}
// serveHead responds to HEAD requests by checking if the named report
// already exists in the system.
func (r *crashReceiver) serveHead(reportID string, w http.ResponseWriter, _ *http.Request) {
r.ignoredMut.RLock()
_, ignored := r.ignored[reportID]
r.ignoredMut.RUnlock()
if ignored {
return // found
}
if !r.store.Exists(reportID) {
http.Error(w, "Not found", http.StatusNotFound)
}
}
// servePut accepts and stores the given report.
func (r *crashReceiver) servePut(reportID string, w http.ResponseWriter, req *http.Request) {
result := "receive_failure"
defer func() {
metricCrashReportsTotal.WithLabelValues(result).Inc()
}()
r.ignoredMut.RLock()
_, ignored := r.ignored[reportID]
r.ignoredMut.RUnlock()
if ignored {
result = "ignored_cached"
io.Copy(io.Discard, req.Body)
return // found
}
// Read at most maxRequestSize of report data.
log.Println("Receiving report", reportID)
lr := io.LimitReader(req.Body, maxRequestSize)
bs, err := io.ReadAll(lr)
if err != nil {
log.Println("Reading report:", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
if r.ignore.match(bs) {
r.ignoredMut.Lock()
if r.ignored == nil {
r.ignored = make(map[string]struct{})
}
r.ignored[reportID] = struct{}{}
r.ignoredMut.Unlock()
result = "ignored"
return
}
result = "success"
// Store the report
if !r.store.Put(reportID, bs) {
log.Println("Failed to store report (queue full):", reportID)
result = "queue_failure"
}
// Send the report to Sentry
if !r.sentry.Send(reportID, userIDFor(req), bs) {
log.Println("Failed to send report to sentry (queue full):", reportID)
result = "sentry_failure"
}
}

View File

@@ -9,14 +9,13 @@ package main
import (
"bytes"
"compress/gzip"
"crypto/sha256"
"fmt"
"net"
"net/http"
"os"
"path/filepath"
"time"
"github.com/syncthing/syncthing/lib/sha256"
)
// userIDFor returns a string we can use as the user ID for the purpose of
@@ -53,5 +52,5 @@ func compressAndWrite(bs []byte, fullPath string) error {
gw.Close()
// Create an output file with the compressed report
return os.WriteFile(fullPath, buf.Bytes(), 0644)
return os.WriteFile(fullPath, buf.Bytes(), 0o644)
}

View File

@@ -21,4 +21,3 @@ See `relaypoolsrv -help` for configuration options.
[oschwald/geoip2-golang](https://github.com/oschwald/geoip2-golang), [oschwald/maxminddb-golang](https://github.com/oschwald/maxminddb-golang), Copyright (C) 2015 [Gregory J. Oschwald](mailto:oschwald@gmail.com).
[lib/pq](https://github.com/lib/pq)</a>, Copyright (C) 2011-2013 'pq' Contributors Portions Copyright (C) 2011 Blake Mizerany.

View File

@@ -4,7 +4,7 @@
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
//go:generate go run ../../../script/genassets.go -o gui.files.go ../gui
//go:generate go run ../../../../script/genassets.go -o gui.files.go ../gui
// Package auto contains auto generated files for web assets.
package auto

View File

@@ -237,7 +237,7 @@
uptimeSeconds: 0,
};
$scope.map = L.map('map').setView([40.90296, 1.90925], 2);
L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png',
L.tileLayer('https://tile.openstreetmap.org/{z}/{x}/{y}.png',
{
attribution: 'Leaflet',
maxZoom: 17
@@ -259,7 +259,7 @@
return a.value > b.value ? 1 : -1;
}
$http.get("/endpoint").then(function(response) {
$http.get("/endpoint/full").then(function(response) {
$scope.relays = response.data.relays;
angular.forEach($scope.relays, function(relay) {
@@ -338,7 +338,7 @@
relay.showMarker = function() {
relay.marker.openPopup();
}
relay.hideMarker = function() {
relay.marker.closePopup();
}
@@ -347,7 +347,7 @@
function addCircleToMap(relay) {
console.log(relay.location.latitude)
L.circle([relay.location.latitude, relay.location.longitude],
L.circle([relay.location.latitude, relay.location.longitude],
{
radius: ((relay.stats.bytesProxied * 100) / $scope.totals.bytesProxied) * 10000,
color: "FF0000",

View File

@@ -3,14 +3,12 @@
package main
import (
"compress/gzip"
"context"
"crypto/tls"
"crypto/x509"
"encoding/json"
"flag"
"fmt"
"io"
"log"
"net"
"net/http"
@@ -19,21 +17,21 @@ import (
"path/filepath"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/golang/groupcache/lru"
"github.com/oschwald/geoip2-golang"
lru "github.com/hashicorp/golang-lru/v2"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/syncthing/syncthing/cmd/strelaypoolsrv/auto"
"github.com/syncthing/syncthing/cmd/infra/strelaypoolsrv/auto"
"github.com/syncthing/syncthing/lib/assets"
"github.com/syncthing/syncthing/lib/rand"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/geoip"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/relay/client"
"github.com/syncthing/syncthing/lib/sync"
"github.com/syncthing/syncthing/lib/tlsutil"
"golang.org/x/time/rate"
)
type location struct {
@@ -52,6 +50,10 @@ type relay struct {
StatsRetrieved time.Time `json:"statsRetrieved"`
}
type relayShort struct {
URL string `json:"url"`
}
type stats struct {
StartTime time.Time `json:"startTime"`
UptimeSeconds int `json:"uptimeSeconds"`
@@ -96,30 +98,18 @@ var (
testCert tls.Certificate
knownRelaysFile = filepath.Join(os.TempDir(), "strelaypoolsrv_known_relays")
listen = ":80"
metricsListen = ":8081"
dir string
evictionTime = time.Hour
debug bool
getLRUSize = 10 << 10
getLimitBurst = 10
getLimitAvg = 2
postLRUSize = 1 << 10
postLimitBurst = 2
postLimitAvg = 2
getLimit time.Duration
postLimit time.Duration
permRelaysFile string
ipHeader string
geoipPath string
proto string
statsRefresh = time.Minute / 2
requestQueueLen = 10
requestProcessors = 1
getMut = sync.NewMutex()
getLRUCache *lru.Cache
postMut = sync.NewMutex()
postLRUCache *lru.Cache
statsRefresh = time.Minute
requestQueueLen = 64
requestProcessors = 8
geoipLicenseKey = os.Getenv("GEOIP_LICENSE_KEY")
geoipAccountID, _ = strconv.Atoi(os.Getenv("GEOIP_ACCOUNT_ID"))
requests chan request
@@ -127,6 +117,7 @@ var (
knownRelays = make([]*relay, 0)
permanentRelays = make([]*relay, 0)
evictionTimers = make(map[string]*time.Timer)
globalBlocklist = newErrorTracker(1000)
)
const (
@@ -138,51 +129,45 @@ func main() {
log.SetFlags(log.Lshortfile)
flag.StringVar(&listen, "listen", listen, "Listen address")
flag.StringVar(&metricsListen, "metrics-listen", metricsListen, "Metrics listen address")
flag.StringVar(&dir, "keys", dir, "Directory where http-cert.pem and http-key.pem is stored for TLS listening")
flag.BoolVar(&debug, "debug", debug, "Enable debug output")
flag.DurationVar(&evictionTime, "eviction", evictionTime, "After how long the relay is evicted")
flag.IntVar(&getLRUSize, "get-limit-cache", getLRUSize, "Get request limiter cache size")
flag.IntVar(&getLimitAvg, "get-limit-avg", getLimitAvg, "Allowed average get request rate, per 10 s")
flag.IntVar(&getLimitBurst, "get-limit-burst", getLimitBurst, "Allowed burst get requests")
flag.IntVar(&postLRUSize, "post-limit-cache", postLRUSize, "Post request limiter cache size")
flag.IntVar(&postLimitAvg, "post-limit-avg", postLimitAvg, "Allowed average post request rate, per minute")
flag.IntVar(&postLimitBurst, "post-limit-burst", postLimitBurst, "Allowed burst post requests")
flag.StringVar(&permRelaysFile, "perm-relays", "", "Path to list of permanent relays")
flag.StringVar(&knownRelaysFile, "known-relays", knownRelaysFile, "Path to list of current relays")
flag.StringVar(&ipHeader, "ip-header", "", "Name of header which holds clients ip:port. Only meaningful when running behind a reverse proxy.")
flag.StringVar(&geoipPath, "geoip", "GeoLite2-City.mmdb", "Path to GeoLite2-City database")
flag.StringVar(&proto, "protocol", "tcp", "Protocol used for listening. 'tcp' for IPv4 and IPv6, 'tcp4' for IPv4, 'tcp6' for IPv6")
flag.DurationVar(&statsRefresh, "stats-refresh", statsRefresh, "Interval at which to refresh relay stats")
flag.IntVar(&requestQueueLen, "request-queue", requestQueueLen, "Queue length for incoming test requests")
flag.IntVar(&requestProcessors, "request-processors", requestProcessors, "Number of request processor routines")
flag.StringVar(&geoipLicenseKey, "geoip-license-key", geoipLicenseKey, "License key for GeoIP database")
flag.Parse()
requests = make(chan request, requestQueueLen)
getLimit = 10 * time.Second / time.Duration(getLimitAvg)
postLimit = time.Minute / time.Duration(postLimitAvg)
getLRUCache = lru.New(getLRUSize)
postLRUCache = lru.New(postLRUSize)
geoip, err := geoip.NewGeoLite2CityProvider(context.Background(), geoipAccountID, geoipLicenseKey, os.TempDir())
if err != nil {
log.Fatalln("Failed to create GeoIP provider:", err)
}
go geoip.Serve(context.TODO())
var listener net.Listener
var err error
if permRelaysFile != "" {
permanentRelays = loadRelays(permRelaysFile)
permanentRelays = loadRelays(permRelaysFile, geoip)
}
testCert = createTestCertificate()
for i := 0; i < requestProcessors; i++ {
go requestProcessor()
go requestProcessor(geoip)
}
// Load relays from cache in the background.
// Load them in a serial fashion to make sure any genuine requests
// are not dropped.
go func() {
for _, relay := range loadRelays(knownRelaysFile) {
for _, relay := range loadRelays(knownRelaysFile, geoip) {
resultChan := make(chan result)
requests <- request{relay, resultChan, nil}
result := <-resultChan
@@ -238,15 +223,40 @@ func main() {
log.Fatalln("listen:", err)
}
handler := http.NewServeMux()
handler.HandleFunc("/", handleAssets)
handler.HandleFunc("/endpoint", handleRequest)
handler.HandleFunc("/metrics", handleMetrics)
if metricsListen != "" {
mmux := http.NewServeMux()
mmux.HandleFunc("/metrics", handleMetrics)
go func() {
if err := http.ListenAndServe(metricsListen, mmux); err != nil {
log.Fatalln("HTTP serve metrics:", err)
}
}()
}
getMux := http.NewServeMux()
getMux.HandleFunc("/", handleAssets)
getMux.HandleFunc("/endpoint", withAPIMetrics(handleEndpointShort))
getMux.HandleFunc("/endpoint/full", withAPIMetrics(handleEndpointFull))
postMux := http.NewServeMux()
postMux.HandleFunc("/endpoint", withAPIMetrics(handleRegister))
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet, http.MethodHead, http.MethodOptions:
getMux.ServeHTTP(w, r)
case http.MethodPost:
postMux.ServeHTTP(w, r)
default:
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
}
})
srv := http.Server{
Handler: handler,
ReadTimeout: 10 * time.Second,
}
srv.SetKeepAlivesEnabled(false)
err = srv.Serve(listener)
if err != nil {
@@ -280,43 +290,24 @@ func handleAssets(w http.ResponseWriter, r *http.Request) {
assets.Serve(w, r, as)
}
func handleRequest(w http.ResponseWriter, r *http.Request) {
timer := prometheus.NewTimer(apiRequestsSeconds.WithLabelValues(r.Method))
w = NewLoggingResponseWriter(w)
defer func() {
timer.ObserveDuration()
lw := w.(*loggingResponseWriter)
apiRequestsTotal.WithLabelValues(r.Method, strconv.Itoa(lw.statusCode)).Inc()
}()
if ipHeader != "" {
r.RemoteAddr = r.Header.Get(ipHeader)
}
w.Header().Set("Access-Control-Allow-Origin", "*")
switch r.Method {
case "GET":
if limit(r.RemoteAddr, getLRUCache, getMut, getLimit, getLimitBurst) {
w.WriteHeader(httpStatusEnhanceYourCalm)
return
}
handleGetRequest(w, r)
case "POST":
if limit(r.RemoteAddr, postLRUCache, postMut, postLimit, postLimitBurst) {
w.WriteHeader(httpStatusEnhanceYourCalm)
return
}
handlePostRequest(w, r)
default:
if debug {
log.Println("Unhandled HTTP method", r.Method)
}
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
func withAPIMetrics(next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
timer := prometheus.NewTimer(apiRequestsSeconds.WithLabelValues(r.Method))
w = NewLoggingResponseWriter(w)
defer func() {
timer.ObserveDuration()
lw := w.(*loggingResponseWriter)
apiRequestsTotal.WithLabelValues(r.Method, strconv.Itoa(lw.statusCode)).Inc()
}()
next(w, r)
}
}
func handleGetRequest(rw http.ResponseWriter, r *http.Request) {
// handleEndpointFull returns the relay list with full metadata and
// statistics. Large, and expensive.
func handleEndpointFull(rw http.ResponseWriter, r *http.Request) {
rw.Header().Set("Content-Type", "application/json; charset=utf-8")
rw.Header().Set("Access-Control-Allow-Origin", "*")
mut.RLock()
relays := make([]*relay, len(permanentRelays)+len(knownRelays))
@@ -324,23 +315,52 @@ func handleGetRequest(rw http.ResponseWriter, r *http.Request) {
copy(relays[n:], knownRelays)
mut.RUnlock()
// Shuffle
rand.Shuffle(relays)
w := io.Writer(rw)
if strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
rw.Header().Set("Content-Encoding", "gzip")
gw := gzip.NewWriter(rw)
defer gw.Close()
w = gw
}
_ = json.NewEncoder(w).Encode(map[string][]*relay{
_ = json.NewEncoder(rw).Encode(map[string][]*relay{
"relays": relays,
})
}
func handlePostRequest(w http.ResponseWriter, r *http.Request) {
// handleEndpointShort returns the relay list with only the URL.
func handleEndpointShort(rw http.ResponseWriter, r *http.Request) {
rw.Header().Set("Content-Type", "application/json; charset=utf-8")
rw.Header().Set("Access-Control-Allow-Origin", "*")
mut.RLock()
relays := make([]relayShort, 0, len(permanentRelays)+len(knownRelays))
for _, r := range append(permanentRelays, knownRelays...) {
relays = append(relays, relayShort{URL: slimURL(r.URL)})
}
mut.RUnlock()
_ = json.NewEncoder(rw).Encode(map[string][]relayShort{
"relays": relays,
})
}
func handleRegister(w http.ResponseWriter, r *http.Request) {
// Get the IP address of the client
rhost := r.RemoteAddr
if ipHeader != "" {
hdr := r.Header.Get(ipHeader)
fields := strings.Split(hdr, ",")
if len(fields) > 0 {
rhost = strings.TrimSpace(fields[len(fields)-1])
}
}
if host, _, err := net.SplitHostPort(rhost); err == nil {
rhost = host
}
// Check the black list. A client is blacklisted if their last 10
// attempts to join have all failed. The "Unauthorized" status return
// causes strelaysrv to cease attempting to join.
if globalBlocklist.IsBlocked(rhost) {
log.Println("Rejected blocked client", rhost)
http.Error(w, "Too many errors", http.StatusUnauthorized)
globalBlocklist.ClearErrors(rhost)
return
}
var relayCert *x509.Certificate
if r.TLS != nil && len(r.TLS.PeerCertificates) > 0 {
relayCert = r.TLS.PeerCertificates[0]
@@ -392,12 +412,6 @@ func handlePostRequest(w http.ResponseWriter, r *http.Request) {
return
}
// Get the IP address of the client
rhost := r.RemoteAddr
if host, _, err := net.SplitHostPort(rhost); err == nil {
rhost = host
}
ip := net.ParseIP(host)
// The client did not provide an IP address, use the IP address of the client.
if ip == nil || ip.IsUnspecified() {
@@ -429,10 +443,14 @@ func handlePostRequest(w http.ResponseWriter, r *http.Request) {
case requests <- request{&newRelay, reschan, prometheus.NewTimer(relayTestActionsSeconds.WithLabelValues("queue"))}:
result := <-reschan
if result.err != nil {
log.Println("Join from", r.RemoteAddr, "failed:", result.err)
globalBlocklist.AddError(rhost)
relayTestsTotal.WithLabelValues("failed").Inc()
http.Error(w, result.err.Error(), http.StatusBadRequest)
return
}
log.Println("Join from", r.RemoteAddr, "succeeded")
globalBlocklist.ClearErrors(rhost)
relayTestsTotal.WithLabelValues("success").Inc()
w.Header().Set("Content-Type", "application/json; charset=utf-8")
json.NewEncoder(w).Encode(map[string]time.Duration{
@@ -448,19 +466,19 @@ func handlePostRequest(w http.ResponseWriter, r *http.Request) {
}
}
func requestProcessor() {
func requestProcessor(geoip *geoip.Provider) {
for request := range requests {
if request.queueTimer != nil {
request.queueTimer.ObserveDuration()
}
timer := prometheus.NewTimer(relayTestActionsSeconds.WithLabelValues("test"))
handleRelayTest(request)
handleRelayTest(request, geoip)
timer.ObserveDuration()
}
}
func handleRelayTest(request request) {
func handleRelayTest(request request, geoip *geoip.Provider) {
if debug {
log.Println("Request for", request.relay)
}
@@ -473,7 +491,7 @@ func handleRelayTest(request request) {
}
stats := fetchStats(request.relay)
location := getLocation(request.relay.uri.Host)
location := getLocation(request.relay.uri.Host, geoip)
mut.Lock()
if stats != nil {
@@ -546,24 +564,7 @@ func evict(relay *relay) func() {
}
}
func limit(addr string, cache *lru.Cache, lock sync.Mutex, intv time.Duration, burst int) bool {
if host, _, err := net.SplitHostPort(addr); err == nil {
addr = host
}
lock.Lock()
v, _ := cache.Get(addr)
bkt, ok := v.(*rate.Limiter)
if !ok {
bkt = rate.NewLimiter(rate.Every(intv), burst)
cache.Add(addr, bkt)
}
lock.Unlock()
return !bkt.Allow()
}
func loadRelays(file string) []*relay {
func loadRelays(file string, geoip *geoip.Provider) []*relay {
content, err := os.ReadFile(file)
if err != nil {
log.Println("Failed to load relays: " + err.Error())
@@ -587,7 +588,7 @@ func loadRelays(file string) []*relay {
relays = append(relays, &relay{
URL: line,
Location: getLocation(uri.Host),
Location: getLocation(uri.Host, geoip),
uri: uri,
})
if debug {
@@ -602,7 +603,7 @@ func saveRelays(file string, relays []*relay) error {
for _, relay := range relays {
content += relay.uri.String() + "\n"
}
return os.WriteFile(file, []byte(content), 0777)
return os.WriteFile(file, []byte(content), 0o777)
}
func createTestCertificate() tls.Certificate {
@@ -620,21 +621,16 @@ func createTestCertificate() tls.Certificate {
return cert
}
func getLocation(host string) location {
func getLocation(host string, geoip *geoip.Provider) location {
timer := prometheus.NewTimer(locationLookupSeconds)
defer timer.ObserveDuration()
db, err := geoip2.Open(geoipPath)
if err != nil {
return location{}
}
defer db.Close()
addr, err := net.ResolveTCPAddr("tcp", host)
if err != nil {
return location{}
}
city, err := db.City(addr.IP)
city, err := geoip.City(addr.IP)
if err != nil {
return location{}
}
@@ -661,3 +657,55 @@ func (lrw *loggingResponseWriter) WriteHeader(code int) {
lrw.statusCode = code
lrw.ResponseWriter.WriteHeader(code)
}
type errorTracker struct {
errors *lru.TwoQueueCache[string, *errorCounter]
}
type errorCounter struct {
count atomic.Int32
}
func newErrorTracker(size int) *errorTracker {
cache, err := lru.New2Q[string, *errorCounter](size)
if err != nil {
panic(err)
}
return &errorTracker{
errors: cache,
}
}
func (b *errorTracker) AddError(host string) {
entry, ok := b.errors.Get(host)
if !ok {
entry = &errorCounter{}
b.errors.Add(host, entry)
}
c := entry.count.Add(1)
log.Printf("Error count for %s is now %d", host, c)
}
func (b *errorTracker) ClearErrors(host string) {
b.errors.Remove(host)
}
func (b *errorTracker) IsBlocked(host string) bool {
if be, ok := b.errors.Get(host); ok {
return be.count.Load() > 10
}
return false
}
func slimURL(u string) string {
p, err := url.Parse(u)
if err != nil {
return u
}
newQuery := url.Values{}
if id := p.Query().Get("id"); id != "" {
newQuery.Set("id", id)
}
p.RawQuery = newQuery.Encode()
return p.String()
}

View File

@@ -42,7 +42,7 @@ func TestHandleGetRequest(t *testing.T) {
w := httptest.NewRecorder()
w.Body = new(bytes.Buffer)
handleGetRequest(w, httptest.NewRequest("GET", "/", nil))
handleEndpointFull(w, httptest.NewRequest("GET", "/", nil))
result := make(map[string][]*relay)
err := json.NewDecoder(w.Body).Decode(&result)
@@ -92,3 +92,18 @@ func TestCanonicalizeQueryValues(t *testing.T) {
t.Errorf("expected %q, got %q", exp, str)
}
}
func TestSlimURL(t *testing.T) {
cases := []struct {
in, out string
}{
{"http://example.com/", "http://example.com/"},
{"relay://192.0.2.42:22067/?globalLimitBps=0&id=EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M&networkTimeout=2m0s&pingInterval=1m0s&providedBy=Test&sessionLimitBps=0&statusAddr=%3A22070", "relay://192.0.2.42:22067/?id=EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M-EIC6B3M"},
}
for _, c := range cases {
if got := slimURL(c.in); got != c.out {
t.Errorf("expected %q, got %q", c.out, got)
}
}
}

View File

@@ -6,27 +6,12 @@ import (
"encoding/json"
"net"
"net/http"
"os"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/collectors"
"github.com/syncthing/syncthing/lib/sync"
)
func init() {
processCollectorOpts := collectors.ProcessCollectorOpts{
Namespace: "syncthing_relaypoolsrv",
PidFn: func() (int, error) {
return os.Getpid(), nil
},
}
prometheus.MustRegister(
collectors.NewProcessCollector(processCollectorOpts),
)
}
var (
statusClient = http.Client{
Timeout: 5 * time.Second,

View File

@@ -0,0 +1,351 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"log/slog"
"net"
"net/http"
"os"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
"github.com/alecthomas/kong"
"github.com/prometheus/client_golang/prometheus/promhttp"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/httpcache"
"github.com/syncthing/syncthing/lib/upgrade"
)
type cli struct {
Listen string `default:":8080" help:"Listen address"`
MetricsListen string `default:":8082" help:"Listen address for metrics"`
URL string `short:"u" default:"https://api.github.com/repos/syncthing/syncthing/releases?per_page=25" help:"GitHub releases url"`
Forward []string `short:"f" help:"Forwarded pages, format: /path->https://example/com/url"`
CacheTime time.Duration `default:"15m" help:"Cache time"`
}
func main() {
var params cli
kong.Parse(&params)
slog.SetDefault(slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
})))
if err := server(&params); err != nil {
fmt.Printf("Error: %v\n", err)
os.Exit(1)
}
}
func server(params *cli) error {
if params.MetricsListen != "" {
mux := http.NewServeMux()
mux.Handle("/metrics", promhttp.Handler())
metricsListen, err := net.Listen("tcp", params.MetricsListen)
if err != nil {
return fmt.Errorf("metrics: %w", err)
}
slog.Info("Metrics listener started", "addr", params.MetricsListen)
go func() {
if err := http.Serve(metricsListen, mux); err != nil {
slog.Warn("Metrics server returned", "error", err)
}
}()
}
cache := &cachedReleases{url: params.URL}
if err := cache.Update(context.Background()); err != nil {
return fmt.Errorf("initial cache update: %w", err)
} else {
slog.Info("Initial cache update done")
}
go func() {
for range time.NewTicker(params.CacheTime).C {
slog.Info("Refreshing cached releases", "url", params.URL)
if err := cache.Update(context.Background()); err != nil {
slog.Error("Failed to refresh cached releases", "url", params.URL, "error", err)
}
}
}()
ghRels := &githubReleases{cache: cache}
mux := http.NewServeMux()
mux.HandleFunc("/ping", ghRels.servePing)
mux.HandleFunc("/meta.json", ghRels.serveReleases)
for _, fwd := range params.Forward {
path, url, ok := strings.Cut(fwd, "->")
if !ok {
return fmt.Errorf("invalid forward: %q", fwd)
}
slog.Info("Forwarding", "from", path, "to", url)
name := strings.ReplaceAll(path, "/", "_")
mux.Handle(path, httpcache.SinglePath(&proxy{name: name, url: url}, params.CacheTime))
}
srv := &http.Server{
Addr: params.Listen,
Handler: mux,
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
}
srv.SetKeepAlivesEnabled(false)
srvListener, err := net.Listen("tcp", params.Listen)
if err != nil {
return fmt.Errorf("listen: %w", err)
}
slog.Info("Main listener started", "addr", params.Listen)
return srv.Serve(srvListener)
}
type githubReleases struct {
cache *cachedReleases
}
func (p *githubReleases) servePing(w http.ResponseWriter, req *http.Request) {
rels := p.cache.Releases()
if len(rels) == 0 {
http.Error(w, "No releases available", http.StatusServiceUnavailable)
return
}
w.Header().Set("Syncthing-Num-Releases", strconv.Itoa(len(rels)))
w.WriteHeader(http.StatusOK)
}
func (p *githubReleases) serveReleases(w http.ResponseWriter, req *http.Request) {
rels := p.cache.Releases()
ua := req.Header.Get("User-Agent")
osv := req.Header.Get("Syncthing-Os-Version")
if ua != "" && osv != "" {
// We should determine the compatibility of the releases.
rels = filterForCompabitility(rels, ua, osv)
} else {
metricFilterCalls.WithLabelValues("no-ua-or-osversion").Inc()
}
rels = filterForLatest(rels)
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Allow-Methods", "GET")
w.Header().Set("Cache-Control", "public, max-age=900")
w.Header().Set("Vary", "User-Agent, Syncthing-Os-Version")
_ = json.NewEncoder(w).Encode(rels)
metricUpgradeChecks.Inc()
}
type proxy struct {
name string
url string
}
func (p *proxy) ServeHTTP(w http.ResponseWriter, req *http.Request) {
req, err := http.NewRequestWithContext(req.Context(), http.MethodGet, p.url, nil)
if err != nil {
metricHTTPRequests.WithLabelValues(p.name, "error").Inc()
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
metricHTTPRequests.WithLabelValues(p.name, "error").Inc()
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer resp.Body.Close()
metricHTTPRequests.WithLabelValues(p.name, "success").Inc()
ct := resp.Header.Get("Content-Type")
w.Header().Set("Content-Type", ct)
if resp.StatusCode == http.StatusOK {
w.Header().Set("Cache-Control", "public, max-age=900")
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Allow-Methods", "GET")
}
w.WriteHeader(resp.StatusCode)
if strings.HasPrefix(ct, "application/json") {
// Special JSON handling; clean it up a bit.
var v interface{}
if err := json.NewDecoder(resp.Body).Decode(&v); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
_ = json.NewEncoder(w).Encode(v)
} else {
_, _ = io.Copy(w, resp.Body)
}
}
// filterForLatest returns the latest stable and prerelease only. If the
// stable version is newer (comes first in the list) there is no need to go
// looking for a prerelease at all.
func filterForLatest(rels []upgrade.Release) []upgrade.Release {
var filtered []upgrade.Release
var havePre bool
for _, rel := range rels {
if !rel.Prerelease {
// We found a stable version, we're good now.
filtered = append(filtered, rel)
break
}
if rel.Prerelease && !havePre {
// We remember the first prerelease we find.
filtered = append(filtered, rel)
havePre = true
}
}
return filtered
}
var userAgentOSArchExp = regexp.MustCompile(`^syncthing.*\(.+ (\w+)-(\w+)\)$`)
func filterForCompabitility(rels []upgrade.Release, ua, osv string) []upgrade.Release {
osArch := userAgentOSArchExp.FindStringSubmatch(ua)
if len(osArch) != 3 {
metricFilterCalls.WithLabelValues("bad-os-arch").Inc()
return rels
}
os := osArch[1]
var filtered []upgrade.Release
for _, rel := range rels {
if rel.Compatibility == nil {
// No requirements means it's compatible with everything.
filtered = append(filtered, rel)
continue
}
req, ok := rel.Compatibility.Requirements[os]
if !ok {
// No entry for the current OS means it's compatible.
filtered = append(filtered, rel)
continue
}
if upgrade.CompareVersions(osv, req) >= 0 {
filtered = append(filtered, rel)
continue
}
}
if len(filtered) != len(rels) {
metricFilterCalls.WithLabelValues("filtered").Inc()
} else {
metricFilterCalls.WithLabelValues("unchanged").Inc()
}
return filtered
}
type cachedReleases struct {
url string
mut sync.RWMutex
current []upgrade.Release
}
func (c *cachedReleases) Releases() []upgrade.Release {
c.mut.RLock()
defer c.mut.RUnlock()
return c.current
}
func (c *cachedReleases) Update(ctx context.Context) error {
rels, err := fetchGithubReleases(ctx, c.url)
if err != nil {
return err
}
c.mut.Lock()
c.current = rels
c.mut.Unlock()
return nil
}
func fetchGithubReleases(ctx context.Context, url string) ([]upgrade.Release, error) {
req, err := http.NewRequestWithContext(context.TODO(), http.MethodGet, url, nil)
if err != nil {
metricHTTPRequests.WithLabelValues("github-releases", "error").Inc()
return nil, err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
metricHTTPRequests.WithLabelValues("github-releases", "error").Inc()
return nil, err
}
defer resp.Body.Close()
var rels []upgrade.Release
if err := json.NewDecoder(resp.Body).Decode(&rels); err != nil {
metricHTTPRequests.WithLabelValues("github-releases", "error").Inc()
return nil, err
}
metricHTTPRequests.WithLabelValues("github-releases", "success").Inc()
// Move the URL used for browser downloads to the URL field, and remove
// the browser URL field. This avoids going via the GitHub API for
// downloads, since Syncthing uses the URL field.
for _, rel := range rels {
for j, asset := range rel.Assets {
rel.Assets[j].URL = asset.BrowserURL
rel.Assets[j].BrowserURL = ""
}
}
addReleaseCompatibility(ctx, rels)
sort.Sort(upgrade.SortByRelease(rels))
return rels, nil
}
func addReleaseCompatibility(ctx context.Context, rels []upgrade.Release) {
for i := range rels {
rel := &rels[i]
for i, asset := range rel.Assets {
if asset.Name != "compat.json" {
continue
}
// Load compat.json into the Compatibility field
req, err := http.NewRequestWithContext(ctx, http.MethodGet, asset.URL, nil)
if err != nil {
metricHTTPRequests.WithLabelValues("compat-json", "error").Inc()
break
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
metricHTTPRequests.WithLabelValues("compat-json", "error").Inc()
break
}
if resp.StatusCode != http.StatusOK {
metricHTTPRequests.WithLabelValues("compat-json", "error").Inc()
resp.Body.Close()
break
}
_ = json.NewDecoder(io.LimitReader(resp.Body, 10<<10)).Decode(&rel.Compatibility)
metricHTTPRequests.WithLabelValues("compat-json", "success").Inc()
resp.Body.Close()
// Remove compat.json from the asset list since it's been processed
rel.Assets = append(rel.Assets[:i], rel.Assets[i+1:]...)
break
}
}
}

View File

@@ -0,0 +1,30 @@
// Copyright (C) 2024 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
metricUpgradeChecks = promauto.NewCounter(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "upgrade",
Name: "metadata_requests",
})
metricFilterCalls = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "upgrade",
Name: "filter_calls",
}, []string{"result"})
metricHTTPRequests = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "upgrade",
Name: "http_requests",
}, []string{"target", "result"})
)

33
cmd/infra/ursrv/main.go Normal file
View File

@@ -0,0 +1,33 @@
// Copyright (C) 2023 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"log"
"log/slog"
"os"
"github.com/alecthomas/kong"
"github.com/syncthing/syncthing/cmd/infra/ursrv/serve"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
)
type CLI struct {
Serve serve.CLI `cmd:"" default:""`
}
func main() {
slog.SetDefault(slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
})))
var cli CLI
ctx := kong.Parse(&cli)
if err := ctx.Run(); err != nil {
log.Fatalf("%s: %v", ctx.Command(), err)
}
}

View File

@@ -4,7 +4,7 @@
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
package serve
import "testing"

View File

@@ -0,0 +1,46 @@
// Copyright (C) 2023 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package serve
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
metricReportsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "ursrv_v2",
Name: "incoming_reports_total",
}, []string{"result"})
metricsCollectsTotal = promauto.NewCounter(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "ursrv_v2",
Name: "collects_total",
})
metricsCollectSecondsTotal = promauto.NewCounter(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "ursrv_v2",
Name: "collect_seconds_total",
})
metricsCollectSecondsLast = promauto.NewGauge(prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "ursrv_v2",
Name: "collect_seconds_last",
})
metricsWriteSecondsLast = promauto.NewGauge(prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "ursrv_v2",
Name: "write_seconds_last",
})
)
func init() {
metricReportsTotal.WithLabelValues("fail")
metricReportsTotal.WithLabelValues("replace")
metricReportsTotal.WithLabelValues("accept")
}

View File

@@ -0,0 +1,314 @@
// Copyright (C) 2024 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package serve
import (
"reflect"
"slices"
"strconv"
"strings"
"sync"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/syncthing/syncthing/lib/ur/contract"
)
const namePrefix = "syncthing_usage_"
type metricsSet struct {
srv *server
gauges map[string]prometheus.Gauge
gaugeVecs map[string]*prometheus.GaugeVec
gaugeVecLabels map[string][]string
summaries map[string]*metricSummary
collectMut sync.Mutex
collectCutoff time.Duration
}
func newMetricsSet(srv *server) *metricsSet {
s := &metricsSet{
srv: srv,
gauges: make(map[string]prometheus.Gauge),
gaugeVecs: make(map[string]*prometheus.GaugeVec),
gaugeVecLabels: make(map[string][]string),
summaries: make(map[string]*metricSummary),
collectCutoff: -24 * time.Hour,
}
var initForType func(reflect.Type)
initForType = func(t reflect.Type) {
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
if field.Type.Kind() == reflect.Struct {
initForType(field.Type)
continue
}
name, typ, label := fieldNameTypeLabel(field)
sname, labels := nameConstLabels(name)
switch typ {
case "gauge":
s.gauges[name] = prometheus.NewGauge(prometheus.GaugeOpts{
Name: namePrefix + sname,
ConstLabels: labels,
})
case "summary":
s.summaries[name] = newMetricSummary(namePrefix+sname, nil, labels)
case "gaugeVec":
s.gaugeVecLabels[name] = append(s.gaugeVecLabels[name], label)
case "summaryVec":
s.summaries[name] = newMetricSummary(namePrefix+sname, []string{label}, labels)
}
}
}
initForType(reflect.ValueOf(contract.Report{}).Type())
for name, labels := range s.gaugeVecLabels {
s.gaugeVecs[name] = prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: namePrefix + name,
}, labels)
}
return s
}
func fieldNameTypeLabel(rf reflect.StructField) (string, string, string) {
metric := rf.Tag.Get("metric")
name, typ, ok := strings.Cut(metric, ",")
if !ok {
return "", "", ""
}
gv, label, ok := strings.Cut(typ, ":")
if ok {
typ = gv
}
return name, typ, label
}
func nameConstLabels(name string) (string, prometheus.Labels) {
if name == "-" {
return "", nil
}
name, labels, ok := strings.Cut(name, "{")
if !ok {
return name, nil
}
lls := strings.Split(labels[:len(labels)-1], ",")
m := make(map[string]string)
for _, l := range lls {
k, v, _ := strings.Cut(l, "=")
m[k] = v
}
return name, m
}
func (s *metricsSet) addReport(r *contract.Report) {
gaugeVecs := make(map[string][]string)
s.addReportStruct(reflect.ValueOf(r).Elem(), gaugeVecs)
for name, lv := range gaugeVecs {
s.gaugeVecs[name].WithLabelValues(lv...).Add(1)
}
}
func (s *metricsSet) addReportStruct(v reflect.Value, gaugeVecs map[string][]string) {
t := v.Type()
for i := 0; i < v.NumField(); i++ {
field := v.Field(i)
if field.Kind() == reflect.Struct {
s.addReportStruct(field, gaugeVecs)
continue
}
name, typ, label := fieldNameTypeLabel(t.Field(i))
switch typ {
case "gauge":
switch v := field.Interface().(type) {
case int:
s.gauges[name].Add(float64(v))
case string:
s.gaugeVecs[name].WithLabelValues(v).Add(1)
case bool:
if v {
s.gauges[name].Add(1)
}
}
case "gaugeVec":
var labelValue string
switch v := field.Interface().(type) {
case string:
labelValue = v
case int:
labelValue = strconv.Itoa(v)
case map[string]int:
for k, v := range v {
labelValue = k
field.SetInt(int64(v))
break
}
}
if _, ok := gaugeVecs[name]; !ok {
gaugeVecs[name] = make([]string, len(s.gaugeVecLabels[name]))
}
for i, l := range s.gaugeVecLabels[name] {
if l == label {
gaugeVecs[name][i] = labelValue
break
}
}
case "summary", "summaryVec":
switch v := field.Interface().(type) {
case int:
s.summaries[name].Observe("", float64(v))
case float64:
s.summaries[name].Observe("", v)
case []int:
for _, v := range v {
s.summaries[name].Observe("", float64(v))
}
case map[string]int:
for k, v := range v {
if k == "" {
// avoid empty string labels as those are the sign
// of a non-vec summary
k = "unknown"
}
s.summaries[name].Observe(k, float64(v))
}
}
}
}
}
func (s *metricsSet) Describe(c chan<- *prometheus.Desc) {
for _, g := range s.gauges {
g.Describe(c)
}
for _, g := range s.gaugeVecs {
g.Describe(c)
}
for _, g := range s.summaries {
g.Describe(c)
}
}
func (s *metricsSet) Collect(c chan<- prometheus.Metric) {
s.collectMut.Lock()
defer s.collectMut.Unlock()
t0 := time.Now()
defer func() {
dur := time.Since(t0).Seconds()
metricsCollectSecondsLast.Set(dur)
metricsCollectSecondsTotal.Add(dur)
metricsCollectsTotal.Inc()
}()
for _, g := range s.gauges {
g.Set(0)
}
for _, g := range s.gaugeVecs {
g.Reset()
}
for _, g := range s.summaries {
g.Reset()
}
cutoff := time.Now().Add(s.collectCutoff)
s.srv.reports.Range(func(key string, r *contract.Report) bool {
if s.collectCutoff < 0 && r.Received.Before(cutoff) {
s.srv.reports.Delete(key)
return true
}
s.addReport(r)
return true
})
for _, g := range s.gauges {
c <- g
}
for _, g := range s.gaugeVecs {
g.Collect(c)
}
for _, g := range s.summaries {
g.Collect(c)
}
}
type metricSummary struct {
name string
values map[string][]float64
zeroes map[string]int
qDesc *prometheus.Desc
countDesc *prometheus.Desc
sumDesc *prometheus.Desc
zDesc *prometheus.Desc
}
func newMetricSummary(name string, labels []string, constLabels prometheus.Labels) *metricSummary {
return &metricSummary{
name: name,
values: make(map[string][]float64),
zeroes: make(map[string]int),
qDesc: prometheus.NewDesc(name, "", append(labels, "quantile"), constLabels),
countDesc: prometheus.NewDesc(name+"_nonzero_count", "", labels, constLabels),
sumDesc: prometheus.NewDesc(name+"_sum", "", labels, constLabels),
zDesc: prometheus.NewDesc(name+"_zero_count", "", labels, constLabels),
}
}
func (q *metricSummary) Observe(labelValue string, v float64) {
if v == 0 {
q.zeroes[labelValue]++
return
}
q.values[labelValue] = append(q.values[labelValue], v)
}
func (q *metricSummary) Describe(c chan<- *prometheus.Desc) {
c <- q.qDesc
c <- q.countDesc
c <- q.sumDesc
c <- q.zDesc
}
func (q *metricSummary) Collect(c chan<- prometheus.Metric) {
for lv, vs := range q.values {
var labelVals []string
if lv != "" {
labelVals = []string{lv}
}
c <- prometheus.MustNewConstMetric(q.countDesc, prometheus.GaugeValue, float64(len(vs)), labelVals...)
c <- prometheus.MustNewConstMetric(q.zDesc, prometheus.GaugeValue, float64(q.zeroes[lv]), labelVals...)
var sum float64
for _, v := range vs {
sum += v
}
c <- prometheus.MustNewConstMetric(q.sumDesc, prometheus.GaugeValue, sum, labelVals...)
if len(vs) == 0 {
return
}
slices.Sort(vs)
c <- prometheus.MustNewConstMetric(q.qDesc, prometheus.GaugeValue, vs[0], append(labelVals, "0")...)
c <- prometheus.MustNewConstMetric(q.qDesc, prometheus.GaugeValue, vs[len(vs)*5/100], append(labelVals, "0.05")...)
c <- prometheus.MustNewConstMetric(q.qDesc, prometheus.GaugeValue, vs[len(vs)/2], append(labelVals, "0.5")...)
c <- prometheus.MustNewConstMetric(q.qDesc, prometheus.GaugeValue, vs[len(vs)*9/10], append(labelVals, "0.9")...)
c <- prometheus.MustNewConstMetric(q.qDesc, prometheus.GaugeValue, vs[len(vs)*95/100], append(labelVals, "0.95")...)
c <- prometheus.MustNewConstMetric(q.qDesc, prometheus.GaugeValue, vs[len(vs)-1], append(labelVals, "1")...)
}
}
func (q *metricSummary) Reset() {
clear(q.values)
clear(q.zeroes)
}

View File

@@ -0,0 +1,406 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package serve
import (
"bufio"
"compress/gzip"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net"
"net/http"
_ "net/http/pprof"
"os"
"regexp"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/puzpuzpuz/xsync/v3"
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/geoip"
"github.com/syncthing/syncthing/lib/s3"
"github.com/syncthing/syncthing/lib/ur/contract"
)
type CLI struct {
Listen string `env:"UR_LISTEN" help:"Usage reporting & metrics endpoint listen address" default:"0.0.0.0:8080"`
ListenInternal string `env:"UR_LISTEN_INTERNAL" help:"Internal metrics endpoint listen address" default:"0.0.0.0:8082"`
GeoIPLicenseKey string `env:"UR_GEOIP_LICENSE_KEY"`
GeoIPAccountID int `env:"UR_GEOIP_ACCOUNT_ID"`
DumpFile string `env:"UR_DUMP_FILE" default:"reports.jsons.gz"`
DumpInterval time.Duration `env:"UR_DUMP_INTERVAL" default:"5m"`
S3Endpoint string `name:"s3-endpoint" hidden:"true" env:"UR_S3_ENDPOINT"`
S3Region string `name:"s3-region" hidden:"true" env:"UR_S3_REGION"`
S3Bucket string `name:"s3-bucket" hidden:"true" env:"UR_S3_BUCKET"`
S3AccessKeyID string `name:"s3-access-key-id" hidden:"true" env:"UR_S3_ACCESS_KEY_ID"`
S3SecretKey string `name:"s3-secret-key" hidden:"true" env:"UR_S3_SECRET_KEY"`
}
var (
compilerRe = regexp.MustCompile(`\(([A-Za-z0-9()., -]+) \w+-\w+(?:| android| default)\) ([\w@.-]+)`)
knownDistributions = []distributionMatch{
// Maps well known builders to the official distribution method that
// they represent
{regexp.MustCompile(`\steamcity@build\.syncthing\.net`), "GitHub"},
{regexp.MustCompile(`\sjenkins@build\.syncthing\.net`), "GitHub"},
{regexp.MustCompile(`\sbuilder@github\.syncthing\.net`), "GitHub"},
{regexp.MustCompile(`\sdeb@build\.syncthing\.net`), "APT"},
{regexp.MustCompile(`\sdebian@github\.syncthing\.net`), "APT"},
{regexp.MustCompile(`\sdocker@syncthing\.net`), "Docker Hub"},
{regexp.MustCompile(`\sdocker@build.syncthing\.net`), "Docker Hub"},
{regexp.MustCompile(`\sdocker@github.syncthing\.net`), "Docker Hub"},
{regexp.MustCompile(`\sandroid-builder@github\.syncthing\.net`), "Google Play"},
{regexp.MustCompile(`\sandroid-.*teamcity@build\.syncthing\.net`), "Google Play"},
{regexp.MustCompile(`\sandroid-.*vagrant@basebox-stretch64`), "F-Droid"},
{regexp.MustCompile(`\svagrant@bullseye`), "F-Droid"},
{regexp.MustCompile(`\svagrant@bookworm`), "F-Droid"},
{regexp.MustCompile(`Anwender@NET2017`), "Syncthing-Fork (3rd party)"},
{regexp.MustCompile(`\sbuilduser@(archlinux|svetlemodry)`), "Arch (3rd party)"},
{regexp.MustCompile(`\ssyncthing@archlinux`), "Arch (3rd party)"},
{regexp.MustCompile(`@debian`), "Debian (3rd party)"},
{regexp.MustCompile(`@fedora`), "Fedora (3rd party)"},
{regexp.MustCompile(`\sbrew@`), "Homebrew (3rd party)"},
{regexp.MustCompile(`\sroot@buildkitsandbox`), "LinuxServer.io (3rd party)"},
{regexp.MustCompile(`\sports@freebsd`), "FreeBSD (3rd party)"},
{regexp.MustCompile(`\snix@nix`), "Nix (3rd party)"},
{regexp.MustCompile(`.`), "Others"},
}
)
type distributionMatch struct {
matcher *regexp.Regexp
distribution string
}
func (cli *CLI) Run() error {
slog.Info("Starting", "version", build.Version)
// Listening
urListener, err := net.Listen("tcp", cli.Listen)
if err != nil {
slog.Error("Failed to listen (usage reports)", "error", err)
return err
}
slog.Info("Listening (usage reports)", "address", urListener.Addr())
internalListener, err := net.Listen("tcp", cli.ListenInternal)
if err != nil {
slog.Error("Failed to listen (internal)", "error", err)
return err
}
slog.Info("Listening (internal)", "address", internalListener.Addr())
var geo *geoip.Provider
if cli.GeoIPAccountID != 0 && cli.GeoIPLicenseKey != "" {
geo, err = geoip.NewGeoLite2CityProvider(context.Background(), cli.GeoIPAccountID, cli.GeoIPLicenseKey, os.TempDir())
if err != nil {
slog.Error("Failed to load GeoIP", "error", err)
return err
}
go geo.Serve(context.TODO())
}
// s3
var s3sess *s3.Session
if cli.S3Endpoint != "" {
s3sess, err = s3.NewSession(cli.S3Endpoint, cli.S3Region, cli.S3Bucket, cli.S3AccessKeyID, cli.S3SecretKey)
if err != nil {
slog.Error("Failed to create S3 session", "error", err)
return err
}
}
if _, err := os.Stat(cli.DumpFile); err != nil && s3sess != nil {
if err := cli.downloadDumpFile(s3sess); err != nil {
slog.Error("Failed to download dump file", "error", err)
}
}
// server
srv := &server{
geo: geo,
reports: xsync.NewMapOf[string, *contract.Report](),
}
if fd, err := os.Open(cli.DumpFile); err == nil {
gr, err := gzip.NewReader(fd)
if err == nil {
srv.load(gr)
}
fd.Close()
}
go func() {
for range time.Tick(cli.DumpInterval) {
if err := cli.saveDumpFile(srv, s3sess); err != nil {
slog.Error("Failed to write dump file", "error", err)
}
}
}()
// The internal metrics endpoint just serves metrics about what the
// server is doing.
http.Handle("/metrics", promhttp.Handler())
internalSrv := http.Server{
ReadTimeout: 5 * time.Second,
WriteTimeout: 15 * time.Second,
}
go internalSrv.Serve(internalListener)
// New external metrics endpoint accepts reports from clients and serves
// aggregated usage reporting metrics.
ms := newMetricsSet(srv)
reg := prometheus.NewRegistry()
reg.MustRegister(ms)
mux := http.NewServeMux()
mux.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{}))
mux.HandleFunc("/newdata", srv.handleNewData)
mux.HandleFunc("/ping", srv.handlePing)
metricsSrv := http.Server{
ReadTimeout: 5 * time.Second,
WriteTimeout: 15 * time.Second,
Handler: mux,
}
slog.Info("Ready to serve")
return metricsSrv.Serve(urListener)
}
func (cli *CLI) downloadDumpFile(s3sess *s3.Session) error {
latestKey, err := s3sess.LatestKey()
if err != nil {
return fmt.Errorf("list latest S3 key: %w", err)
}
fd, err := os.Create(cli.DumpFile)
if err != nil {
return fmt.Errorf("create dump file: %w", err)
}
if err := s3sess.Download(fd, latestKey); err != nil {
_ = fd.Close()
return fmt.Errorf("download dump file: %w", err)
}
if err := fd.Close(); err != nil {
return fmt.Errorf("close dump file: %w", err)
}
slog.Info("Dump file downloaded", "key", latestKey)
return nil
}
func (cli *CLI) saveDumpFile(srv *server, s3sess *s3.Session) error {
fd, err := os.Create(cli.DumpFile + ".tmp")
if err != nil {
return fmt.Errorf("creating dump file: %w", err)
}
gw := gzip.NewWriter(fd)
if err := srv.save(gw); err != nil {
return fmt.Errorf("saving dump file: %w", err)
}
if err := gw.Close(); err != nil {
fd.Close()
return fmt.Errorf("closing gzip writer: %w", err)
}
if err := fd.Close(); err != nil {
return fmt.Errorf("closing dump file: %w", err)
}
if err := os.Rename(cli.DumpFile+".tmp", cli.DumpFile); err != nil {
return fmt.Errorf("renaming dump file: %w", err)
}
slog.Info("Dump file saved")
if s3sess != nil {
key := fmt.Sprintf("reports-%s.jsons.gz", time.Now().UTC().Format("2006-01-02"))
fd, err := os.Open(cli.DumpFile)
if err != nil {
return fmt.Errorf("opening dump file: %w", err)
}
if err := s3sess.Upload(fd, key); err != nil {
return fmt.Errorf("uploading dump file: %w", err)
}
_ = fd.Close()
slog.Info("Dump file uploaded")
}
return nil
}
type server struct {
geo *geoip.Provider
reports *xsync.MapOf[string, *contract.Report]
}
func (s *server) handlePing(w http.ResponseWriter, r *http.Request) {
}
func (s *server) handleNewData(w http.ResponseWriter, r *http.Request) {
result := "fail"
defer func() {
// result is "accept" (new report), "replace" (existing report) or
// "fail"
metricReportsTotal.WithLabelValues(result).Inc()
}()
defer r.Body.Close()
if r.Method != http.MethodPost {
http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
return
}
addr := r.Header.Get("X-Forwarded-For")
if addr != "" {
addr = strings.Split(addr, ", ")[0]
} else {
addr = r.RemoteAddr
}
if host, _, err := net.SplitHostPort(addr); err == nil {
addr = host
}
log := slog.With("addr", addr)
if net.ParseIP(addr) == nil {
addr = ""
}
var rep contract.Report
lr := &io.LimitedReader{R: r.Body, N: 40 * 1024}
bs, _ := io.ReadAll(lr)
if err := json.Unmarshal(bs, &rep); err != nil {
log.Error("Failed to decode JSON", "error", err)
http.Error(w, "JSON Decode Error", http.StatusInternalServerError)
return
}
rep.Received = time.Now()
rep.Date = rep.Received.UTC().Format("20060102")
rep.Address = addr
if err := rep.Validate(); err != nil {
log.Error("Failed to validate report", "error", err)
http.Error(w, "Validation Error", http.StatusInternalServerError)
return
}
if s.addReport(&rep) {
result = "replace"
} else {
result = "accept"
}
}
func (s *server) addReport(rep *contract.Report) bool {
if s.geo != nil {
if ip := net.ParseIP(rep.Address); ip != nil {
if city, err := s.geo.City(ip); err == nil {
rep.Country = city.Country.Names["en"]
rep.CountryCode = city.Country.IsoCode
}
}
}
if rep.Country == "" {
rep.Country = "Unknown"
}
if rep.CountryCode == "" {
rep.CountryCode = "ZZ"
}
rep.Version = transformVersion(rep.Version)
if strings.Contains(rep.Version, ".") {
split := strings.SplitN(rep.Version, ".", 3)
if len(split) == 3 {
rep.MajorVersion = strings.Join(split[:2], ".")
}
}
rep.OS, rep.Arch, _ = strings.Cut(rep.Platform, "-")
if m := compilerRe.FindStringSubmatch(rep.LongVersion); len(m) == 3 {
rep.Compiler = m[1]
rep.Builder = m[2]
}
for _, d := range knownDistributions {
if d.matcher.MatchString(rep.LongVersion) {
rep.Distribution = d.distribution
break
}
}
_, loaded := s.reports.LoadAndStore(rep.UniqueID, rep)
return loaded
}
func (s *server) save(w io.Writer) error {
bw := bufio.NewWriter(w)
enc := json.NewEncoder(bw)
var err error
s.reports.Range(func(k string, v *contract.Report) bool {
err = enc.Encode(v)
return err == nil
})
if err != nil {
return err
}
return bw.Flush()
}
func (s *server) load(r io.Reader) {
dec := json.NewDecoder(r)
s.reports.Clear()
for {
var rep contract.Report
if err := dec.Decode(&rep); errors.Is(err, io.EOF) {
break
} else if err != nil {
slog.Error("Failed to load record", "error", err)
break
}
s.addReport(&rep)
}
slog.Info("Loaded reports", "count", s.reports.Size())
}
var (
plusRe = regexp.MustCompile(`(\+.*|[.-]dev\..*)$`)
plusStr = "-dev"
)
// transformVersion returns a version number formatted correctly, with all
// development versions aggregated into one.
func transformVersion(v string) string {
if v == "unknown-dev" {
return v
}
if !strings.HasPrefix(v, "v") {
v = "v" + v
}
v = plusRe.ReplaceAllString(v, plusStr)
return v
}

View File

@@ -1,125 +0,0 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
// Command stcrashreceiver is a trivial HTTP server that allows two things:
//
// - uploading files (crash reports) named like a SHA256 hash using a PUT request
// - checking whether such file exists using a HEAD request
//
// Typically this should be deployed behind something that manages HTTPS.
package main
import (
"encoding/json"
"flag"
"fmt"
"io"
"log"
"net/http"
"os"
"path/filepath"
"github.com/syncthing/syncthing/lib/sha256"
"github.com/syncthing/syncthing/lib/ur"
raven "github.com/getsentry/raven-go"
)
const maxRequestSize = 1 << 20 // 1 MiB
func main() {
dir := flag.String("dir", ".", "Parent directory to store crash and failure reports in")
dsn := flag.String("dsn", "", "Sentry DSN")
listen := flag.String("listen", ":22039", "HTTP listen address")
flag.Parse()
mux := http.NewServeMux()
cr := &crashReceiver{
dir: filepath.Join(*dir, "crash_reports"),
dsn: *dsn,
}
mux.Handle("/", cr)
if *dsn != "" {
mux.HandleFunc("/newcrash/failure", handleFailureFn(*dsn, filepath.Join(*dir, "failure_reports")))
}
log.SetOutput(os.Stdout)
if err := http.ListenAndServe(*listen, mux); err != nil {
log.Fatalln("HTTP serve:", err)
}
}
func handleFailureFn(dsn, failureDir string) func(w http.ResponseWriter, req *http.Request) {
return func(w http.ResponseWriter, req *http.Request) {
lr := io.LimitReader(req.Body, maxRequestSize)
bs, err := io.ReadAll(lr)
req.Body.Close()
if err != nil {
http.Error(w, err.Error(), 500)
return
}
var reports []ur.FailureReport
err = json.Unmarshal(bs, &reports)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
if len(reports) == 0 {
// Shouldn't happen
log.Printf("Got zero failure reports")
return
}
version, err := parseVersion(reports[0].Version)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
for _, r := range reports {
pkt := packet(version, "failure")
pkt.Message = r.Description
pkt.Extra = raven.Extra{
"count": r.Count,
}
for k, v := range r.Extra {
pkt.Extra[k] = v
}
if r.Goroutines != "" {
url, err := saveFailureWithGoroutines(r.FailureData, failureDir)
if err != nil {
log.Println("Saving failure report:", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
pkt.Extra["goroutinesURL"] = url
}
message := sanitizeMessageLDB(r.Description)
pkt.Fingerprint = []string{message}
if err := sendReport(dsn, pkt, userIDFor(req)); err != nil {
log.Println("Failed to send failure report:", err)
} else {
log.Println("Sent failure report:", r.Description)
}
}
}
}
func saveFailureWithGoroutines(data ur.FailureData, failureDir string) (string, error) {
bs := make([]byte, len(data.Description)+len(data.Goroutines))
copy(bs, data.Description)
copy(bs[len(data.Description):], data.Goroutines)
id := fmt.Sprintf("%x", sha256.Sum256(bs))
path := fullPathCompressed(failureDir, id)
err := compressAndWrite(bs, path)
if err != nil {
return "", err
}
return reportServer + path, nil
}

View File

@@ -1,141 +0,0 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"compress/gzip"
"io"
"log"
"net/http"
"os"
"path"
"path/filepath"
"strings"
)
type crashReceiver struct {
dir string
dsn string
}
func (r *crashReceiver) ServeHTTP(w http.ResponseWriter, req *http.Request) {
// The final path component should be a SHA256 hash in hex, so 64 hex
// characters. We don't care about case on the request but use lower
// case internally.
reportID := strings.ToLower(path.Base(req.URL.Path))
if len(reportID) != 64 {
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
for _, c := range reportID {
if c >= 'a' && c <= 'f' {
continue
}
if c >= '0' && c <= '9' {
continue
}
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
// The location of the report on disk, compressed
fullPath := filepath.Join(r.dir, r.dirFor(reportID), reportID) + ".gz"
switch req.Method {
case http.MethodGet:
r.serveGet(fullPath, w, req)
case http.MethodHead:
r.serveHead(fullPath, w, req)
case http.MethodPut:
r.servePut(reportID, fullPath, w, req)
default:
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
}
}
// serveGet responds to GET requests by serving the uncompressed report.
func (*crashReceiver) serveGet(fullPath string, w http.ResponseWriter, _ *http.Request) {
fd, err := os.Open(fullPath)
if err != nil {
http.Error(w, "Not found", http.StatusNotFound)
return
}
defer fd.Close()
gr, err := gzip.NewReader(fd)
if err != nil {
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
_, _ = io.Copy(w, gr) // best effort
}
// serveHead responds to HEAD requests by checking if the named report
// already exists in the system.
func (*crashReceiver) serveHead(fullPath string, w http.ResponseWriter, _ *http.Request) {
if _, err := os.Lstat(fullPath); err != nil {
http.Error(w, "Not found", http.StatusNotFound)
}
}
// servePut accepts and stores the given report.
func (r *crashReceiver) servePut(reportID, fullPath string, w http.ResponseWriter, req *http.Request) {
// Ensure the destination directory exists
if err := os.MkdirAll(filepath.Dir(fullPath), 0755); err != nil {
log.Println("Creating directory:", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
// Read at most maxRequestSize of report data.
log.Println("Receiving report", reportID)
lr := io.LimitReader(req.Body, maxRequestSize)
bs, err := io.ReadAll(lr)
if err != nil {
log.Println("Reading report:", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
// Compress the report for storage
buf := new(bytes.Buffer)
gw := gzip.NewWriter(buf)
_, _ = gw.Write(bs) // can't fail
gw.Close()
// Create an output file with the compressed report
err = os.WriteFile(fullPath, buf.Bytes(), 0644)
if err != nil {
log.Println("Saving report:", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
// Send the report to Sentry
if r.dsn != "" {
// Remote ID
user := userIDFor(req)
go func() {
// There's no need for the client to have to wait for this part.
pkt, err := parseCrashReport(reportID, bs)
if err != nil {
log.Println("Failed to parse crash report:", err)
return
}
if err := sendReport(r.dsn, pkt, user); err != nil {
log.Println("Failed to send crash report:", err)
}
}()
}
}
// 01234567890abcdef... => 01/23
func (*crashReceiver) dirFor(base string) string {
return filepath.Join(base[0:2], base[2:4])
}

261
cmd/stdiscosrv/amqp.go Normal file
View File

@@ -0,0 +1,261 @@
// Copyright (C) 2024 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"context"
"fmt"
"io"
"log"
amqp "github.com/rabbitmq/amqp091-go"
"github.com/thejerf/suture/v4"
"google.golang.org/protobuf/proto"
"github.com/syncthing/syncthing/internal/gen/discosrv"
"github.com/syncthing/syncthing/internal/protoutil"
"github.com/syncthing/syncthing/lib/protocol"
)
type amqpReplicator struct {
suture.Service
broker string
sender *amqpSender
receiver *amqpReceiver
outbox chan *discosrv.ReplicationRecord
}
func newAMQPReplicator(broker, clientID string, db database) *amqpReplicator {
svc := suture.New("amqpReplicator", suture.Spec{PassThroughPanics: true})
sender := &amqpSender{
broker: broker,
clientID: clientID,
outbox: make(chan *discosrv.ReplicationRecord, replicationOutboxSize),
}
svc.Add(sender)
receiver := &amqpReceiver{
broker: broker,
clientID: clientID,
db: db,
}
svc.Add(receiver)
return &amqpReplicator{
Service: svc,
broker: broker,
sender: sender,
receiver: receiver,
outbox: make(chan *discosrv.ReplicationRecord, replicationOutboxSize),
}
}
func (s *amqpReplicator) send(key *protocol.DeviceID, ps []*discosrv.DatabaseAddress, seen int64) {
s.sender.send(key, ps, seen)
}
type amqpSender struct {
broker string
clientID string
outbox chan *discosrv.ReplicationRecord
}
func (s *amqpSender) Serve(ctx context.Context) error {
conn, ch, err := amqpChannel(s.broker)
if err != nil {
return err
}
defer ch.Close()
defer conn.Close()
buf := make([]byte, 1024)
for {
select {
case rec := <-s.outbox:
size := proto.Size(rec)
if len(buf) < size {
buf = make([]byte, size)
}
n, err := protoutil.MarshalTo(buf, rec)
if err != nil {
replicationSendsTotal.WithLabelValues("error").Inc()
return fmt.Errorf("replication marshal: %w", err)
}
err = ch.PublishWithContext(ctx,
"discovery", // exchange
"", // routing key
false, // mandatory
false, // immediate
amqp.Publishing{
ContentType: "application/protobuf",
Body: buf[:n],
AppId: s.clientID,
})
if err != nil {
replicationSendsTotal.WithLabelValues("error").Inc()
return fmt.Errorf("replication publish: %w", err)
}
replicationSendsTotal.WithLabelValues("success").Inc()
case <-ctx.Done():
return nil
}
}
}
func (s *amqpSender) String() string {
return fmt.Sprintf("amqpSender(%q)", s.broker)
}
func (s *amqpSender) send(key *protocol.DeviceID, ps []*discosrv.DatabaseAddress, seen int64) {
item := &discosrv.ReplicationRecord{
Key: key[:],
Addresses: ps,
Seen: seen,
}
// The send should never block. The inbox is suitably buffered for at
// least a few seconds of stalls, which shouldn't happen in practice.
select {
case s.outbox <- item:
default:
replicationSendsTotal.WithLabelValues("drop").Inc()
}
}
type amqpReceiver struct {
broker string
clientID string
db database
}
func (s *amqpReceiver) Serve(ctx context.Context) error {
conn, ch, err := amqpChannel(s.broker)
if err != nil {
return err
}
defer ch.Close()
defer conn.Close()
msgs, err := amqpConsume(ch)
if err != nil {
return err
}
for {
select {
case msg, ok := <-msgs:
if !ok {
return fmt.Errorf("subscription closed: %w", io.EOF)
}
// ignore messages from ourself
if msg.AppId == s.clientID {
continue
}
var rec discosrv.ReplicationRecord
if err := proto.Unmarshal(msg.Body, &rec); err != nil {
replicationRecvsTotal.WithLabelValues("error").Inc()
return fmt.Errorf("replication unmarshal: %w", err)
}
id, err := protocol.DeviceIDFromBytes(rec.Key)
if err != nil {
id, err = protocol.DeviceIDFromString(string(rec.Key))
}
if err != nil {
log.Println("Replication device ID:", err)
replicationRecvsTotal.WithLabelValues("error").Inc()
continue
}
if err := s.db.merge(&id, rec.Addresses, rec.Seen); err != nil {
return fmt.Errorf("replication database merge: %w", err)
}
replicationRecvsTotal.WithLabelValues("success").Inc()
case <-ctx.Done():
return nil
}
}
}
func (s *amqpReceiver) String() string {
return fmt.Sprintf("amqpReceiver(%q)", s.broker)
}
func amqpChannel(dst string) (*amqp.Connection, *amqp.Channel, error) {
conn, err := amqp.Dial(dst)
if err != nil {
return nil, nil, fmt.Errorf("AMQP dial: %w", err)
}
ch, err := conn.Channel()
if err != nil {
return nil, nil, fmt.Errorf("AMQP channel: %w", err)
}
err = ch.ExchangeDeclare(
"discovery", // name
"fanout", // type
false, // durable
false, // auto-deleted
false, // internal
false, // no-wait
nil, // arguments
)
if err != nil {
return nil, nil, fmt.Errorf("AMQP declare exchange: %w", err)
}
return conn, ch, nil
}
func amqpConsume(ch *amqp.Channel) (<-chan amqp.Delivery, error) {
q, err := ch.QueueDeclare(
"", // name
false, // durable
false, // delete when unused
true, // exclusive
false, // no-wait
nil, // arguments
)
if err != nil {
return nil, fmt.Errorf("AMQP declare queue: %w", err)
}
err = ch.QueueBind(
q.Name, // queue name
"", // routing key
"discovery", // exchange
false,
nil,
)
if err != nil {
return nil, fmt.Errorf("AMQP bind queue: %w", err)
}
msgs, err := ch.Consume(
q.Name, // queue
"", // consumer
true, // auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
if err != nil {
return nil, fmt.Errorf("AMQP consume: %w", err)
}
return msgs, nil
}

View File

@@ -8,6 +8,7 @@ package main
import (
"bytes"
"compress/gzip"
"context"
"crypto/tls"
"encoding/base64"
@@ -15,18 +16,21 @@ import (
"encoding/pem"
"errors"
"fmt"
io "io"
"log"
"math/rand"
"net"
"net/http"
"net/url"
"sort"
"slices"
"strconv"
"strings"
"sync"
"time"
"github.com/syncthing/syncthing/internal/gen/discosrv"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/stringutil"
)
// announcement is the format received from and sent to clients
@@ -36,15 +40,20 @@ type announcement struct {
}
type apiSrv struct {
addr string
cert tls.Certificate
db database
listener net.Listener
repl replicator // optional
useHTTP bool
addr string
cert tls.Certificate
db database
listener net.Listener
repl replicator // optional
useHTTP bool
compression bool
gzipWriters sync.Pool
seenTracker *retryAfterTracker
notSeenTracker *retryAfterTracker
}
mapsMut sync.Mutex
misses map[string]int32
type replicator interface {
send(key *protocol.DeviceID, addrs []*discosrv.DatabaseAddress, seen int64)
}
type requestID int64
@@ -57,18 +66,30 @@ type contextKey int
const idKey contextKey = iota
func newAPISrv(addr string, cert tls.Certificate, db database, repl replicator, useHTTP bool) *apiSrv {
func newAPISrv(addr string, cert tls.Certificate, db database, repl replicator, useHTTP, compression bool) *apiSrv {
return &apiSrv{
addr: addr,
cert: cert,
db: db,
repl: repl,
useHTTP: useHTTP,
misses: make(map[string]int32),
addr: addr,
cert: cert,
db: db,
repl: repl,
useHTTP: useHTTP,
compression: compression,
seenTracker: &retryAfterTracker{
name: "seenTracker",
bucketStarts: time.Now(),
desiredRate: 250,
currentDelay: notFoundRetryUnknownMinSeconds,
},
notSeenTracker: &retryAfterTracker{
name: "notSeenTracker",
bucketStarts: time.Now(),
desiredRate: 250,
currentDelay: notFoundRetryUnknownMaxSeconds / 2,
},
}
}
func (s *apiSrv) Serve(_ context.Context) error {
func (s *apiSrv) Serve(ctx context.Context) error {
if s.useHTTP {
listener, err := net.Listen("tcp", s.addr)
if err != nil {
@@ -78,18 +99,10 @@ func (s *apiSrv) Serve(_ context.Context) error {
s.listener = listener
} else {
tlsCfg := &tls.Config{
Certificates: []tls.Certificate{s.cert},
ClientAuth: tls.RequestClientCert,
SessionTicketsDisabled: true,
MinVersion: tls.VersionTLS12,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
},
Certificates: []tls.Certificate{s.cert},
ClientAuth: tls.RequestClientCert,
MinVersion: tls.VersionTLS12,
NextProtos: []string{"h2", "http/1.1"},
}
tlsListener, err := tls.Listen("tcp", s.addr, tlsCfg)
@@ -108,6 +121,14 @@ func (s *apiSrv) Serve(_ context.Context) error {
WriteTimeout: httpWriteTimeout,
MaxHeaderBytes: httpMaxHeaderBytes,
}
if !debug {
srv.ErrorLog = log.New(io.Discard, "", 0)
}
go func() {
<-ctx.Done()
srv.Shutdown(context.Background())
}()
err := srv.Serve(s.listener)
if err != nil {
@@ -116,8 +137,6 @@ func (s *apiSrv) Serve(_ context.Context) error {
return err
}
var topCtx = context.Background()
func (s *apiSrv) handler(w http.ResponseWriter, req *http.Request) {
t0 := time.Now()
@@ -130,10 +149,10 @@ func (s *apiSrv) handler(w http.ResponseWriter, req *http.Request) {
}()
reqID := requestID(rand.Int63())
ctx := context.WithValue(topCtx, idKey, reqID)
req = req.WithContext(context.WithValue(req.Context(), idKey, reqID))
if debug {
log.Println(reqID, req.Method, req.URL)
log.Println(reqID, req.Method, req.URL, req.Proto)
}
remoteAddr := &net.TCPAddr{
@@ -142,7 +161,12 @@ func (s *apiSrv) handler(w http.ResponseWriter, req *http.Request) {
}
if s.useHTTP {
remoteAddr.IP = net.ParseIP(req.Header.Get("X-Forwarded-For"))
// X-Forwarded-For can have multiple client IPs; split using the comma separator
forwardIP, _, _ := strings.Cut(req.Header.Get("X-Forwarded-For"), ",")
// net.ParseIP will return nil if leading/trailing whitespace exists; use strings.TrimSpace()
remoteAddr.IP = net.ParseIP(strings.TrimSpace(forwardIP))
if parsedPort, err := strconv.ParseInt(req.Header.Get("X-Client-Port"), 10, 0); err == nil {
remoteAddr.Port = int(parsedPort)
}
@@ -159,22 +183,22 @@ func (s *apiSrv) handler(w http.ResponseWriter, req *http.Request) {
}
switch req.Method {
case "GET":
s.handleGET(ctx, lw, req)
case "POST":
s.handlePOST(ctx, remoteAddr, lw, req)
case http.MethodGet:
s.handleGET(lw, req)
case http.MethodPost:
s.handlePOST(remoteAddr, lw, req)
default:
http.Error(lw, "Method Not Allowed", http.StatusMethodNotAllowed)
}
}
func (s *apiSrv) handleGET(ctx context.Context, w http.ResponseWriter, req *http.Request) {
reqID := ctx.Value(idKey).(requestID)
func (s *apiSrv) handleGET(w http.ResponseWriter, req *http.Request) {
reqID := req.Context().Value(idKey).(requestID)
deviceID, err := protocol.DeviceIDFromString(req.URL.Query().Get("device"))
if err != nil {
if debug {
log.Println(reqID, "bad device param")
log.Println(reqID, "bad device param:", err)
}
lookupRequestsTotal.WithLabelValues("bad_request").Inc()
w.Header().Set("Retry-After", errorRetryAfterString())
@@ -182,8 +206,7 @@ func (s *apiSrv) handleGET(ctx context.Context, w http.ResponseWriter, req *http
return
}
key := deviceID.String()
rec, err := s.db.get(key)
rec, err := s.db.get(&deviceID)
if err != nil {
// some sort of internal error
lookupRequestsTotal.WithLabelValues("internal_error").Inc()
@@ -193,43 +216,46 @@ func (s *apiSrv) handleGET(ctx context.Context, w http.ResponseWriter, req *http
}
if len(rec.Addresses) == 0 {
lookupRequestsTotal.WithLabelValues("not_found").Inc()
s.mapsMut.Lock()
misses := s.misses[key]
if misses < rec.Misses {
misses = rec.Misses + 1
var afterS int
if rec.Seen == 0 {
afterS = s.notSeenTracker.retryAfterS()
lookupRequestsTotal.WithLabelValues("not_found_ever").Inc()
} else {
misses++
afterS = s.seenTracker.retryAfterS()
lookupRequestsTotal.WithLabelValues("not_found_recent").Inc()
}
s.misses[key] = misses
s.mapsMut.Unlock()
if misses%notFoundMissesWriteInterval == 0 {
rec.Misses = misses
rec.Missed = time.Now().UnixNano()
rec.Addresses = nil
// rec.Seen retained from get
s.db.put(key, rec)
}
w.Header().Set("Retry-After", notFoundRetryAfterString(int(misses)))
w.Header().Set("Retry-After", strconv.Itoa(afterS))
http.Error(w, "Not Found", http.StatusNotFound)
return
}
lookupRequestsTotal.WithLabelValues("success").Inc()
bs, _ := json.Marshal(announcement{
Seen: time.Unix(0, rec.Seen),
w.Header().Set("Content-Type", "application/json")
var bw io.Writer = w
// Use compression if the client asks for it
if s.compression && strings.Contains(req.Header.Get("Accept-Encoding"), "gzip") {
gw, ok := s.gzipWriters.Get().(*gzip.Writer)
if ok {
gw.Reset(w)
} else {
gw = gzip.NewWriter(w)
}
w.Header().Set("Content-Encoding", "gzip")
defer gw.Close()
defer s.gzipWriters.Put(gw)
bw = gw
}
json.NewEncoder(bw).Encode(announcement{
Seen: time.Unix(0, rec.Seen).Truncate(time.Second),
Addresses: addressStrs(rec.Addresses),
})
w.Header().Set("Content-Type", "application/json")
w.Write(bs)
}
func (s *apiSrv) handlePOST(ctx context.Context, remoteAddr *net.TCPAddr, w http.ResponseWriter, req *http.Request) {
reqID := ctx.Value(idKey).(requestID)
func (s *apiSrv) handlePOST(remoteAddr *net.TCPAddr, w http.ResponseWriter, req *http.Request) {
reqID := req.Context().Value(idKey).(requestID)
rawCert, err := certificateBytes(req)
if err != nil {
@@ -257,6 +283,9 @@ func (s *apiSrv) handlePOST(ctx context.Context, remoteAddr *net.TCPAddr, w http
addresses := fixupAddresses(remoteAddr, ann.Addresses)
if len(addresses) == 0 {
if debug {
log.Println(reqID, "no addresses")
}
announceRequestsTotal.WithLabelValues("bad_request").Inc()
w.Header().Set("Retry-After", errorRetryAfterString())
http.Error(w, "Bad Request", http.StatusBadRequest)
@@ -264,6 +293,9 @@ func (s *apiSrv) handlePOST(ctx context.Context, remoteAddr *net.TCPAddr, w http
}
if err := s.handleAnnounce(deviceID, addresses); err != nil {
if debug {
log.Println(reqID, "handle:", err)
}
announceRequestsTotal.WithLabelValues("internal_error").Inc()
w.Header().Set("Retry-After", errorRetryAfterString())
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
@@ -274,6 +306,9 @@ func (s *apiSrv) handlePOST(ctx context.Context, remoteAddr *net.TCPAddr, w http
w.Header().Set("Reannounce-After", reannounceAfterString())
w.WriteHeader(http.StatusNoContent)
if debug {
log.Println(reqID, "announced", deviceID, addresses)
}
}
func (s *apiSrv) Stop() {
@@ -281,29 +316,31 @@ func (s *apiSrv) Stop() {
}
func (s *apiSrv) handleAnnounce(deviceID protocol.DeviceID, addresses []string) error {
key := deviceID.String()
now := time.Now()
expire := now.Add(addressExpiryTime).UnixNano()
dbAddrs := make([]DatabaseAddress, len(addresses))
for i := range addresses {
dbAddrs[i].Address = addresses[i]
dbAddrs[i].Expires = expire
}
// The address slice must always be sorted for database merges to work
// properly.
sort.Sort(databaseAddressOrder(dbAddrs))
slices.Sort(addresses)
addresses = slices.Compact(addresses)
dbAddrs := make([]*discosrv.DatabaseAddress, len(addresses))
for i := range addresses {
dbAddrs[i] = &discosrv.DatabaseAddress{
Address: addresses[i],
Expires: expire,
}
}
seen := now.UnixNano()
if s.repl != nil {
s.repl.send(key, dbAddrs, seen)
s.repl.send(&deviceID, dbAddrs, seen)
}
return s.db.merge(key, dbAddrs, seen)
return s.db.merge(&deviceID, dbAddrs, seen)
}
func handlePing(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(204)
w.WriteHeader(http.StatusNoContent)
}
func certificateBytes(req *http.Request) ([]byte, error) {
@@ -349,24 +386,44 @@ func certificateBytes(req *http.Request) ([]byte, error) {
}
bs = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: hdr})
} else if hdr := req.Header.Get("X-Forwarded-Tls-Client-Cert"); hdr != "" {
} else if cert := req.Header.Get("X-Forwarded-Tls-Client-Cert"); cert != "" {
// Traefik 2 passtlsclientcert
// The certificate is in PEM format with url encoding but without newlines
// and start/end statements. We need to decode, reinstate the newlines every 64
//
// The certificate is in PEM format, maybe with URL encoding
// (depends on Traefik version) but without newlines and start/end
// statements. We need to decode, reinstate the newlines every 64
// character and add statements for the PEM decoder
hdr, err := url.QueryUnescape(hdr)
if err != nil {
// Decoding failed
return nil, err
if strings.Contains(cert, "%") {
if unesc, err := url.QueryUnescape(cert); err == nil {
cert = unesc
}
}
for i := 64; i < len(hdr); i += 65 {
hdr = hdr[:i] + "\n" + hdr[i:]
const (
header = "-----BEGIN CERTIFICATE-----"
footer = "-----END CERTIFICATE-----"
)
var b bytes.Buffer
b.Grow(len(header) + 1 + len(cert) + len(cert)/64 + 1 + len(footer) + 1)
b.WriteString(header)
b.WriteByte('\n')
for i := 0; i < len(cert); i += 64 {
end := i + 64
if end > len(cert) {
end = len(cert)
}
b.WriteString(cert[i:end])
b.WriteByte('\n')
}
hdr = "-----BEGIN CERTIFICATE-----\n" + hdr
hdr = hdr + "\n-----END CERTIFICATE-----\n"
bs = []byte(hdr)
b.WriteString(footer)
b.WriteByte('\n')
bs = b.Bytes()
}
if bs == nil {
@@ -404,13 +461,13 @@ func fixupAddresses(remote *net.TCPAddr, addresses []string) []string {
continue
}
if remote != nil {
if host == "" || ip.IsUnspecified() {
if host == "" || ip.IsUnspecified() {
if remote != nil {
// Replace the unspecified IP with the request source.
// ... unless the request source is the loopback address or
// multicast/unspecified (can't happen, really).
if remote.IP.IsLoopback() || remote.IP.IsMulticast() || remote.IP.IsUnspecified() {
if remote.IP == nil || remote.IP.IsLoopback() || remote.IP.IsMulticast() || remote.IP.IsUnspecified() {
continue
}
@@ -426,11 +483,21 @@ func fixupAddresses(remote *net.TCPAddr, addresses []string) []string {
}
host = remote.IP.String()
}
// If zero port was specified, use remote port.
if port == "0" && remote.Port > 0 {
} else {
// remote is nil, unable to determine host IP
continue
}
}
// If zero port was specified, use remote port.
if port == "0" {
if remote != nil && remote.Port > 0 {
// use remote port
port = strconv.Itoa(remote.Port)
} else {
// unable to determine remote port
continue
}
}
@@ -438,6 +505,9 @@ func fixupAddresses(remote *net.TCPAddr, addresses []string) []string {
fixed = append(fixed, uri.String())
}
// Remove duplicate addresses
fixed = stringutil.UniqueTrimmedStrings(fixed)
return fixed
}
@@ -455,7 +525,7 @@ func (lrw *loggingResponseWriter) WriteHeader(code int) {
lrw.ResponseWriter.WriteHeader(code)
}
func addressStrs(dbAddrs []DatabaseAddress) []string {
func addressStrs(dbAddrs []*discosrv.DatabaseAddress) []string {
res := make([]string, len(dbAddrs))
for i, a := range dbAddrs {
res[i] = a.Address
@@ -467,15 +537,44 @@ func errorRetryAfterString() string {
return strconv.Itoa(errorRetryAfterSeconds + rand.Intn(errorRetryFuzzSeconds))
}
func notFoundRetryAfterString(misses int) string {
retryAfterS := notFoundRetryMinSeconds + notFoundRetryIncSeconds*misses
if retryAfterS > notFoundRetryMaxSeconds {
retryAfterS = notFoundRetryMaxSeconds
}
retryAfterS += rand.Intn(notFoundRetryFuzzSeconds)
return strconv.Itoa(retryAfterS)
}
func reannounceAfterString() string {
return strconv.Itoa(reannounceAfterSeconds + rand.Intn(reannounzeFuzzSeconds))
}
type retryAfterTracker struct {
name string
desiredRate float64 // requests per second
mut sync.Mutex
lastCount int // requests in the last bucket
curCount int // requests in the current bucket
bucketStarts time.Time // start of the current bucket
currentDelay int // current delay in seconds
}
func (t *retryAfterTracker) retryAfterS() int {
now := time.Now()
t.mut.Lock()
if durS := now.Sub(t.bucketStarts).Seconds(); durS > float64(t.currentDelay) {
t.bucketStarts = now
t.lastCount = t.curCount
lastRate := float64(t.lastCount) / durS
switch {
case t.currentDelay > notFoundRetryUnknownMinSeconds &&
lastRate < 0.75*t.desiredRate:
t.currentDelay = max(8*t.currentDelay/10, notFoundRetryUnknownMinSeconds)
case t.currentDelay < notFoundRetryUnknownMaxSeconds &&
lastRate > 1.25*t.desiredRate:
t.currentDelay = min(3*t.currentDelay/2, notFoundRetryUnknownMaxSeconds)
}
t.curCount = 0
}
if t.curCount == 0 {
retryAfterLevel.WithLabelValues(t.name).Set(float64(t.currentDelay))
}
t.curCount++
t.mut.Unlock()
return t.currentDelay + rand.Intn(t.currentDelay/4)
}

View File

@@ -7,9 +7,20 @@
package main
import (
"context"
"crypto/tls"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"regexp"
"strings"
"testing"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/tlsutil"
)
func TestFixupAddresses(t *testing.T) {
@@ -69,6 +80,14 @@ func TestFixupAddresses(t *testing.T) {
remote: addr("123.123.123.123", 9000),
in: []string{"tcp://44.44.44.44:0"},
out: []string{"tcp://44.44.44.44:9000"},
}, { // remote ip nil
remote: addr("", 9000),
in: []string{"tcp://:22000", "tcp://44.44.44.44:9000"},
out: []string{"tcp://44.44.44.44:9000"},
}, { // remote port 0
remote: addr("123.123.123.123", 0),
in: []string{"tcp://:22000", "tcp://44.44.44.44"},
out: []string{"tcp://123.123.123.123:22000"},
},
}
@@ -86,3 +105,79 @@ func addr(host string, port int) *net.TCPAddr {
Port: port,
}
}
func BenchmarkAPIRequests(b *testing.B) {
db := newInMemoryStore(b.TempDir(), 0, nil)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go db.Serve(ctx)
api := newAPISrv("127.0.0.1:0", tls.Certificate{}, db, nil, true, true)
srv := httptest.NewServer(http.HandlerFunc(api.handler))
kf := b.TempDir() + "/cert"
crt, err := tlsutil.NewCertificate(kf+".crt", kf+".key", "localhost", 7)
if err != nil {
b.Fatal(err)
}
certBs, err := os.ReadFile(kf + ".crt")
if err != nil {
b.Fatal(err)
}
certBs = regexp.MustCompile(`---[^\n]+---\n`).ReplaceAll(certBs, nil)
certString := string(strings.ReplaceAll(string(certBs), "\n", " "))
devID := protocol.NewDeviceID(crt.Certificate[0])
devIDString := devID.String()
b.Run("Announce", func(b *testing.B) {
b.ReportAllocs()
url := srv.URL + "/v2/?device=" + devIDString
for i := 0; i < b.N; i++ {
req, _ := http.NewRequest(http.MethodPost, url, strings.NewReader(`{"addresses":["tcp://10.10.10.10:42000"]}`))
req.Header.Set("X-Forwarded-Tls-Client-Cert", certString)
resp, err := http.DefaultClient.Do(req)
if err != nil {
b.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
b.Fatalf("unexpected status %s", resp.Status)
}
}
})
b.Run("Lookup", func(b *testing.B) {
b.ReportAllocs()
url := srv.URL + "/v2/?device=" + devIDString
for i := 0; i < b.N; i++ {
req, _ := http.NewRequest(http.MethodGet, url, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
b.Fatal(err)
}
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
b.Fatalf("unexpected status %s", resp.Status)
}
}
})
b.Run("LookupNoCompression", func(b *testing.B) {
b.ReportAllocs()
url := srv.URL + "/v2/?device=" + devIDString
for i := 0; i < b.N; i++ {
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Accept-Encoding", "identity") // disable compression
resp, err := http.DefaultClient.Do(req)
if err != nil {
b.Fatal(err)
}
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
b.Fatalf("unexpected status %s", resp.Status)
}
}
})
}

View File

@@ -4,19 +4,31 @@
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
//go:generate go run ../../proto/scripts/protofmt.go database.proto
//go:generate protoc -I ../../ -I . --gogofast_out=. database.proto
package main
import (
"bufio"
"cmp"
"context"
"encoding/binary"
"errors"
"io"
"log"
"sort"
"os"
"path"
"runtime"
"slices"
"strings"
"time"
"github.com/syndtr/goleveldb/leveldb"
"github.com/syndtr/goleveldb/leveldb/util"
"github.com/puzpuzpuz/xsync/v3"
"google.golang.org/protobuf/proto"
"github.com/syncthing/syncthing/internal/gen/discosrv"
"github.com/syncthing/syncthing/internal/protoutil"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/rand"
"github.com/syncthing/syncthing/lib/s3"
)
type clock interface {
@@ -30,347 +42,401 @@ func (defaultClock) Now() time.Time {
}
type database interface {
put(key string, rec DatabaseRecord) error
merge(key string, addrs []DatabaseAddress, seen int64) error
get(key string) (DatabaseRecord, error)
put(key *protocol.DeviceID, rec *discosrv.DatabaseRecord) error
merge(key *protocol.DeviceID, addrs []*discosrv.DatabaseAddress, seen int64) error
get(key *protocol.DeviceID) (*discosrv.DatabaseRecord, error)
}
type levelDBStore struct {
db *leveldb.DB
inbox chan func()
clock clock
marshalBuf []byte
type inMemoryStore struct {
m *xsync.MapOf[protocol.DeviceID, *discosrv.DatabaseRecord]
dir string
flushInterval time.Duration
s3 *s3.Session
objKey string
clock clock
}
func newLevelDBStore(dir string) (*levelDBStore, error) {
db, err := leveldb.OpenFile(dir, levelDBOptions)
func newInMemoryStore(dir string, flushInterval time.Duration, s3sess *s3.Session) *inMemoryStore {
hn, err := os.Hostname()
if err != nil {
return nil, err
hn = rand.String(8)
}
return &levelDBStore{
db: db,
inbox: make(chan func(), 16),
clock: defaultClock{},
}, nil
}
func (s *levelDBStore) put(key string, rec DatabaseRecord) error {
t0 := time.Now()
defer func() {
databaseOperationSeconds.WithLabelValues(dbOpPut).Observe(time.Since(t0).Seconds())
}()
rc := make(chan error)
s.inbox <- func() {
size := rec.Size()
if len(s.marshalBuf) < size {
s.marshalBuf = make([]byte, size)
s := &inMemoryStore{
m: xsync.NewMapOf[protocol.DeviceID, *discosrv.DatabaseRecord](),
dir: dir,
flushInterval: flushInterval,
s3: s3sess,
objKey: hn + ".db",
clock: defaultClock{},
}
nr, err := s.read()
if os.IsNotExist(err) && s3sess != nil {
// Try to read from AWS
latestKey, cerr := s3sess.LatestKey()
if cerr != nil {
log.Println("Error reading database from S3:", err)
return s
}
n, _ := rec.MarshalTo(s.marshalBuf)
rc <- s.db.Put([]byte(key), s.marshalBuf[:n], nil)
fd, cerr := os.Create(path.Join(s.dir, "records.db"))
if cerr != nil {
log.Println("Error creating database file:", err)
return s
}
if cerr := s3sess.Download(fd, latestKey); cerr != nil {
log.Printf("Error reading database from S3: %v", err)
}
_ = fd.Close()
nr, err = s.read()
}
err := <-rc
if err != nil {
databaseOperations.WithLabelValues(dbOpPut, dbResError).Inc()
} else {
databaseOperations.WithLabelValues(dbOpPut, dbResSuccess).Inc()
log.Println("Error reading database:", err)
}
return err
log.Printf("Read %d records from database", nr)
s.expireAndCalculateStatistics()
return s
}
func (s *levelDBStore) merge(key string, addrs []DatabaseAddress, seen int64) error {
func (s *inMemoryStore) put(key *protocol.DeviceID, rec *discosrv.DatabaseRecord) error {
t0 := time.Now()
defer func() {
databaseOperationSeconds.WithLabelValues(dbOpMerge).Observe(time.Since(t0).Seconds())
}()
s.m.Store(*key, rec)
databaseOperations.WithLabelValues(dbOpPut, dbResSuccess).Inc()
databaseOperationSeconds.WithLabelValues(dbOpPut).Observe(time.Since(t0).Seconds())
return nil
}
rc := make(chan error)
newRec := DatabaseRecord{
func (s *inMemoryStore) merge(key *protocol.DeviceID, addrs []*discosrv.DatabaseAddress, seen int64) error {
t0 := time.Now()
newRec := &discosrv.DatabaseRecord{
Addresses: addrs,
Seen: seen,
}
s.inbox <- func() {
// grab the existing record
oldRec, err := s.get(key)
if err != nil {
// "not found" is not an error from get, so this is serious
// stuff only
rc <- err
return
}
newRec = merge(newRec, oldRec)
// We replicate s.put() functionality here ourselves instead of
// calling it because we want to serialize our get above together
// with the put in the same function.
size := newRec.Size()
if len(s.marshalBuf) < size {
s.marshalBuf = make([]byte, size)
}
n, _ := newRec.MarshalTo(s.marshalBuf)
rc <- s.db.Put([]byte(key), s.marshalBuf[:n], nil)
if oldRec, ok := s.m.Load(*key); ok {
newRec = merge(oldRec, newRec)
}
s.m.Store(*key, newRec)
err := <-rc
if err != nil {
databaseOperations.WithLabelValues(dbOpMerge, dbResError).Inc()
} else {
databaseOperations.WithLabelValues(dbOpMerge, dbResSuccess).Inc()
}
databaseOperations.WithLabelValues(dbOpMerge, dbResSuccess).Inc()
databaseOperationSeconds.WithLabelValues(dbOpMerge).Observe(time.Since(t0).Seconds())
return err
return nil
}
func (s *levelDBStore) get(key string) (DatabaseRecord, error) {
func (s *inMemoryStore) get(key *protocol.DeviceID) (*discosrv.DatabaseRecord, error) {
t0 := time.Now()
defer func() {
databaseOperationSeconds.WithLabelValues(dbOpGet).Observe(time.Since(t0).Seconds())
}()
keyBs := []byte(key)
val, err := s.db.Get(keyBs, nil)
if err == leveldb.ErrNotFound {
rec, ok := s.m.Load(*key)
if !ok {
databaseOperations.WithLabelValues(dbOpGet, dbResNotFound).Inc()
return DatabaseRecord{}, nil
}
if err != nil {
databaseOperations.WithLabelValues(dbOpGet, dbResError).Inc()
return DatabaseRecord{}, err
return &discosrv.DatabaseRecord{}, nil
}
var rec DatabaseRecord
if err := rec.Unmarshal(val); err != nil {
databaseOperations.WithLabelValues(dbOpGet, dbResUnmarshalError).Inc()
return DatabaseRecord{}, nil
}
rec.Addresses = expire(rec.Addresses, s.clock.Now().UnixNano())
rec.Addresses = expire(rec.Addresses, s.clock.Now())
databaseOperations.WithLabelValues(dbOpGet, dbResSuccess).Inc()
return rec, nil
}
func (s *levelDBStore) Serve(ctx context.Context) error {
t := time.NewTimer(0)
defer t.Stop()
defer s.db.Close()
func (s *inMemoryStore) Serve(ctx context.Context) error {
if s.flushInterval <= 0 {
<-ctx.Done()
return nil
}
// Start the statistics serve routine. It will exit with us when
// statisticsTrigger is closed.
statisticsTrigger := make(chan struct{})
statisticsDone := make(chan struct{})
go s.statisticsServe(statisticsTrigger, statisticsDone)
t := time.NewTimer(s.flushInterval)
defer t.Stop()
loop:
for {
select {
case fn := <-s.inbox:
// Run function in serialized order.
fn()
case <-t.C:
// Trigger the statistics routine to do its thing in the
// background.
statisticsTrigger <- struct{}{}
case <-statisticsDone:
// The statistics routine is done with one iteratation, schedule
// the next.
t.Reset(databaseStatisticsInterval)
log.Println("Calculating statistics")
s.expireAndCalculateStatistics()
log.Println("Flushing database")
if err := s.write(); err != nil {
log.Println("Error writing database:", err)
}
log.Println("Finished flushing database")
t.Reset(s.flushInterval)
case <-ctx.Done():
// We're done.
close(statisticsTrigger)
break loop
}
}
// Also wait for statisticsServe to return
<-statisticsDone
return s.write()
}
func (s *inMemoryStore) expireAndCalculateStatistics() {
now := s.clock.Now()
cutoff24h := now.Add(-24 * time.Hour).UnixNano()
cutoff1w := now.Add(-7 * 24 * time.Hour).UnixNano()
current, currentIPv4, currentIPv6, currentIPv6GUA, last24h, last1w := 0, 0, 0, 0, 0, 0
n := 0
s.m.Range(func(key protocol.DeviceID, rec *discosrv.DatabaseRecord) bool {
if n%1000 == 0 {
runtime.Gosched()
}
n++
addresses := expire(rec.Addresses, now)
if len(addresses) == 0 {
rec.Addresses = nil
s.m.Store(key, rec)
} else if len(addresses) != len(rec.Addresses) {
rec.Addresses = addresses
s.m.Store(key, rec)
}
switch {
case len(rec.Addresses) > 0:
current++
seenIPv4, seenIPv6, seenIPv6GUA := false, false, false
for _, addr := range rec.Addresses {
// We do fast and loose matching on strings here instead of
// parsing the address and the IP and doing "proper" checks,
// to keep things fast and generate less garbage.
if strings.Contains(addr.Address, "[") {
seenIPv6 = true
if strings.Contains(addr.Address, "[2") {
seenIPv6GUA = true
}
} else {
seenIPv4 = true
}
if seenIPv4 && seenIPv6 && seenIPv6GUA {
break
}
}
if seenIPv4 {
currentIPv4++
}
if seenIPv6 {
currentIPv6++
}
if seenIPv6GUA {
currentIPv6GUA++
}
case rec.Seen > cutoff24h:
last24h++
case rec.Seen > cutoff1w:
last1w++
default:
// drop the record if it's older than a week
s.m.Delete(key)
}
return true
})
databaseKeys.WithLabelValues("current").Set(float64(current))
databaseKeys.WithLabelValues("currentIPv4").Set(float64(currentIPv4))
databaseKeys.WithLabelValues("currentIPv6").Set(float64(currentIPv6))
databaseKeys.WithLabelValues("currentIPv6GUA").Set(float64(currentIPv6GUA))
databaseKeys.WithLabelValues("last24h").Set(float64(last24h))
databaseKeys.WithLabelValues("last1w").Set(float64(last1w))
databaseStatisticsSeconds.Set(time.Since(now).Seconds())
}
func (s *inMemoryStore) write() (err error) {
t0 := time.Now()
defer func() {
if err == nil {
databaseWriteSeconds.Set(time.Since(t0).Seconds())
databaseLastWritten.Set(float64(t0.Unix()))
}
}()
dbf := path.Join(s.dir, "records.db")
fd, err := os.Create(dbf + ".tmp")
if err != nil {
return err
}
bw := bufio.NewWriter(fd)
var buf []byte
var rangeErr error
now := s.clock.Now()
cutoff1w := now.Add(-7 * 24 * time.Hour).UnixNano()
n := 0
s.m.Range(func(key protocol.DeviceID, value *discosrv.DatabaseRecord) bool {
if n%1000 == 0 {
runtime.Gosched()
}
n++
if value.Seen < cutoff1w {
// drop the record if it's older than a week
return true
}
rec := &discosrv.ReplicationRecord{
Key: key[:],
Addresses: value.Addresses,
Seen: value.Seen,
}
s := proto.Size(rec)
if s+4 > len(buf) {
buf = make([]byte, s+4)
}
n, err := protoutil.MarshalTo(buf[4:], rec)
if err != nil {
rangeErr = err
return false
}
binary.BigEndian.PutUint32(buf, uint32(n))
if _, err := bw.Write(buf[:n+4]); err != nil {
rangeErr = err
return false
}
return true
})
if rangeErr != nil {
_ = fd.Close()
return rangeErr
}
if err := bw.Flush(); err != nil {
_ = fd.Close
return err
}
if err := fd.Close(); err != nil {
return err
}
if err := os.Rename(dbf+".tmp", dbf); err != nil {
return err
}
// Upload to S3
if s.s3 != nil {
fd, err = os.Open(dbf)
if err != nil {
log.Printf("Error uploading database to S3: %v", err)
return nil
}
defer fd.Close()
if err := s.s3.Upload(fd, s.objKey); err != nil {
log.Printf("Error uploading database to S3: %v", err)
}
log.Println("Finished uploading database")
}
return nil
}
func (s *levelDBStore) statisticsServe(trigger <-chan struct{}, done chan<- struct{}) {
defer close(done)
func (s *inMemoryStore) read() (int, error) {
fd, err := os.Open(path.Join(s.dir, "records.db"))
if err != nil {
return 0, err
}
defer fd.Close()
for range trigger {
t0 := time.Now()
nowNanos := t0.UnixNano()
cutoff24h := t0.Add(-24 * time.Hour).UnixNano()
cutoff1w := t0.Add(-7 * 24 * time.Hour).UnixNano()
cutoff2Mon := t0.Add(-60 * 24 * time.Hour).UnixNano()
current, last24h, last1w, inactive, errors := 0, 0, 0, 0, 0
iter := s.db.NewIterator(&util.Range{}, nil)
for iter.Next() {
// Attempt to unmarshal the record and count the
// failure if there's something wrong with it.
var rec DatabaseRecord
if err := rec.Unmarshal(iter.Value()); err != nil {
errors++
continue
}
// If there are addresses that have not expired it's a current
// record, otherwise account it based on when it was last seen
// (last 24 hours or last week) or finally as inactice.
switch {
case len(expire(rec.Addresses, nowNanos)) > 0:
current++
case rec.Seen > cutoff24h:
last24h++
case rec.Seen > cutoff1w:
last1w++
case rec.Seen > cutoff2Mon:
inactive++
case rec.Missed < cutoff2Mon:
// It hasn't been seen lately and we haven't recorded
// someone asking for this device in a long time either;
// delete the record.
if err := s.db.Delete(iter.Key(), nil); err != nil {
databaseOperations.WithLabelValues(dbOpDelete, dbResError).Inc()
} else {
databaseOperations.WithLabelValues(dbOpDelete, dbResSuccess).Inc()
}
default:
inactive++
br := bufio.NewReader(fd)
var buf []byte
nr := 0
for {
var n uint32
if err := binary.Read(br, binary.BigEndian, &n); err != nil {
if errors.Is(err, io.EOF) {
break
}
return nr, err
}
if int(n) > len(buf) {
buf = make([]byte, n)
}
if _, err := io.ReadFull(br, buf[:n]); err != nil {
return nr, err
}
rec := &discosrv.ReplicationRecord{}
if err := proto.Unmarshal(buf[:n], rec); err != nil {
return nr, err
}
key, err := protocol.DeviceIDFromBytes(rec.Key)
if err != nil {
key, err = protocol.DeviceIDFromString(string(rec.Key))
}
if err != nil {
log.Println("Bad device ID:", err)
continue
}
iter.Release()
databaseKeys.WithLabelValues("current").Set(float64(current))
databaseKeys.WithLabelValues("last24h").Set(float64(last24h))
databaseKeys.WithLabelValues("last1w").Set(float64(last1w))
databaseKeys.WithLabelValues("inactive").Set(float64(inactive))
databaseKeys.WithLabelValues("error").Set(float64(errors))
databaseStatisticsSeconds.Set(time.Since(t0).Seconds())
// Signal that we are done and can be scheduled again.
done <- struct{}{}
slices.SortFunc(rec.Addresses, Cmp)
rec.Addresses = slices.CompactFunc(rec.Addresses, Equal)
s.m.Store(key, &discosrv.DatabaseRecord{
Addresses: expire(rec.Addresses, s.clock.Now()),
Seen: rec.Seen,
})
nr++
}
return nr, nil
}
// merge returns the merged result of the two database records a and b. The
// result is the union of the two address sets, with the newer expiry time
// chosen for any duplicates.
func merge(a, b DatabaseRecord) DatabaseRecord {
// chosen for any duplicates. The address list in a is overwritten and
// reused for the result.
func merge(a, b *discosrv.DatabaseRecord) *discosrv.DatabaseRecord {
// Both lists must be sorted for this to work.
if !sort.IsSorted(databaseAddressOrder(a.Addresses)) {
log.Println("Warning: bug: addresses not correctly sorted in merge")
a.Addresses = sortedAddressCopy(a.Addresses)
}
if !sort.IsSorted(databaseAddressOrder(b.Addresses)) {
// no warning because this is the side we read from disk and it may
// legitimately predate correct sorting.
b.Addresses = sortedAddressCopy(b.Addresses)
}
res := DatabaseRecord{
Addresses: make([]DatabaseAddress, 0, len(a.Addresses)+len(b.Addresses)),
Seen: a.Seen,
}
if b.Seen > a.Seen {
res.Seen = b.Seen
}
a.Seen = max(a.Seen, b.Seen)
aIdx := 0
bIdx := 0
aAddrs := a.Addresses
bAddrs := b.Addresses
loop:
for {
switch {
case aIdx == len(aAddrs) && bIdx == len(bAddrs):
// both lists are exhausted, we are done
break loop
case aIdx == len(aAddrs):
// a is exhausted, pick from b and continue
res.Addresses = append(res.Addresses, bAddrs[bIdx])
bIdx++
continue
case bIdx == len(bAddrs):
// b is exhausted, pick from a and continue
res.Addresses = append(res.Addresses, aAddrs[aIdx])
aIdx++
continue
}
// We have values left on both sides.
aVal := aAddrs[aIdx]
bVal := bAddrs[bIdx]
switch {
case aVal.Address == bVal.Address:
// update for same address, pick newer
if aVal.Expires > bVal.Expires {
res.Addresses = append(res.Addresses, aVal)
} else {
res.Addresses = append(res.Addresses, bVal)
}
for aIdx < len(a.Addresses) && bIdx < len(b.Addresses) {
switch cmp.Compare(a.Addresses[aIdx].Address, b.Addresses[bIdx].Address) {
case 0:
// a == b, choose the newer expiry time
a.Addresses[aIdx].Expires = max(a.Addresses[aIdx].Expires, b.Addresses[bIdx].Expires)
aIdx++
bIdx++
case aVal.Address < bVal.Address:
// a is smallest, pick it and continue
res.Addresses = append(res.Addresses, aVal)
case -1:
// a < b, keep a and move on
aIdx++
default:
// b is smallest, pick it and continue
res.Addresses = append(res.Addresses, bVal)
case 1:
// a > b, insert b before a
a.Addresses = append(a.Addresses[:aIdx], append([]*discosrv.DatabaseAddress{b.Addresses[bIdx]}, a.Addresses[aIdx:]...)...)
bIdx++
}
}
return res
if bIdx < len(b.Addresses) {
a.Addresses = append(a.Addresses, b.Addresses[bIdx:]...)
}
return a
}
// expire returns the list of addresses after removing expired entries.
// Expiration happen in place, so the slice given as the parameter is
// destroyed. Internal order is not preserved.
func expire(addrs []DatabaseAddress, now int64) []DatabaseAddress {
i := 0
for i < len(addrs) {
if addrs[i].Expires < now {
// This item is expired. Replace it with the last in the list
// (noop if we are at the last item).
addrs[i] = addrs[len(addrs)-1]
// Wipe the last item of the list to release references to
// strings and stuff.
addrs[len(addrs)-1] = DatabaseAddress{}
// Shorten the slice.
addrs = addrs[:len(addrs)-1]
// destroyed. Internal order is preserved.
func expire(addrs []*discosrv.DatabaseAddress, now time.Time) []*discosrv.DatabaseAddress {
cutoff := now.UnixNano()
naddrs := addrs[:0]
for i := range addrs {
if i > 0 && addrs[i].Address == addrs[i-1].Address {
// Skip duplicates
continue
}
i++
if addrs[i].Expires >= cutoff {
naddrs = append(naddrs, addrs[i])
}
}
return addrs
if len(naddrs) == 0 {
return nil
}
return naddrs
}
func sortedAddressCopy(addrs []DatabaseAddress) []DatabaseAddress {
sorted := make([]DatabaseAddress, len(addrs))
copy(sorted, addrs)
sort.Sort(databaseAddressOrder(sorted))
return sorted
func Cmp(d, other *discosrv.DatabaseAddress) (n int) {
if c := cmp.Compare(d.Address, other.Address); c != 0 {
return c
}
return cmp.Compare(d.Expires, other.Expires)
}
type databaseAddressOrder []DatabaseAddress
func (s databaseAddressOrder) Less(a, b int) bool {
return s[a].Address < s[b].Address
}
func (s databaseAddressOrder) Swap(a, b int) {
s[a], s[b] = s[b], s[a]
}
func (s databaseAddressOrder) Len() int {
return len(s)
func Equal(d, other *discosrv.DatabaseAddress) bool {
return d.Address == other.Address
}

View File

@@ -1,847 +0,0 @@
// Code generated by protoc-gen-gogo. DO NOT EDIT.
// source: database.proto
package main
import (
fmt "fmt"
_ "github.com/gogo/protobuf/gogoproto"
proto "github.com/gogo/protobuf/proto"
io "io"
math "math"
math_bits "math/bits"
)
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = fmt.Errorf
var _ = math.Inf
// This is a compile-time assertion to ensure that this generated file
// is compatible with the proto package it is being compiled against.
// A compilation error at this line likely means your copy of the
// proto package needs to be updated.
const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package
type DatabaseRecord struct {
Addresses []DatabaseAddress `protobuf:"bytes,1,rep,name=addresses,proto3" json:"addresses"`
Misses int32 `protobuf:"varint,2,opt,name=misses,proto3" json:"misses,omitempty"`
Seen int64 `protobuf:"varint,3,opt,name=seen,proto3" json:"seen,omitempty"`
Missed int64 `protobuf:"varint,4,opt,name=missed,proto3" json:"missed,omitempty"`
}
func (m *DatabaseRecord) Reset() { *m = DatabaseRecord{} }
func (m *DatabaseRecord) String() string { return proto.CompactTextString(m) }
func (*DatabaseRecord) ProtoMessage() {}
func (*DatabaseRecord) Descriptor() ([]byte, []int) {
return fileDescriptor_b90fe3356ea5df07, []int{0}
}
func (m *DatabaseRecord) XXX_Unmarshal(b []byte) error {
return m.Unmarshal(b)
}
func (m *DatabaseRecord) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
if deterministic {
return xxx_messageInfo_DatabaseRecord.Marshal(b, m, deterministic)
} else {
b = b[:cap(b)]
n, err := m.MarshalToSizedBuffer(b)
if err != nil {
return nil, err
}
return b[:n], nil
}
}
func (m *DatabaseRecord) XXX_Merge(src proto.Message) {
xxx_messageInfo_DatabaseRecord.Merge(m, src)
}
func (m *DatabaseRecord) XXX_Size() int {
return m.Size()
}
func (m *DatabaseRecord) XXX_DiscardUnknown() {
xxx_messageInfo_DatabaseRecord.DiscardUnknown(m)
}
var xxx_messageInfo_DatabaseRecord proto.InternalMessageInfo
type ReplicationRecord struct {
Key string `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
Addresses []DatabaseAddress `protobuf:"bytes,2,rep,name=addresses,proto3" json:"addresses"`
Seen int64 `protobuf:"varint,3,opt,name=seen,proto3" json:"seen,omitempty"`
}
func (m *ReplicationRecord) Reset() { *m = ReplicationRecord{} }
func (m *ReplicationRecord) String() string { return proto.CompactTextString(m) }
func (*ReplicationRecord) ProtoMessage() {}
func (*ReplicationRecord) Descriptor() ([]byte, []int) {
return fileDescriptor_b90fe3356ea5df07, []int{1}
}
func (m *ReplicationRecord) XXX_Unmarshal(b []byte) error {
return m.Unmarshal(b)
}
func (m *ReplicationRecord) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
if deterministic {
return xxx_messageInfo_ReplicationRecord.Marshal(b, m, deterministic)
} else {
b = b[:cap(b)]
n, err := m.MarshalToSizedBuffer(b)
if err != nil {
return nil, err
}
return b[:n], nil
}
}
func (m *ReplicationRecord) XXX_Merge(src proto.Message) {
xxx_messageInfo_ReplicationRecord.Merge(m, src)
}
func (m *ReplicationRecord) XXX_Size() int {
return m.Size()
}
func (m *ReplicationRecord) XXX_DiscardUnknown() {
xxx_messageInfo_ReplicationRecord.DiscardUnknown(m)
}
var xxx_messageInfo_ReplicationRecord proto.InternalMessageInfo
type DatabaseAddress struct {
Address string `protobuf:"bytes,1,opt,name=address,proto3" json:"address,omitempty"`
Expires int64 `protobuf:"varint,2,opt,name=expires,proto3" json:"expires,omitempty"`
}
func (m *DatabaseAddress) Reset() { *m = DatabaseAddress{} }
func (m *DatabaseAddress) String() string { return proto.CompactTextString(m) }
func (*DatabaseAddress) ProtoMessage() {}
func (*DatabaseAddress) Descriptor() ([]byte, []int) {
return fileDescriptor_b90fe3356ea5df07, []int{2}
}
func (m *DatabaseAddress) XXX_Unmarshal(b []byte) error {
return m.Unmarshal(b)
}
func (m *DatabaseAddress) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
if deterministic {
return xxx_messageInfo_DatabaseAddress.Marshal(b, m, deterministic)
} else {
b = b[:cap(b)]
n, err := m.MarshalToSizedBuffer(b)
if err != nil {
return nil, err
}
return b[:n], nil
}
}
func (m *DatabaseAddress) XXX_Merge(src proto.Message) {
xxx_messageInfo_DatabaseAddress.Merge(m, src)
}
func (m *DatabaseAddress) XXX_Size() int {
return m.Size()
}
func (m *DatabaseAddress) XXX_DiscardUnknown() {
xxx_messageInfo_DatabaseAddress.DiscardUnknown(m)
}
var xxx_messageInfo_DatabaseAddress proto.InternalMessageInfo
func init() {
proto.RegisterType((*DatabaseRecord)(nil), "main.DatabaseRecord")
proto.RegisterType((*ReplicationRecord)(nil), "main.ReplicationRecord")
proto.RegisterType((*DatabaseAddress)(nil), "main.DatabaseAddress")
}
func init() { proto.RegisterFile("database.proto", fileDescriptor_b90fe3356ea5df07) }
var fileDescriptor_b90fe3356ea5df07 = []byte{
// 270 bytes of a gzipped FileDescriptorProto
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x94, 0x90, 0x41, 0x4a, 0xc4, 0x30,
0x18, 0x85, 0x9b, 0x49, 0x1d, 0x99, 0x08, 0xa3, 0x06, 0x94, 0x20, 0x12, 0x4b, 0xdd, 0x74, 0xd5,
0x01, 0x5d, 0xb9, 0x74, 0xd0, 0x0b, 0xe4, 0x06, 0xe9, 0xe4, 0x77, 0x08, 0x3a, 0x4d, 0x49, 0x2a,
0xe8, 0x29, 0xf4, 0x58, 0x5d, 0xce, 0xd2, 0x95, 0x68, 0x7b, 0x11, 0x69, 0x26, 0x55, 0x14, 0x37,
0xb3, 0x7b, 0xdf, 0xff, 0xbf, 0x97, 0xbc, 0x84, 0x4c, 0x95, 0xac, 0x65, 0x21, 0x1d, 0xe4, 0x95,
0x35, 0xb5, 0xa1, 0xf1, 0x4a, 0xea, 0xf2, 0xe4, 0xdc, 0x42, 0x65, 0xdc, 0xcc, 0x8f, 0x8a, 0xc7,
0xbb, 0xd9, 0xd2, 0x2c, 0x8d, 0x07, 0xaf, 0x36, 0xd6, 0xf4, 0x05, 0x91, 0xe9, 0x4d, 0x48, 0x0b,
0x58, 0x18, 0xab, 0xe8, 0x15, 0x99, 0x48, 0xa5, 0x2c, 0x38, 0x07, 0x8e, 0xa1, 0x04, 0x67, 0x7b,
0x17, 0x47, 0x79, 0x7f, 0x62, 0x3e, 0x18, 0xaf, 0x37, 0xeb, 0x79, 0xdc, 0xbc, 0x9f, 0x45, 0xe2,
0xc7, 0x4d, 0x8f, 0xc9, 0x78, 0xa5, 0x7d, 0x6e, 0x94, 0xa0, 0x6c, 0x47, 0x04, 0xa2, 0x94, 0xc4,
0x0e, 0xa0, 0x64, 0x38, 0x41, 0x19, 0x16, 0x5e, 0x7f, 0x7b, 0x15, 0x8b, 0xfd, 0x34, 0x50, 0x5a,
0x93, 0x43, 0x01, 0xd5, 0x83, 0x5e, 0xc8, 0x5a, 0x9b, 0x32, 0x74, 0x3a, 0x20, 0xf8, 0x1e, 0x9e,
0x19, 0x4a, 0x50, 0x36, 0x11, 0xbd, 0xfc, 0xdd, 0x72, 0xb4, 0x55, 0xcb, 0x7f, 0xda, 0xa4, 0xb7,
0x64, 0xff, 0x4f, 0x8e, 0x32, 0xb2, 0x1b, 0x32, 0xe1, 0xde, 0x01, 0xfb, 0x0d, 0x3c, 0x55, 0xda,
0x86, 0x77, 0x62, 0x31, 0xe0, 0xfc, 0xb4, 0xf9, 0xe4, 0x51, 0xd3, 0x72, 0xb4, 0x6e, 0x39, 0xfa,
0x68, 0x39, 0x7a, 0xed, 0x78, 0xb4, 0xee, 0x78, 0xf4, 0xd6, 0xf1, 0xa8, 0x18, 0xfb, 0x3f, 0xbf,
0xfc, 0x0a, 0x00, 0x00, 0xff, 0xff, 0x7a, 0xa2, 0xf6, 0x1e, 0xb0, 0x01, 0x00, 0x00,
}
func (m *DatabaseRecord) Marshal() (dAtA []byte, err error) {
size := m.Size()
dAtA = make([]byte, size)
n, err := m.MarshalToSizedBuffer(dAtA[:size])
if err != nil {
return nil, err
}
return dAtA[:n], nil
}
func (m *DatabaseRecord) MarshalTo(dAtA []byte) (int, error) {
size := m.Size()
return m.MarshalToSizedBuffer(dAtA[:size])
}
func (m *DatabaseRecord) MarshalToSizedBuffer(dAtA []byte) (int, error) {
i := len(dAtA)
_ = i
var l int
_ = l
if m.Missed != 0 {
i = encodeVarintDatabase(dAtA, i, uint64(m.Missed))
i--
dAtA[i] = 0x20
}
if m.Seen != 0 {
i = encodeVarintDatabase(dAtA, i, uint64(m.Seen))
i--
dAtA[i] = 0x18
}
if m.Misses != 0 {
i = encodeVarintDatabase(dAtA, i, uint64(m.Misses))
i--
dAtA[i] = 0x10
}
if len(m.Addresses) > 0 {
for iNdEx := len(m.Addresses) - 1; iNdEx >= 0; iNdEx-- {
{
size, err := m.Addresses[iNdEx].MarshalToSizedBuffer(dAtA[:i])
if err != nil {
return 0, err
}
i -= size
i = encodeVarintDatabase(dAtA, i, uint64(size))
}
i--
dAtA[i] = 0xa
}
}
return len(dAtA) - i, nil
}
func (m *ReplicationRecord) Marshal() (dAtA []byte, err error) {
size := m.Size()
dAtA = make([]byte, size)
n, err := m.MarshalToSizedBuffer(dAtA[:size])
if err != nil {
return nil, err
}
return dAtA[:n], nil
}
func (m *ReplicationRecord) MarshalTo(dAtA []byte) (int, error) {
size := m.Size()
return m.MarshalToSizedBuffer(dAtA[:size])
}
func (m *ReplicationRecord) MarshalToSizedBuffer(dAtA []byte) (int, error) {
i := len(dAtA)
_ = i
var l int
_ = l
if m.Seen != 0 {
i = encodeVarintDatabase(dAtA, i, uint64(m.Seen))
i--
dAtA[i] = 0x18
}
if len(m.Addresses) > 0 {
for iNdEx := len(m.Addresses) - 1; iNdEx >= 0; iNdEx-- {
{
size, err := m.Addresses[iNdEx].MarshalToSizedBuffer(dAtA[:i])
if err != nil {
return 0, err
}
i -= size
i = encodeVarintDatabase(dAtA, i, uint64(size))
}
i--
dAtA[i] = 0x12
}
}
if len(m.Key) > 0 {
i -= len(m.Key)
copy(dAtA[i:], m.Key)
i = encodeVarintDatabase(dAtA, i, uint64(len(m.Key)))
i--
dAtA[i] = 0xa
}
return len(dAtA) - i, nil
}
func (m *DatabaseAddress) Marshal() (dAtA []byte, err error) {
size := m.Size()
dAtA = make([]byte, size)
n, err := m.MarshalToSizedBuffer(dAtA[:size])
if err != nil {
return nil, err
}
return dAtA[:n], nil
}
func (m *DatabaseAddress) MarshalTo(dAtA []byte) (int, error) {
size := m.Size()
return m.MarshalToSizedBuffer(dAtA[:size])
}
func (m *DatabaseAddress) MarshalToSizedBuffer(dAtA []byte) (int, error) {
i := len(dAtA)
_ = i
var l int
_ = l
if m.Expires != 0 {
i = encodeVarintDatabase(dAtA, i, uint64(m.Expires))
i--
dAtA[i] = 0x10
}
if len(m.Address) > 0 {
i -= len(m.Address)
copy(dAtA[i:], m.Address)
i = encodeVarintDatabase(dAtA, i, uint64(len(m.Address)))
i--
dAtA[i] = 0xa
}
return len(dAtA) - i, nil
}
func encodeVarintDatabase(dAtA []byte, offset int, v uint64) int {
offset -= sovDatabase(v)
base := offset
for v >= 1<<7 {
dAtA[offset] = uint8(v&0x7f | 0x80)
v >>= 7
offset++
}
dAtA[offset] = uint8(v)
return base
}
func (m *DatabaseRecord) Size() (n int) {
if m == nil {
return 0
}
var l int
_ = l
if len(m.Addresses) > 0 {
for _, e := range m.Addresses {
l = e.Size()
n += 1 + l + sovDatabase(uint64(l))
}
}
if m.Misses != 0 {
n += 1 + sovDatabase(uint64(m.Misses))
}
if m.Seen != 0 {
n += 1 + sovDatabase(uint64(m.Seen))
}
if m.Missed != 0 {
n += 1 + sovDatabase(uint64(m.Missed))
}
return n
}
func (m *ReplicationRecord) Size() (n int) {
if m == nil {
return 0
}
var l int
_ = l
l = len(m.Key)
if l > 0 {
n += 1 + l + sovDatabase(uint64(l))
}
if len(m.Addresses) > 0 {
for _, e := range m.Addresses {
l = e.Size()
n += 1 + l + sovDatabase(uint64(l))
}
}
if m.Seen != 0 {
n += 1 + sovDatabase(uint64(m.Seen))
}
return n
}
func (m *DatabaseAddress) Size() (n int) {
if m == nil {
return 0
}
var l int
_ = l
l = len(m.Address)
if l > 0 {
n += 1 + l + sovDatabase(uint64(l))
}
if m.Expires != 0 {
n += 1 + sovDatabase(uint64(m.Expires))
}
return n
}
func sovDatabase(x uint64) (n int) {
return (math_bits.Len64(x|1) + 6) / 7
}
func sozDatabase(x uint64) (n int) {
return sovDatabase(uint64((x << 1) ^ uint64((int64(x) >> 63))))
}
func (m *DatabaseRecord) Unmarshal(dAtA []byte) error {
l := len(dAtA)
iNdEx := 0
for iNdEx < l {
preIndex := iNdEx
var wire uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
wire |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
fieldNum := int32(wire >> 3)
wireType := int(wire & 0x7)
if wireType == 4 {
return fmt.Errorf("proto: DatabaseRecord: wiretype end group for non-group")
}
if fieldNum <= 0 {
return fmt.Errorf("proto: DatabaseRecord: illegal tag %d (wire type %d)", fieldNum, wire)
}
switch fieldNum {
case 1:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field Addresses", wireType)
}
var msglen int
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
msglen |= int(b&0x7F) << shift
if b < 0x80 {
break
}
}
if msglen < 0 {
return ErrInvalidLengthDatabase
}
postIndex := iNdEx + msglen
if postIndex < 0 {
return ErrInvalidLengthDatabase
}
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.Addresses = append(m.Addresses, DatabaseAddress{})
if err := m.Addresses[len(m.Addresses)-1].Unmarshal(dAtA[iNdEx:postIndex]); err != nil {
return err
}
iNdEx = postIndex
case 2:
if wireType != 0 {
return fmt.Errorf("proto: wrong wireType = %d for field Misses", wireType)
}
m.Misses = 0
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
m.Misses |= int32(b&0x7F) << shift
if b < 0x80 {
break
}
}
case 3:
if wireType != 0 {
return fmt.Errorf("proto: wrong wireType = %d for field Seen", wireType)
}
m.Seen = 0
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
m.Seen |= int64(b&0x7F) << shift
if b < 0x80 {
break
}
}
case 4:
if wireType != 0 {
return fmt.Errorf("proto: wrong wireType = %d for field Missed", wireType)
}
m.Missed = 0
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
m.Missed |= int64(b&0x7F) << shift
if b < 0x80 {
break
}
}
default:
iNdEx = preIndex
skippy, err := skipDatabase(dAtA[iNdEx:])
if err != nil {
return err
}
if (skippy < 0) || (iNdEx+skippy) < 0 {
return ErrInvalidLengthDatabase
}
if (iNdEx + skippy) > l {
return io.ErrUnexpectedEOF
}
iNdEx += skippy
}
}
if iNdEx > l {
return io.ErrUnexpectedEOF
}
return nil
}
func (m *ReplicationRecord) Unmarshal(dAtA []byte) error {
l := len(dAtA)
iNdEx := 0
for iNdEx < l {
preIndex := iNdEx
var wire uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
wire |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
fieldNum := int32(wire >> 3)
wireType := int(wire & 0x7)
if wireType == 4 {
return fmt.Errorf("proto: ReplicationRecord: wiretype end group for non-group")
}
if fieldNum <= 0 {
return fmt.Errorf("proto: ReplicationRecord: illegal tag %d (wire type %d)", fieldNum, wire)
}
switch fieldNum {
case 1:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field Key", wireType)
}
var stringLen uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
stringLen |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
intStringLen := int(stringLen)
if intStringLen < 0 {
return ErrInvalidLengthDatabase
}
postIndex := iNdEx + intStringLen
if postIndex < 0 {
return ErrInvalidLengthDatabase
}
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.Key = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
case 2:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field Addresses", wireType)
}
var msglen int
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
msglen |= int(b&0x7F) << shift
if b < 0x80 {
break
}
}
if msglen < 0 {
return ErrInvalidLengthDatabase
}
postIndex := iNdEx + msglen
if postIndex < 0 {
return ErrInvalidLengthDatabase
}
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.Addresses = append(m.Addresses, DatabaseAddress{})
if err := m.Addresses[len(m.Addresses)-1].Unmarshal(dAtA[iNdEx:postIndex]); err != nil {
return err
}
iNdEx = postIndex
case 3:
if wireType != 0 {
return fmt.Errorf("proto: wrong wireType = %d for field Seen", wireType)
}
m.Seen = 0
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
m.Seen |= int64(b&0x7F) << shift
if b < 0x80 {
break
}
}
default:
iNdEx = preIndex
skippy, err := skipDatabase(dAtA[iNdEx:])
if err != nil {
return err
}
if (skippy < 0) || (iNdEx+skippy) < 0 {
return ErrInvalidLengthDatabase
}
if (iNdEx + skippy) > l {
return io.ErrUnexpectedEOF
}
iNdEx += skippy
}
}
if iNdEx > l {
return io.ErrUnexpectedEOF
}
return nil
}
func (m *DatabaseAddress) Unmarshal(dAtA []byte) error {
l := len(dAtA)
iNdEx := 0
for iNdEx < l {
preIndex := iNdEx
var wire uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
wire |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
fieldNum := int32(wire >> 3)
wireType := int(wire & 0x7)
if wireType == 4 {
return fmt.Errorf("proto: DatabaseAddress: wiretype end group for non-group")
}
if fieldNum <= 0 {
return fmt.Errorf("proto: DatabaseAddress: illegal tag %d (wire type %d)", fieldNum, wire)
}
switch fieldNum {
case 1:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field Address", wireType)
}
var stringLen uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
stringLen |= uint64(b&0x7F) << shift
if b < 0x80 {
break
}
}
intStringLen := int(stringLen)
if intStringLen < 0 {
return ErrInvalidLengthDatabase
}
postIndex := iNdEx + intStringLen
if postIndex < 0 {
return ErrInvalidLengthDatabase
}
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.Address = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
case 2:
if wireType != 0 {
return fmt.Errorf("proto: wrong wireType = %d for field Expires", wireType)
}
m.Expires = 0
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowDatabase
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
m.Expires |= int64(b&0x7F) << shift
if b < 0x80 {
break
}
}
default:
iNdEx = preIndex
skippy, err := skipDatabase(dAtA[iNdEx:])
if err != nil {
return err
}
if (skippy < 0) || (iNdEx+skippy) < 0 {
return ErrInvalidLengthDatabase
}
if (iNdEx + skippy) > l {
return io.ErrUnexpectedEOF
}
iNdEx += skippy
}
}
if iNdEx > l {
return io.ErrUnexpectedEOF
}
return nil
}
func skipDatabase(dAtA []byte) (n int, err error) {
l := len(dAtA)
iNdEx := 0
depth := 0
for iNdEx < l {
var wire uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return 0, ErrIntOverflowDatabase
}
if iNdEx >= l {
return 0, io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
wire |= (uint64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
wireType := int(wire & 0x7)
switch wireType {
case 0:
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return 0, ErrIntOverflowDatabase
}
if iNdEx >= l {
return 0, io.ErrUnexpectedEOF
}
iNdEx++
if dAtA[iNdEx-1] < 0x80 {
break
}
}
case 1:
iNdEx += 8
case 2:
var length int
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return 0, ErrIntOverflowDatabase
}
if iNdEx >= l {
return 0, io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
length |= (int(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
if length < 0 {
return 0, ErrInvalidLengthDatabase
}
iNdEx += length
case 3:
depth++
case 4:
if depth == 0 {
return 0, ErrUnexpectedEndOfGroupDatabase
}
depth--
case 5:
iNdEx += 4
default:
return 0, fmt.Errorf("proto: illegal wireType %d", wireType)
}
if iNdEx < 0 {
return 0, ErrInvalidLengthDatabase
}
if depth == 0 {
return iNdEx, nil
}
}
return 0, io.ErrUnexpectedEOF
}
var (
ErrInvalidLengthDatabase = fmt.Errorf("proto: negative length found during unmarshaling")
ErrIntOverflowDatabase = fmt.Errorf("proto: integer overflow")
ErrUnexpectedEndOfGroupDatabase = fmt.Errorf("proto: unexpected end of group")
)

View File

@@ -1,36 +0,0 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
syntax = "proto3";
package main;
import "repos/protobuf/gogoproto/gogo.proto";
option (gogoproto.goproto_getters_all) = false;
option (gogoproto.goproto_unkeyed_all) = false;
option (gogoproto.goproto_unrecognized_all) = false;
option (gogoproto.goproto_sizecache_all) = false;
message DatabaseRecord {
repeated DatabaseAddress addresses = 1 [(gogoproto.nullable) = false];
int32 misses = 2; // Number of lookups* without hits
int64 seen = 3; // Unix nanos, last device announce
int64 missed = 4; // Unix nanos, last* failed lookup
}
// *) Not every lookup results in a write, so may not be completely accurate
message ReplicationRecord {
string key = 1;
repeated DatabaseAddress addresses = 2 [(gogoproto.nullable) = false];
int64 seen = 3; // Unix nanos, last device announce
}
message DatabaseAddress {
string address = 1;
int64 expires = 2; // Unix nanos
}

View File

@@ -9,34 +9,28 @@ package main
import (
"context"
"fmt"
"os"
"testing"
"time"
"github.com/syncthing/syncthing/internal/gen/discosrv"
"github.com/syncthing/syncthing/lib/protocol"
)
func TestDatabaseGetSet(t *testing.T) {
os.RemoveAll("_database")
defer os.RemoveAll("_database")
db, err := newLevelDBStore("_database")
if err != nil {
t.Fatal(err)
}
db := newInMemoryStore(t.TempDir(), 0, nil)
ctx, cancel := context.WithCancel(context.Background())
go db.Serve(ctx)
defer cancel()
// Check missing record
rec, err := db.get("abcd")
rec, err := db.get(&protocol.EmptyDeviceID)
if err != nil {
t.Error("not found should not be an error")
}
if len(rec.Addresses) != 0 {
t.Error("addresses should be empty")
}
if rec.Misses != 0 {
t.Error("missing should be zero")
}
// Set up a clock
@@ -46,16 +40,16 @@ func TestDatabaseGetSet(t *testing.T) {
// Put a record
rec.Addresses = []DatabaseAddress{
rec.Addresses = []*discosrv.DatabaseAddress{
{Address: "tcp://1.2.3.4:5", Expires: tc.Now().Add(time.Minute).UnixNano()},
}
if err := db.put("abcd", rec); err != nil {
if err := db.put(&protocol.EmptyDeviceID, rec); err != nil {
t.Fatal(err)
}
// Verify it
rec, err = db.get("abcd")
rec, err = db.get(&protocol.EmptyDeviceID)
if err != nil {
t.Fatal(err)
}
@@ -72,16 +66,16 @@ func TestDatabaseGetSet(t *testing.T) {
tc.wind(30 * time.Second)
addrs := []DatabaseAddress{
addrs := []*discosrv.DatabaseAddress{
{Address: "tcp://6.7.8.9:0", Expires: tc.Now().Add(time.Minute).UnixNano()},
}
if err := db.merge("abcd", addrs, tc.Now().UnixNano()); err != nil {
if err := db.merge(&protocol.EmptyDeviceID, addrs, tc.Now().UnixNano()); err != nil {
t.Fatal(err)
}
// Verify it
rec, err = db.get("abcd")
rec, err = db.get(&protocol.EmptyDeviceID)
if err != nil {
t.Fatal(err)
}
@@ -104,7 +98,7 @@ func TestDatabaseGetSet(t *testing.T) {
// Verify it
rec, err = db.get("abcd")
rec, err = db.get(&protocol.EmptyDeviceID)
if err != nil {
t.Fatal(err)
}
@@ -117,40 +111,18 @@ func TestDatabaseGetSet(t *testing.T) {
t.Error("incorrect address")
}
// Put a record with misses
rec = DatabaseRecord{Misses: 42}
if err := db.put("efgh", rec); err != nil {
t.Fatal(err)
}
// Verify it
rec, err = db.get("efgh")
if err != nil {
t.Fatal(err)
}
if len(rec.Addresses) != 0 {
t.Log(rec.Addresses)
t.Fatal("should have no addresses")
}
if rec.Misses != 42 {
t.Log(rec.Misses)
t.Error("incorrect misses")
}
// Set an address
addrs = []DatabaseAddress{
addrs = []*discosrv.DatabaseAddress{
{Address: "tcp://6.7.8.9:0", Expires: tc.Now().Add(time.Minute).UnixNano()},
}
if err := db.merge("efgh", addrs, tc.Now().UnixNano()); err != nil {
if err := db.merge(&protocol.GlobalDeviceID, addrs, tc.Now().UnixNano()); err != nil {
t.Fatal(err)
}
// Verify it
rec, err = db.get("efgh")
rec, err = db.get(&protocol.GlobalDeviceID)
if err != nil {
t.Fatal(err)
}
@@ -158,48 +130,126 @@ func TestDatabaseGetSet(t *testing.T) {
t.Log(rec.Addresses)
t.Fatal("should have one address")
}
if rec.Misses != 0 {
t.Log(rec.Misses)
t.Error("should have no misses")
}
}
func TestFilter(t *testing.T) {
// all cases are expired with t=10
cases := []struct {
a []DatabaseAddress
b []DatabaseAddress
a []*discosrv.DatabaseAddress
b []*discosrv.DatabaseAddress
}{
{
a: nil,
b: nil,
},
{
a: []DatabaseAddress{{Address: "a", Expires: 9}, {Address: "b", Expires: 9}, {Address: "c", Expires: 9}},
b: []DatabaseAddress{},
a: []*discosrv.DatabaseAddress{{Address: "a", Expires: 9}, {Address: "b", Expires: 9}, {Address: "c", Expires: 9}},
b: []*discosrv.DatabaseAddress{},
},
{
a: []DatabaseAddress{{Address: "a", Expires: 10}},
b: []DatabaseAddress{{Address: "a", Expires: 10}},
a: []*discosrv.DatabaseAddress{{Address: "a", Expires: 10}},
b: []*discosrv.DatabaseAddress{{Address: "a", Expires: 10}},
},
{
a: []DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 10}, {Address: "c", Expires: 10}},
b: []DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 10}, {Address: "c", Expires: 10}},
a: []*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 10}, {Address: "c", Expires: 10}},
b: []*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 10}, {Address: "c", Expires: 10}},
},
{
a: []DatabaseAddress{{Address: "a", Expires: 5}, {Address: "b", Expires: 15}, {Address: "c", Expires: 5}, {Address: "d", Expires: 15}, {Address: "e", Expires: 5}},
b: []DatabaseAddress{{Address: "d", Expires: 15}, {Address: "b", Expires: 15}}, // gets reordered
a: []*discosrv.DatabaseAddress{{Address: "a", Expires: 5}, {Address: "b", Expires: 15}, {Address: "c", Expires: 5}, {Address: "d", Expires: 15}, {Address: "e", Expires: 5}},
b: []*discosrv.DatabaseAddress{{Address: "b", Expires: 15}, {Address: "d", Expires: 15}},
},
}
for _, tc := range cases {
res := expire(tc.a, 10)
res := expire(tc.a, time.Unix(0, 10))
if fmt.Sprint(res) != fmt.Sprint(tc.b) {
t.Errorf("Incorrect result %v, expected %v", res, tc.b)
}
}
}
func TestMerge(t *testing.T) {
cases := []struct {
a, b, res []*discosrv.DatabaseAddress
}{
{nil, nil, nil},
{
nil,
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}},
},
{
nil,
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 10}, {Address: "c", Expires: 10}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 10}, {Address: "c", Expires: 10}},
},
{
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 15}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 15}},
},
{
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}},
[]*discosrv.DatabaseAddress{{Address: "b", Expires: 15}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 15}},
},
{
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 15}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 15}, {Address: "b", Expires: 15}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 15}, {Address: "b", Expires: 15}},
},
{
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 15}},
[]*discosrv.DatabaseAddress{{Address: "b", Expires: 15}, {Address: "c", Expires: 20}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 15}, {Address: "c", Expires: 20}},
},
{
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 15}},
[]*discosrv.DatabaseAddress{{Address: "b", Expires: 5}, {Address: "c", Expires: 20}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 15}, {Address: "c", Expires: 20}},
},
{
[]*discosrv.DatabaseAddress{{Address: "y", Expires: 10}, {Address: "z", Expires: 10}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 5}, {Address: "b", Expires: 15}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 5}, {Address: "b", Expires: 15}, {Address: "y", Expires: 10}, {Address: "z", Expires: 10}},
},
{
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 15}, {Address: "d", Expires: 10}},
[]*discosrv.DatabaseAddress{{Address: "b", Expires: 5}, {Address: "c", Expires: 20}},
[]*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 15}, {Address: "c", Expires: 20}, {Address: "d", Expires: 10}},
},
}
for _, tc := range cases {
rec := merge(&discosrv.DatabaseRecord{Addresses: tc.a}, &discosrv.DatabaseRecord{Addresses: tc.b})
if fmt.Sprint(rec.Addresses) != fmt.Sprint(tc.res) {
t.Errorf("Incorrect result %v, expected %v", rec.Addresses, tc.res)
}
rec = merge(&discosrv.DatabaseRecord{Addresses: tc.b}, &discosrv.DatabaseRecord{Addresses: tc.a})
if fmt.Sprint(rec.Addresses) != fmt.Sprint(tc.res) {
t.Errorf("Incorrect result %v, expected %v", rec.Addresses, tc.res)
}
}
}
func BenchmarkMergeEqual(b *testing.B) {
for i := 0; i < b.N; i++ {
ar := []*discosrv.DatabaseAddress{{Address: "a", Expires: 10}, {Address: "b", Expires: 15}}
br := []*discosrv.DatabaseAddress{{Address: "a", Expires: 15}, {Address: "b", Expires: 10}}
res := merge(&discosrv.DatabaseRecord{Addresses: ar}, &discosrv.DatabaseRecord{Addresses: br})
if len(res.Addresses) != 2 {
b.Fatal("wrong length")
}
if res.Addresses[0].Address != "a" || res.Addresses[1].Address != "b" {
b.Fatal("wrong address")
}
if res.Addresses[0].Expires != 15 || res.Addresses[1].Expires != 15 {
b.Fatal("wrong expiry")
}
}
b.ReportAllocs() // should be zero per operation
}
type testClock struct {
now time.Time
}
@@ -209,5 +259,6 @@ func (t *testClock) wind(d time.Duration) {
}
func (t *testClock) Now() time.Time {
t.now = t.now.Add(time.Nanosecond)
return t.now
}

View File

@@ -9,20 +9,24 @@ package main
import (
"context"
"crypto/tls"
"flag"
"log"
"net"
"net/http"
_ "net/http/pprof"
"os"
"strings"
"os/signal"
"runtime"
"time"
"github.com/alecthomas/kong"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/thejerf/suture/v4"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/rand"
"github.com/syncthing/syncthing/lib/s3"
"github.com/syncthing/syncthing/lib/tlsutil"
"github.com/syndtr/goleveldb/leveldb/opt"
"github.com/thejerf/suture/v4"
)
const (
@@ -36,17 +40,12 @@ const (
errorRetryAfterSeconds = 1500
errorRetryFuzzSeconds = 300
// Retry for not found is minSeconds + failures * incSeconds +
// random(fuzz), where failures is the number of consecutive lookups
// with no answer, up to maxSeconds. The fuzz is applied after capping
// to maxSeconds.
notFoundRetryMinSeconds = 60
notFoundRetryMaxSeconds = 3540
notFoundRetryIncSeconds = 10
notFoundRetryFuzzSeconds = 60
// How often (in requests) we serialize the missed counter to database.
notFoundMissesWriteInterval = 10
// Retry for not found is notFoundRetrySeenSeconds for records we have
// seen an announcement for (but it's not active right now) and
// notFoundRetryUnknownSeconds for records we have never seen (or not
// seen within the last week).
notFoundRetryUnknownMinSeconds = 60
notFoundRetryUnknownMaxSeconds = 3600
httpReadTimeout = 5 * time.Second
httpWriteTimeout = 5 * time.Second
@@ -56,139 +55,116 @@ const (
replicationOutboxSize = 10000
)
// These options make the database a little more optimized for writes, at
// the expense of some memory usage and risk of losing writes in a (system)
// crash.
var levelDBOptions = &opt.Options{
NoSync: true,
WriteBuffer: 32 << 20, // default 4<<20
var debug = false
type CLI struct {
Cert string `group:"Listen" help:"Certificate file" default:"./cert.pem" env:"DISCOVERY_CERT_FILE"`
Key string `group:"Listen" help:"Key file" default:"./key.pem" env:"DISCOVERY_KEY_FILE"`
HTTP bool `group:"Listen" help:"Listen on HTTP (behind an HTTPS proxy)" env:"DISCOVERY_HTTP"`
Compression bool `group:"Listen" help:"Enable GZIP compression of responses" env:"DISCOVERY_COMPRESSION"`
Listen string `group:"Listen" help:"Listen address" default:":8443" env:"DISCOVERY_LISTEN"`
MetricsListen string `group:"Listen" help:"Metrics listen address" env:"DISCOVERY_METRICS_LISTEN"`
DBDir string `group:"Database" help:"Database directory" default:"." env:"DISCOVERY_DB_DIR"`
DBFlushInterval time.Duration `group:"Database" help:"Interval between database flushes" default:"5m" env:"DISCOVERY_DB_FLUSH_INTERVAL"`
DBS3Endpoint string `name:"db-s3-endpoint" group:"Database (S3 backup)" hidden:"true" help:"S3 endpoint for database" env:"DISCOVERY_DB_S3_ENDPOINT"`
DBS3Region string `name:"db-s3-region" group:"Database (S3 backup)" hidden:"true" help:"S3 region for database" env:"DISCOVERY_DB_S3_REGION"`
DBS3Bucket string `name:"db-s3-bucket" group:"Database (S3 backup)" hidden:"true" help:"S3 bucket for database" env:"DISCOVERY_DB_S3_BUCKET"`
DBS3AccessKeyID string `name:"db-s3-access-key-id" group:"Database (S3 backup)" hidden:"true" help:"S3 access key ID for database" env:"DISCOVERY_DB_S3_ACCESS_KEY_ID"`
DBS3SecretKey string `name:"db-s3-secret-key" group:"Database (S3 backup)" hidden:"true" help:"S3 secret key for database" env:"DISCOVERY_DB_S3_SECRET_KEY"`
AMQPAddress string `group:"AMQP replication" hidden:"true" help:"Address to AMQP broker" env:"DISCOVERY_AMQP_ADDRESS"`
Debug bool `short:"d" help:"Print debug output" env:"DISCOVERY_DEBUG"`
Version bool `short:"v" help:"Print version and exit"`
}
var (
debug = false
)
func main() {
var listen string
var dir string
var metricsListen string
var replicationListen string
var replicationPeers string
var certFile string
var keyFile string
var useHTTP bool
log.SetOutput(os.Stdout)
log.SetFlags(0)
flag.StringVar(&certFile, "cert", "./cert.pem", "Certificate file")
flag.StringVar(&dir, "db-dir", "./discovery.db", "Database directory")
flag.BoolVar(&debug, "debug", false, "Print debug output")
flag.BoolVar(&useHTTP, "http", false, "Listen on HTTP (behind an HTTPS proxy)")
flag.StringVar(&listen, "listen", ":8443", "Listen address")
flag.StringVar(&keyFile, "key", "./key.pem", "Key file")
flag.StringVar(&metricsListen, "metrics-listen", "", "Metrics listen address")
flag.StringVar(&replicationPeers, "replicate", "", "Replication peers, id@address, comma separated")
flag.StringVar(&replicationListen, "replication-listen", ":19200", "Replication listen address")
showVersion := flag.Bool("version", false, "Show version")
flag.Parse()
var cli CLI
kong.Parse(&cli)
debug = cli.Debug
log.Println(build.LongVersionFor("stdiscosrv"))
if *showVersion {
if cli.Version {
return
}
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if os.IsNotExist(err) {
log.Println("Failed to load keypair. Generating one, this might take a while...")
cert, err = tlsutil.NewCertificate(certFile, keyFile, "stdiscosrv", 20*365)
if err != nil {
log.Fatalln("Failed to generate X509 key pair:", err)
}
} else if err != nil {
log.Fatalln("Failed to load keypair:", err)
}
devID := protocol.NewDeviceID(cert.Certificate[0])
log.Println("Server device ID is", devID)
buildInfo.WithLabelValues(build.Version, runtime.Version(), build.User, build.Date.UTC().Format("2006-01-02T15:04:05Z")).Set(1)
// Parse the replication specs, if any.
var allowedReplicationPeers []protocol.DeviceID
var replicationDestinations []string
parts := strings.Split(replicationPeers, ",")
for _, part := range parts {
if part == "" {
continue
}
fields := strings.Split(part, "@")
switch len(fields) {
case 2:
// This is an id@address specification. Grab the address for the
// destination list. Try to resolve it once to catch obvious
// syntax errors here rather than having the sender service fail
// repeatedly later.
_, err := net.ResolveTCPAddr("tcp", fields[1])
var cert tls.Certificate
if !cli.HTTP {
var err error
cert, err = tls.LoadX509KeyPair(cli.Cert, cli.Key)
if os.IsNotExist(err) {
log.Println("Failed to load keypair. Generating one, this might take a while...")
cert, err = tlsutil.NewCertificate(cli.Cert, cli.Key, "stdiscosrv", 20*365)
if err != nil {
log.Fatalln("Resolving address:", err)
log.Fatalln("Failed to generate X509 key pair:", err)
}
replicationDestinations = append(replicationDestinations, fields[1])
fallthrough // N.B.
case 1:
// The first part is always a device ID.
id, err := protocol.DeviceIDFromString(fields[0])
if err != nil {
log.Fatalln("Parsing device ID:", err)
}
if id == protocol.EmptyDeviceID {
log.Fatalf("Missing device ID for peer in %q", part)
}
allowedReplicationPeers = append(allowedReplicationPeers, id)
default:
log.Fatalln("Unrecognized replication spec:", part)
} else if err != nil {
log.Fatalln("Failed to load keypair:", err)
}
devID := protocol.NewDeviceID(cert.Certificate[0])
log.Println("Server device ID is", devID)
}
// Root of the service tree.
main := suture.New("main", suture.Spec{
PassThroughPanics: true,
Timeout: 2 * time.Minute,
})
// Start the database.
db, err := newLevelDBStore(dir)
if err != nil {
log.Fatalln("Open database:", err)
// If configured, use S3 for database backups.
var s3c *s3.Session
if cli.DBS3Endpoint != "" {
var err error
s3c, err = s3.NewSession(cli.DBS3Endpoint, cli.DBS3Region, cli.DBS3Bucket, cli.DBS3AccessKeyID, cli.DBS3SecretKey)
if err != nil {
log.Fatalf("Failed to create S3 session: %v", err)
}
}
// Start the database.
db := newInMemoryStore(cli.DBDir, cli.DBFlushInterval, s3c)
main.Add(db)
// Start any replication senders.
var repl replicationMultiplexer
for _, dst := range replicationDestinations {
rs := newReplicationSender(dst, cert, allowedReplicationPeers)
main.Add(rs)
repl = append(repl, rs)
}
// If we have replication configured, start the replication listener.
if len(allowedReplicationPeers) > 0 {
rl := newReplicationListener(replicationListen, cert, allowedReplicationPeers, db)
main.Add(rl)
// If we have an AMQP broker for replication, start that
var repl replicator
if cli.AMQPAddress != "" {
clientID := rand.String(10)
kr := newAMQPReplicator(cli.AMQPAddress, clientID, db)
main.Add(kr)
repl = kr
}
// Start the main API server.
qs := newAPISrv(listen, cert, db, repl, useHTTP)
qs := newAPISrv(cli.Listen, cert, db, repl, cli.HTTP, cli.Compression)
main.Add(qs)
// If we have a metrics port configured, start a metrics handler.
if metricsListen != "" {
if cli.MetricsListen != "" {
go func() {
mux := http.NewServeMux()
mux.Handle("/metrics", promhttp.Handler())
log.Fatal(http.ListenAndServe(metricsListen, mux))
log.Fatal(http.ListenAndServe(cli.MetricsListen, mux))
}()
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Cancel on signal
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, os.Interrupt)
go func() {
sig := <-signalChan
log.Printf("Received signal %s; shutting down", sig)
cancel()
}()
// Engage!
main.Serve(context.Background())
main.Serve(ctx)
}

View File

@@ -1,315 +0,0 @@
// Copyright (C) 2018 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"context"
"crypto/tls"
"encoding/binary"
"fmt"
io "io"
"log"
"net"
"time"
"github.com/syncthing/syncthing/lib/protocol"
)
const replicationReadTimeout = time.Minute
const replicationHeartbeatInterval = time.Second * 30
type replicator interface {
send(key string, addrs []DatabaseAddress, seen int64)
}
// a replicationSender tries to connect to the remote address and provide
// them with a feed of replication updates.
type replicationSender struct {
dst string
cert tls.Certificate // our certificate
allowedIDs []protocol.DeviceID
outbox chan ReplicationRecord
}
func newReplicationSender(dst string, cert tls.Certificate, allowedIDs []protocol.DeviceID) *replicationSender {
return &replicationSender{
dst: dst,
cert: cert,
allowedIDs: allowedIDs,
outbox: make(chan ReplicationRecord, replicationOutboxSize),
}
}
func (s *replicationSender) Serve(ctx context.Context) error {
// Sleep a little at startup. Peers often restart at the same time, and
// this avoid the service failing and entering backoff state
// unnecessarily, while also reducing the reconnect rate to something
// reasonable by default.
time.Sleep(2 * time.Second)
tlsCfg := &tls.Config{
Certificates: []tls.Certificate{s.cert},
MinVersion: tls.VersionTLS12,
InsecureSkipVerify: true,
}
// Dial the TLS connection.
conn, err := tls.Dial("tcp", s.dst, tlsCfg)
if err != nil {
log.Println("Replication connect:", err)
return err
}
defer func() {
conn.SetWriteDeadline(time.Now().Add(time.Second))
conn.Close()
}()
// Get the other side device ID.
remoteID, err := deviceID(conn)
if err != nil {
log.Println("Replication connect:", err)
return err
}
// Verify it's in the set of allowed device IDs.
if !deviceIDIn(remoteID, s.allowedIDs) {
log.Println("Replication connect: unexpected device ID:", remoteID)
return err
}
heartBeatTicker := time.NewTicker(replicationHeartbeatInterval)
defer heartBeatTicker.Stop()
// Send records.
buf := make([]byte, 1024)
for {
select {
case <-heartBeatTicker.C:
if len(s.outbox) > 0 {
// No need to send heartbeats if there are events/prevrious
// heartbeats to send, they will keep the connection alive.
continue
}
// Empty replication message is the heartbeat:
s.outbox <- ReplicationRecord{}
case rec := <-s.outbox:
// Buffer must hold record plus four bytes for size
size := rec.Size()
if len(buf) < size+4 {
buf = make([]byte, size+4)
}
// Record comes after the four bytes size
n, err := rec.MarshalTo(buf[4:])
if err != nil {
// odd to get an error here, but we haven't sent anything
// yet so it's not fatal
replicationSendsTotal.WithLabelValues("error").Inc()
log.Println("Replication marshal:", err)
continue
}
binary.BigEndian.PutUint32(buf, uint32(n))
// Send
conn.SetWriteDeadline(time.Now().Add(5 * time.Second))
if _, err := conn.Write(buf[:4+n]); err != nil {
replicationSendsTotal.WithLabelValues("error").Inc()
log.Println("Replication write:", err)
// Yes, we are losing the replication event here.
return err
}
replicationSendsTotal.WithLabelValues("success").Inc()
case <-ctx.Done():
return nil
}
}
}
func (s *replicationSender) String() string {
return fmt.Sprintf("replicationSender(%q)", s.dst)
}
func (s *replicationSender) send(key string, ps []DatabaseAddress, _ int64) {
item := ReplicationRecord{
Key: key,
Addresses: ps,
}
// The send should never block. The inbox is suitably buffered for at
// least a few seconds of stalls, which shouldn't happen in practice.
select {
case s.outbox <- item:
default:
replicationSendsTotal.WithLabelValues("drop").Inc()
}
}
// a replicationMultiplexer sends to multiple replicators
type replicationMultiplexer []replicator
func (m replicationMultiplexer) send(key string, ps []DatabaseAddress, seen int64) {
for _, s := range m {
// each send is nonblocking
s.send(key, ps, seen)
}
}
// replicationListener accepts incoming connections and reads replication
// items from them. Incoming items are applied to the KV store.
type replicationListener struct {
addr string
cert tls.Certificate
allowedIDs []protocol.DeviceID
db database
}
func newReplicationListener(addr string, cert tls.Certificate, allowedIDs []protocol.DeviceID, db database) *replicationListener {
return &replicationListener{
addr: addr,
cert: cert,
allowedIDs: allowedIDs,
db: db,
}
}
func (l *replicationListener) Serve(ctx context.Context) error {
tlsCfg := &tls.Config{
Certificates: []tls.Certificate{l.cert},
ClientAuth: tls.RequestClientCert,
MinVersion: tls.VersionTLS12,
InsecureSkipVerify: true,
}
lst, err := tls.Listen("tcp", l.addr, tlsCfg)
if err != nil {
log.Println("Replication listen:", err)
return err
}
defer lst.Close()
for {
select {
case <-ctx.Done():
return nil
default:
}
// Accept a connection
conn, err := lst.Accept()
if err != nil {
log.Println("Replication accept:", err)
return err
}
// Figure out the other side device ID
remoteID, err := deviceID(conn.(*tls.Conn))
if err != nil {
log.Println("Replication accept:", err)
conn.SetWriteDeadline(time.Now().Add(time.Second))
conn.Close()
continue
}
// Verify it is in the set of allowed device IDs
if !deviceIDIn(remoteID, l.allowedIDs) {
log.Println("Replication accept: unexpected device ID:", remoteID)
conn.SetWriteDeadline(time.Now().Add(time.Second))
conn.Close()
continue
}
go l.handle(ctx, conn)
}
}
func (l *replicationListener) String() string {
return fmt.Sprintf("replicationListener(%q)", l.addr)
}
func (l *replicationListener) handle(ctx context.Context, conn net.Conn) {
defer func() {
conn.SetWriteDeadline(time.Now().Add(time.Second))
conn.Close()
}()
buf := make([]byte, 1024)
for {
select {
case <-ctx.Done():
return
default:
}
conn.SetReadDeadline(time.Now().Add(replicationReadTimeout))
// First four bytes are the size
if _, err := io.ReadFull(conn, buf[:4]); err != nil {
log.Println("Replication read size:", err)
replicationRecvsTotal.WithLabelValues("error").Inc()
return
}
// Read the rest of the record
size := int(binary.BigEndian.Uint32(buf[:4]))
if len(buf) < size {
buf = make([]byte, size)
}
if size == 0 {
// Heartbeat, ignore
continue
}
if _, err := io.ReadFull(conn, buf[:size]); err != nil {
log.Println("Replication read record:", err)
replicationRecvsTotal.WithLabelValues("error").Inc()
return
}
// Unmarshal
var rec ReplicationRecord
if err := rec.Unmarshal(buf[:size]); err != nil {
log.Println("Replication unmarshal:", err)
replicationRecvsTotal.WithLabelValues("error").Inc()
continue
}
// Store
l.db.merge(rec.Key, rec.Addresses, rec.Seen)
replicationRecvsTotal.WithLabelValues("success").Inc()
}
}
func deviceID(conn *tls.Conn) (protocol.DeviceID, error) {
// Handshake may not be complete on the server side yet, which we need
// to get the client certificate.
if !conn.ConnectionState().HandshakeComplete {
if err := conn.Handshake(); err != nil {
return protocol.DeviceID{}, err
}
}
// We expect exactly one certificate.
certs := conn.ConnectionState().PeerCertificates
if len(certs) != 1 {
return protocol.DeviceID{}, fmt.Errorf("unexpected number of certificates (%d != 1)", len(certs))
}
return protocol.NewDeviceID(certs[0].Raw), nil
}
func deviceIDIn(id protocol.DeviceID, ids []protocol.DeviceID) bool {
for _, candidate := range ids {
if id == candidate {
return true
}
}
return false
}

View File

@@ -7,13 +7,18 @@
package main
import (
"os"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/collectors"
)
var (
buildInfo = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "discovery",
Name: "build_info",
Help: "A metric with a constant '1' value labeled by version, goversion, builduser and builddate from which stdiscosrv was built.",
}, []string{"version", "goversion", "builduser", "builddate"})
apiRequestsTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Namespace: "syncthing",
@@ -90,6 +95,29 @@ var (
Help: "Latency of database operations.",
Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
}, []string{"operation"})
databaseWriteSeconds = prometheus.NewGauge(
prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "discovery",
Name: "database_write_seconds",
Help: "Time spent writing the database.",
})
databaseLastWritten = prometheus.NewGauge(
prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "discovery",
Name: "database_last_written",
Help: "Timestamp of the last successful database write.",
})
retryAfterLevel = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "discovery",
Name: "retry_after_seconds",
Help: "Retry-After header value in seconds.",
}, []string{"name"})
)
const (
@@ -104,21 +132,12 @@ const (
)
func init() {
prometheus.MustRegister(apiRequestsTotal, apiRequestsSeconds,
prometheus.MustRegister(buildInfo,
apiRequestsTotal, apiRequestsSeconds,
lookupRequestsTotal, announceRequestsTotal,
replicationSendsTotal, replicationRecvsTotal,
databaseKeys, databaseStatisticsSeconds,
databaseOperations, databaseOperationSeconds)
processCollectorOpts := collectors.ProcessCollectorOpts{
Namespace: "syncthing_discovery",
PidFn: func() (int, error) {
return os.Getpid(), nil
},
}
prometheus.MustRegister(
collectors.NewProcessCollector(processCollectorOpts),
)
databaseOperations, databaseOperationSeconds,
databaseWriteSeconds, databaseLastWritten,
retryAfterLevel)
}

View File

@@ -12,15 +12,14 @@ import (
"time"
syncthingprotocol "github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/tlsutil"
"github.com/syncthing/syncthing/lib/relay/protocol"
"github.com/syncthing/syncthing/lib/tlsutil"
)
var (
outboxesMut = sync.RWMutex{}
outboxes = make(map[syncthingprotocol.DeviceID]chan interface{})
numConnections int64
numConnections atomic.Int64
)
func listener(_, addr string, config *tls.Config, token string) {
@@ -36,8 +35,14 @@ func listener(_, addr string, config *tls.Config, token string) {
for {
conn, isTLS, err := listener.AcceptNoWrapTLS()
if err != nil {
// Conn may be nil if accept failed, or non-nil if the initial
// read to figure out if it's TLS or not failed. In the latter
// case, close the connection before moving on.
if conn != nil {
conn.Close()
}
if debug {
log.Println("Listener failed to accept connection from", conn.RemoteAddr(), ". Possibly a TCP Ping.")
log.Println("Listener failed to accept:", err)
}
continue
}
@@ -128,7 +133,7 @@ func protocolConnectionHandler(tcpConn net.Conn, config *tls.Config, token strin
continue
}
if atomic.LoadInt32(&overLimit) > 0 {
if overLimit.Load() {
protocol.WriteMessage(conn, protocol.RelayFull{})
if debug {
log.Println("Refusing join request from", id, "due to being over limits")
@@ -267,7 +272,7 @@ func protocolConnectionHandler(tcpConn net.Conn, config *tls.Config, token strin
conn.Close()
}
if atomic.LoadInt32(&overLimit) > 0 && !hasSessions(id) {
if overLimit.Load() && !hasSessions(id) {
if debug {
log.Println("Dropping", id, "as it has no sessions and we are over our limits")
}
@@ -360,8 +365,8 @@ func sessionConnectionHandler(conn net.Conn) {
}
func messageReader(conn net.Conn, messages chan<- interface{}, errors chan<- error) {
atomic.AddInt64(&numConnections, 1)
defer atomic.AddInt64(&numConnections, -1)
numConnections.Add(1)
defer numConnections.Add(-1)
for {
msg, err := protocol.ReadMessage(conn)

View File

@@ -19,19 +19,19 @@ import (
"syscall"
"time"
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/osutil"
"github.com/syncthing/syncthing/lib/relay/protocol"
"github.com/syncthing/syncthing/lib/tlsutil"
"golang.org/x/time/rate"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/nat"
"github.com/syncthing/syncthing/lib/osutil"
_ "github.com/syncthing/syncthing/lib/pmp"
_ "github.com/syncthing/syncthing/lib/upnp"
syncthingprotocol "github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/relay/protocol"
"github.com/syncthing/syncthing/lib/tlsutil"
_ "github.com/syncthing/syncthing/lib/upnp"
)
var (
@@ -49,7 +49,7 @@ var (
sessionLimitBps int
globalLimitBps int
overLimit int32
overLimit atomic.Bool
descriptorLimit int64
sessionLimiter *rate.Limiter
globalLimiter *rate.Limiter
@@ -194,7 +194,15 @@ func main() {
cfg.Options.NATTimeoutS = natTimeout
})
natSvc := nat.NewService(id, wrapper)
mapping := mapping{natSvc.NewMapping(nat.TCP, addr.IP, addr.Port)}
var ipVersion nat.IPVersion
if strings.HasSuffix(proto, "4") {
ipVersion = nat.IPv4Only
} else if strings.HasSuffix(proto, "6") {
ipVersion = nat.IPv6Only
} else {
ipVersion = nat.IPvAny
}
mapping := mapping{natSvc.NewMapping(nat.TCP, ipVersion, addr.IP, addr.Port)}
if natEnabled {
ctx, cancel := context.WithCancel(context.Background())
@@ -308,10 +316,10 @@ func main() {
func monitorLimits() {
limitCheckTimer = time.NewTimer(time.Minute)
for range limitCheckTimer.C {
if atomic.LoadInt64(&numConnections)+atomic.LoadInt64(&numProxies) > descriptorLimit {
atomic.StoreInt32(&overLimit, 1)
if numConnections.Load()+numProxies.Load() > descriptorLimit {
overLimit.Store(true)
log.Println("Gone past our connection limits. Starting to refuse new/drop idle connections.")
} else if atomic.CompareAndSwapInt32(&overLimit, 1, 0) {
} else if overLimit.CompareAndSwap(true, false) {
log.Println("Dropped below our connection limits. Accepting new connections.")
}
limitCheckTimer.Reset(time.Minute)

View File

@@ -23,8 +23,8 @@ var (
sessionMut = sync.RWMutex{}
activeSessions = make([]*session, 0)
pendingSessions = make(map[string]*session)
numProxies int64
bytesProxied int64
numProxies atomic.Int64
bytesProxied atomic.Int64
)
func newSession(serverid, clientid syncthingprotocol.DeviceID, sessionRateLimit, globalRateLimit *rate.Limiter) *session {
@@ -68,7 +68,6 @@ func findSession(key string) *session {
ses, ok := pendingSessions[key]
if !ok {
return nil
}
delete(pendingSessions, key)
return ses
@@ -251,8 +250,8 @@ func (s *session) proxy(c1, c2 net.Conn) error {
log.Println("Proxy", c1.RemoteAddr(), "->", c2.RemoteAddr())
}
atomic.AddInt64(&numProxies, 1)
defer atomic.AddInt64(&numProxies, -1)
numProxies.Add(1)
defer numProxies.Add(-1)
buf := make([]byte, networkBufferSize)
for {
@@ -262,7 +261,7 @@ func (s *session) proxy(c1, c2 net.Conn) error {
return err
}
atomic.AddInt64(&bytesProxied, int64(n))
bytesProxied.Add(int64(n))
if debug {
log.Printf("%d bytes from %s to %s", n, c1.RemoteAddr(), c2.RemoteAddr())

View File

@@ -51,9 +51,9 @@ func getStatus(w http.ResponseWriter, _ *http.Request) {
status["numPendingSessionKeys"] = len(pendingSessions)
status["numActiveSessions"] = len(activeSessions)
sessionMut.Unlock()
status["numConnections"] = atomic.LoadInt64(&numConnections)
status["numProxies"] = atomic.LoadInt64(&numProxies)
status["bytesProxied"] = atomic.LoadInt64(&bytesProxied)
status["numConnections"] = numConnections.Load()
status["numProxies"] = numProxies.Load()
status["bytesProxied"] = bytesProxied.Load()
status["goVersion"] = runtime.Version()
status["goOS"] = runtime.GOOS
status["goArch"] = runtime.GOARCH
@@ -88,13 +88,13 @@ func getStatus(w http.ResponseWriter, _ *http.Request) {
}
type rateCalculator struct {
counter *int64 // atomic, must remain 64-bit aligned
counter *atomic.Int64
rates []int64
prev int64
startTime time.Time
}
func newRateCalculator(keepIntervals int, interval time.Duration, counter *int64) *rateCalculator {
func newRateCalculator(keepIntervals int, interval time.Duration, counter *atomic.Int64) *rateCalculator {
r := &rateCalculator{
rates: make([]int64, keepIntervals),
counter: counter,
@@ -112,7 +112,7 @@ func (r *rateCalculator) updateRates(interval time.Duration) {
next := now.Truncate(interval).Add(interval)
time.Sleep(next.Sub(now))
cur := atomic.LoadInt64(r.counter)
cur := r.counter.Load()
rate := int64(float64(cur-r.prev) / interval.Seconds())
copy(r.rates[1:], r.rates)
r.rates[0] = rate

View File

@@ -14,6 +14,7 @@ import (
"path/filepath"
"time"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
syncthingprotocol "github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/relay/client"
"github.com/syncthing/syncthing/lib/relay/protocol"

View File

@@ -1,57 +0,0 @@
// Copyright (C) 2019 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at https://mozilla.org/MPL/2.0/.
package main
import (
"encoding/json"
"flag"
"os"
"sort"
"github.com/syncthing/syncthing/lib/upgrade"
)
const defaultURL = "https://api.github.com/repos/syncthing/syncthing/releases?per_page=25"
func main() {
url := flag.String("u", defaultURL, "GitHub releases url")
flag.Parse()
rels := upgrade.FetchLatestReleases(*url, "")
if rels == nil {
// An error was already logged
os.Exit(1)
}
sort.Sort(upgrade.SortByRelease(rels))
rels = filterForLatest(rels)
if err := json.NewEncoder(os.Stdout).Encode(rels); err != nil {
os.Exit(1)
}
}
// filterForLatest returns the latest stable and prerelease only. If the
// stable version is newer (comes first in the list) there is no need to go
// looking for a prerelease at all.
func filterForLatest(rels []upgrade.Release) []upgrade.Release {
var filtered []upgrade.Release
var havePre bool
for _, rel := range rels {
if !rel.Prerelease {
// We found a stable version, we're good now.
filtered = append(filtered, rel)
break
}
if rel.Prerelease && !havePre {
// We remember the first prerelease we find.
filtered = append(filtered, rel)
havePre = true
}
}
return filtered
}

View File

@@ -13,6 +13,7 @@ import (
"reflect"
"github.com/AudriusButkevicius/recli"
"github.com/alecthomas/kong"
"github.com/syncthing/syncthing/lib/config"
"github.com/urfave/cli"
)
@@ -23,9 +24,20 @@ type configHandler struct {
err error
}
func getConfigCommand(f *apiClientFactory) (cli.Command, error) {
type configCommand struct {
Args []string `arg:"" default:"-h"`
}
func (c *configCommand) Run(ctx Context, _ *kong.Context) error {
app := cli.NewApp()
app.Name = "syncthing"
app.Author = "The Syncthing Authors"
app.Metadata = map[string]interface{}{
"clientFactory": ctx.clientFactory,
}
h := new(configHandler)
h.client, h.err = f.getClient()
h.client, h.err = ctx.clientFactory.getClient()
if h.err == nil {
h.cfg, h.err = getConfig(h.client)
}
@@ -38,17 +50,15 @@ func getConfigCommand(f *apiClientFactory) (cli.Command, error) {
commands, err := recli.New(recliCfg).Construct(&h.cfg)
if err != nil {
return cli.Command{}, fmt.Errorf("config reflect: %w", err)
return fmt.Errorf("config reflect: %w", err)
}
return cli.Command{
Name: "config",
HideHelp: true,
Usage: "Configuration modification command group",
Subcommands: commands,
Before: h.configBefore,
After: h.configAfter,
}, nil
app.Commands = commands
app.HideHelp = true
app.Before = h.configBefore
app.After = h.configAfter
return app.Run(append([]string{app.Name}, c.Args...))
}
func (h *configHandler) configBefore(c *cli.Context) error {

View File

@@ -9,47 +9,37 @@ package cli
import (
"fmt"
"net/url"
"github.com/urfave/cli"
)
var debugCommand = cli.Command{
Name: "debug",
HideHelp: true,
Usage: "Debug command group",
Subcommands: []cli.Command{
{
Name: "file",
Usage: "Show information about a file (or directory/symlink)",
ArgsUsage: "FOLDER-ID PATH",
Action: expects(2, debugFile()),
},
indexCommand,
{
Name: "profile",
Usage: "Save a profile to help figuring out what Syncthing does.",
ArgsUsage: "cpu | heap",
Action: expects(1, profile()),
},
},
type fileCommand struct {
FolderID string `arg:""`
Path string `arg:""`
}
func debugFile() cli.ActionFunc {
return func(c *cli.Context) error {
query := make(url.Values)
query.Set("folder", c.Args()[0])
query.Set("file", normalizePath(c.Args()[1]))
return indexDumpOutput("debug/file?" + query.Encode())(c)
func (f *fileCommand) Run(ctx Context) error {
indexDumpOutput := indexDumpOutputWrapper(ctx.clientFactory)
query := make(url.Values)
query.Set("folder", f.FolderID)
query.Set("file", normalizePath(f.Path))
return indexDumpOutput("debug/file?" + query.Encode())
}
type profileCommand struct {
Type string `arg:"" help:"cpu | heap"`
}
func (p *profileCommand) Run(ctx Context) error {
switch t := p.Type; t {
case "cpu", "heap":
return saveToFile(fmt.Sprintf("debug/%vprof", p.Type), ctx.clientFactory)
default:
return fmt.Errorf("expected cpu or heap as argument, got %v", t)
}
}
func profile() cli.ActionFunc {
return func(c *cli.Context) error {
switch t := c.Args()[0]; t {
case "cpu", "heap":
return saveToFile(fmt.Sprintf("debug/%vprof", c.Args()[0]))(c)
default:
return fmt.Errorf("expected cpu or heap as argument, got %v", t)
}
}
type debugCommand struct {
File fileCommand `cmd:"" help:"Show information about a file (or directory/symlink)"`
Profile profileCommand `cmd:"" help:"Save a profile to help figuring out what Syncthing does"`
Index indexCommand `cmd:"" help:"Show information about the index (database)"`
}

View File

@@ -11,36 +11,25 @@ import (
"fmt"
"strings"
"github.com/urfave/cli"
"github.com/alecthomas/kong"
)
var errorsCommand = cli.Command{
Name: "errors",
HideHelp: true,
Usage: "Error command group",
Subcommands: []cli.Command{
{
Name: "show",
Usage: "Show pending errors",
Action: expects(0, indexDumpOutput("system/error")),
},
{
Name: "push",
Usage: "Push an error to active clients",
ArgsUsage: "ERROR-MESSAGE",
Action: expects(1, errorsPush),
},
{
Name: "clear",
Usage: "Clear pending errors",
Action: expects(0, emptyPost("system/error/clear")),
},
},
type errorsCommand struct {
Show struct{} `cmd:"" help:"Show pending errors"`
Push errorsPushCommand `cmd:"" help:"Push an error to active clients"`
Clear struct{} `cmd:"" help:"Clear pending errors"`
}
func errorsPush(c *cli.Context) error {
client := c.App.Metadata["client"].(APIClient)
errStr := strings.Join(c.Args(), " ")
type errorsPushCommand struct {
ErrorMessage string `arg:""`
}
func (e *errorsPushCommand) Run(ctx Context) error {
client, err := ctx.clientFactory.getClient()
if err != nil {
return err
}
errStr := e.ErrorMessage
response, err := client.Post("system/error", strings.TrimSpace(errStr))
if err != nil {
return err
@@ -59,3 +48,13 @@ func errorsPush(c *cli.Context) error {
}
return nil
}
func (*errorsCommand) Run(ctx Context, kongCtx *kong.Context) error {
switch kongCtx.Selected().Name {
case "show":
return indexDumpOutput("system/error", ctx.clientFactory)
case "clear":
return emptyPost("system/error/clear", ctx.clientFactory)
}
return nil
}

View File

@@ -7,32 +7,26 @@
package cli
import (
"github.com/urfave/cli"
"github.com/alecthomas/kong"
)
var indexCommand = cli.Command{
Name: "index",
Usage: "Show information about the index (database)",
Subcommands: []cli.Command{
{
Name: "dump",
Usage: "Print the entire db",
Action: expects(0, indexDump),
},
{
Name: "dump-size",
Usage: "Print the db size of different categories of information",
Action: expects(0, indexDumpSize),
},
{
Name: "check",
Usage: "Check the database for inconsistencies",
Action: expects(0, indexCheck),
},
{
Name: "account",
Usage: "Print key and value size statistics per key type",
Action: expects(0, indexAccount),
},
},
type indexCommand struct {
Dump struct{} `cmd:"" help:"Print the entire db"`
DumpSize struct{} `cmd:"" help:"Print the db size of different categories of information"`
Check struct{} `cmd:"" help:"Check the database for inconsistencies"`
Account struct{} `cmd:"" help:"Print key and value size statistics per key type"`
}
func (*indexCommand) Run(kongCtx *kong.Context) error {
switch kongCtx.Selected().Name {
case "dump":
return indexDump()
case "dump-size":
return indexDumpSize()
case "check":
return indexCheck()
case "account":
return indexAccount()
}
return nil
}

View File

@@ -10,12 +10,10 @@ import (
"fmt"
"os"
"text/tabwriter"
"github.com/urfave/cli"
)
// indexAccount prints key and data size statistics per class
func indexAccount(*cli.Context) error {
func indexAccount() error {
ldb, err := getDB()
if err != nil {
return err

View File

@@ -11,13 +11,15 @@ import (
"fmt"
"time"
"github.com/urfave/cli"
"google.golang.org/protobuf/proto"
"github.com/syncthing/syncthing/internal/gen/bep"
"github.com/syncthing/syncthing/internal/gen/dbproto"
"github.com/syncthing/syncthing/lib/db"
"github.com/syncthing/syncthing/lib/protocol"
)
func indexDump(*cli.Context) error {
func indexDump() error {
ldb, err := getDB()
if err != nil {
return err
@@ -35,19 +37,19 @@ func indexDump(*cli.Context) error {
name := nulString(key[1+4+4:])
fmt.Printf("[device] F:%d D:%d N:%q", folder, device, name)
var f protocol.FileInfo
err := f.Unmarshal(it.Value())
var f bep.FileInfo
err := proto.Unmarshal(it.Value(), &f)
if err != nil {
return err
}
fmt.Printf(" V:%v\n", f)
fmt.Printf(" V:%v\n", &f)
case db.KeyTypeGlobal:
folder := binary.BigEndian.Uint32(key[1:])
name := nulString(key[1+4:])
var flv db.VersionList
flv.Unmarshal(it.Value())
fmt.Printf("[global] F:%d N:%q V:%s\n", folder, name, flv)
var flv dbproto.VersionList
proto.Unmarshal(it.Value(), &flv)
fmt.Printf("[global] F:%d N:%q V:%s\n", folder, name, &flv)
case db.KeyTypeBlock:
folder := binary.BigEndian.Uint32(key[1:])
@@ -96,11 +98,11 @@ func indexDump(*cli.Context) error {
case db.KeyTypeFolderMeta:
folder := binary.BigEndian.Uint32(key[1:])
fmt.Printf("[foldermeta] F:%d", folder)
var cs db.CountsSet
if err := cs.Unmarshal(it.Value()); err != nil {
var cs dbproto.CountsSet
if err := proto.Unmarshal(it.Value(), &cs); err != nil {
fmt.Printf(" (invalid)\n")
} else {
fmt.Printf(" V:%v\n", cs)
fmt.Printf(" V:%v\n", &cs)
}
case db.KeyTypeMiscData:
@@ -127,20 +129,20 @@ func indexDump(*cli.Context) error {
case db.KeyTypeVersion:
fmt.Printf("[version] H:%x", key[1:])
var v protocol.Vector
err := v.Unmarshal(it.Value())
var v bep.Vector
err := proto.Unmarshal(it.Value(), &v)
if err != nil {
fmt.Printf(" (invalid)\n")
} else {
fmt.Printf(" V:%v\n", v)
fmt.Printf(" V:%v\n", &v)
}
case db.KeyTypePendingFolder:
device := binary.BigEndian.Uint32(key[1:])
folder := string(key[5:])
var of db.ObservedFolder
of.Unmarshal(it.Value())
fmt.Printf("[pendingFolder] D:%d F:%s V:%v\n", device, folder, of)
var of dbproto.ObservedFolder
proto.Unmarshal(it.Value(), &of)
fmt.Printf("[pendingFolder] D:%d F:%s V:%v\n", device, folder, &of)
case db.KeyTypePendingDevice:
device := "<invalid>"
@@ -148,9 +150,9 @@ func indexDump(*cli.Context) error {
if err == nil {
device = dev.String()
}
var od db.ObservedDevice
od.Unmarshal(it.Value())
fmt.Printf("[pendingDevice] D:%v V:%v\n", device, od)
var od dbproto.ObservedDevice
proto.Unmarshal(it.Value(), &od)
fmt.Printf("[pendingDevice] D:%v V:%v\n", device, &od)
default:
fmt.Printf("[??? %d]\n %x\n %x\n", key[0], key, it.Value())

View File

@@ -11,12 +11,10 @@ import (
"fmt"
"sort"
"github.com/urfave/cli"
"github.com/syncthing/syncthing/lib/db"
)
func indexDumpSize(*cli.Context) error {
func indexDumpSize() error {
type sizedElement struct {
key string
size int

View File

@@ -13,8 +13,10 @@ import (
"fmt"
"sort"
"github.com/urfave/cli"
"google.golang.org/protobuf/proto"
"github.com/syncthing/syncthing/internal/gen/bep"
"github.com/syncthing/syncthing/internal/gen/dbproto"
"github.com/syncthing/syncthing/lib/db"
"github.com/syncthing/syncthing/lib/protocol"
)
@@ -35,7 +37,7 @@ type sequenceKey struct {
sequence uint64
}
func indexCheck(*cli.Context) (err error) {
func indexCheck() (err error) {
ldb, err := getDB()
if err != nil {
return err
@@ -44,12 +46,12 @@ func indexCheck(*cli.Context) (err error) {
folders := make(map[uint32]string)
devices := make(map[uint32]string)
deviceToIDs := make(map[string]uint32)
fileInfos := make(map[fileInfoKey]protocol.FileInfo)
globals := make(map[globalKey]db.VersionList)
fileInfos := make(map[fileInfoKey]*bep.FileInfo)
globals := make(map[globalKey]*dbproto.VersionList)
sequences := make(map[sequenceKey]string)
needs := make(map[globalKey]struct{})
blocklists := make(map[string]struct{})
versions := make(map[string]protocol.Vector)
versions := make(map[string]*bep.Vector)
usedBlocklists := make(map[string]struct{})
usedVersions := make(map[string]struct{})
var localDeviceKey uint32
@@ -76,26 +78,26 @@ func indexCheck(*cli.Context) (err error) {
device := binary.BigEndian.Uint32(key[1+4:])
name := nulString(key[1+4+4:])
var f protocol.FileInfo
err := f.Unmarshal(it.Value())
var f bep.FileInfo
err := proto.Unmarshal(it.Value(), &f)
if err != nil {
fmt.Println("Unable to unmarshal FileInfo:", err)
success = false
continue
}
fileInfos[fileInfoKey{folder, device, name}] = f
fileInfos[fileInfoKey{folder, device, name}] = &f
case db.KeyTypeGlobal:
folder := binary.BigEndian.Uint32(key[1:])
name := nulString(key[1+4:])
var flv db.VersionList
if err := flv.Unmarshal(it.Value()); err != nil {
var flv dbproto.VersionList
if err := proto.Unmarshal(it.Value(), &flv); err != nil {
fmt.Println("Unable to unmarshal VersionList:", err)
success = false
continue
}
globals[globalKey{folder, name}] = flv
globals[globalKey{folder, name}] = &flv
case db.KeyTypeFolderIdx:
key := binary.BigEndian.Uint32(it.Key()[1:])
@@ -126,13 +128,13 @@ func indexCheck(*cli.Context) (err error) {
case db.KeyTypeVersion:
hash := string(key[1:])
var v protocol.Vector
if err := v.Unmarshal(it.Value()); err != nil {
var v bep.Vector
if err := proto.Unmarshal(it.Value(), &v); err != nil {
fmt.Println("Unable to unmarshal Vector:", err)
success = false
continue
}
versions[hash] = v
versions[hash] = &v
}
}
@@ -250,25 +252,27 @@ func indexCheck(*cli.Context) (err error) {
if fi.VersionHash != nil {
fiv = versions[string(fi.VersionHash)]
}
if !fiv.Equal(version) {
if !protocol.VectorFromWire(fiv).Equal(version) {
fmt.Printf("VersionList %q, folder %q, entry %d, FileInfo version mismatch, %v (VersionList) != %v (FileInfo)\n", gk.name, folder, i, version, fi.Version)
success = false
}
if fi.IsInvalid() != invalid {
fmt.Printf("VersionList %q, folder %q, entry %d, FileInfo invalid mismatch, %v (VersionList) != %v (FileInfo)\n", gk.name, folder, i, invalid, fi.IsInvalid())
ffi := protocol.FileInfoFromDB(fi)
if ffi.IsInvalid() != invalid {
fmt.Printf("VersionList %q, folder %q, entry %d, FileInfo invalid mismatch, %v (VersionList) != %v (FileInfo)\n", gk.name, folder, i, invalid, ffi.IsInvalid())
success = false
}
if fi.IsDeleted() != deleted {
fmt.Printf("VersionList %q, folder %q, entry %d, FileInfo deleted mismatch, %v (VersionList) != %v (FileInfo)\n", gk.name, folder, i, deleted, fi.IsDeleted())
if ffi.IsDeleted() != deleted {
fmt.Printf("VersionList %q, folder %q, entry %d, FileInfo deleted mismatch, %v (VersionList) != %v (FileInfo)\n", gk.name, folder, i, deleted, ffi.IsDeleted())
success = false
}
}
for i, fv := range vl.RawVersions {
for i, fv := range vl.Versions {
ver := protocol.VectorFromWire(fv.Version)
for _, device := range fv.Devices {
checkGlobal(i, device, fv.Version, false, fv.Deleted)
checkGlobal(i, device, ver, false, fv.Deleted)
}
for _, device := range fv.InvalidDevices {
checkGlobal(i, device, fv.Version, true, fv.Deleted)
checkGlobal(i, device, ver, true, fv.Deleted)
}
}
@@ -278,10 +282,10 @@ func indexCheck(*cli.Context) (err error) {
if needsLocally(vl) {
_, ok := needs[gk]
if !ok {
fv, _ := vl.GetGlobal()
devB, _ := fv.FirstDevice()
fv, _ := vlGetGlobal(vl)
devB, _ := fvFirstDevice(fv)
dev := deviceToIDs[string(devB)]
fi := fileInfos[fileInfoKey{gk.folder, dev, gk.name}]
fi := protocol.FileInfoFromDB(fileInfos[fileInfoKey{gk.folder, dev, gk.name}])
if !fi.IsDeleted() && !fi.IsIgnored() {
fmt.Printf("Missing need entry for needed file %q, folder %q\n", gk.name, folder)
}
@@ -347,11 +351,84 @@ func indexCheck(*cli.Context) (err error) {
return nil
}
func needsLocally(vl db.VersionList) bool {
gfv, gok := vl.GetGlobal()
func needsLocally(vl *dbproto.VersionList) bool {
gfv, gok := vlGetGlobal(vl)
if !gok { // That's weird, but we hardly need something non-existent
return false
}
fv, ok := vl.Get(protocol.LocalDeviceID[:])
return db.Need(gfv, ok, fv.Version)
fv, ok := vlGet(vl, protocol.LocalDeviceID[:])
return db.Need(gfv, ok, protocol.VectorFromWire(fv.Version))
}
// Get returns a FileVersion that contains the given device and whether it has
// been found at all.
func vlGet(vl *dbproto.VersionList, device []byte) (*dbproto.FileVersion, bool) {
_, i, _, ok := vlFindDevice(vl, device)
if !ok {
return &dbproto.FileVersion{}, false
}
return vl.Versions[i], true
}
// GetGlobal returns the current global FileVersion. The returned FileVersion
// may be invalid, if all FileVersions are invalid. Returns false only if
// VersionList is empty.
func vlGetGlobal(vl *dbproto.VersionList) (*dbproto.FileVersion, bool) {
i := vlFindGlobal(vl)
if i == -1 {
return nil, false
}
return vl.Versions[i], true
}
// findGlobal returns the first version that isn't invalid, or if all versions are
// invalid just the first version (i.e. 0) or -1, if there's no versions at all.
func vlFindGlobal(vl *dbproto.VersionList) int {
for i := range vl.Versions {
if !fvIsInvalid(vl.Versions[i]) {
return i
}
}
if len(vl.Versions) == 0 {
return -1
}
return 0
}
// findDevice returns whether the device is in InvalidVersions or Versions and
// in InvalidDevices or Devices (true for invalid), the positions in the version
// and device slices and whether it has been found at all.
func vlFindDevice(vl *dbproto.VersionList, device []byte) (bool, int, int, bool) {
for i, v := range vl.Versions {
if j := deviceIndex(v.Devices, device); j != -1 {
return false, i, j, true
}
if j := deviceIndex(v.InvalidDevices, device); j != -1 {
return true, i, j, true
}
}
return false, -1, -1, false
}
func deviceIndex(devices [][]byte, device []byte) int {
for i, dev := range devices {
if bytes.Equal(device, dev) {
return i
}
}
return -1
}
func fvFirstDevice(fv *dbproto.FileVersion) ([]byte, bool) {
if len(fv.Devices) != 0 {
return fv.Devices[0], true
}
if len(fv.InvalidDevices) != 0 {
return fv.InvalidDevices[0], true
}
return nil, false
}
func fvIsInvalid(fv *dbproto.FileVersion) bool {
return fv == nil || len(fv.Devices) == 0
}

View File

@@ -8,165 +8,88 @@ package cli
import (
"bufio"
"errors"
"fmt"
"io"
"os"
"strings"
"github.com/alecthomas/kong"
"github.com/flynn-archive/go-shlex"
"github.com/urfave/cli"
"github.com/kballard/go-shellquote"
"github.com/syncthing/syncthing/cmd/syncthing/cmdutil"
"github.com/syncthing/syncthing/lib/config"
)
type preCli struct {
type CLI struct {
cmdutil.CommonOptions
DataDir string `name:"data" placeholder:"PATH" env:"STDATADIR" help:"Set data directory (database and logs)"`
GUIAddress string `name:"gui-address"`
GUIAPIKey string `name:"gui-apikey"`
HomeDir string `name:"home"`
ConfDir string `name:"config"`
DataDir string `name:"data"`
Show showCommand `cmd:"" help:"Show command group"`
Debug debugCommand `cmd:"" help:"Debug command group"`
Operations operationCommand `cmd:"" help:"Operation command group"`
Errors errorsCommand `cmd:"" help:"Error command group"`
Config configCommand `cmd:"" help:"Configuration modification command group" passthrough:""`
Stdin stdinCommand `cmd:"" name:"-" help:"Read commands from stdin"`
}
func Run() error {
// This is somewhat a hack around a chicken and egg problem. We need to set
// the home directory and potentially other flags to know where the
// syncthing instance is running in order to get it's config ... which we
// then use to construct the actual CLI ... at which point it's too late to
// add flags there...
c := preCli{}
parseFlags(&c)
return runInternal(c, os.Args)
type Context struct {
clientFactory *apiClientFactory
}
func RunWithArgs(cliArgs []string) error {
c := preCli{}
parseFlagsWithArgs(cliArgs, &c)
return runInternal(c, cliArgs)
}
func runInternal(c preCli, cliArgs []string) error {
// Not set as default above because the strings can be really long.
err := cmdutil.SetConfigDataLocationsFromFlags(c.HomeDir, c.ConfDir, c.DataDir)
func (cli CLI) AfterApply(kongCtx *kong.Context) error {
err := cmdutil.SetConfigDataLocationsFromFlags(cli.HomeDir, cli.ConfDir, cli.DataDir)
if err != nil {
return fmt.Errorf("Command line options: %w", err)
return fmt.Errorf("command line options: %w", err)
}
clientFactory := &apiClientFactory{
cfg: config.GUIConfiguration{
RawAddress: c.GUIAddress,
APIKey: c.GUIAPIKey,
RawAddress: cli.GUIAddress,
APIKey: cli.GUIAPIKey,
},
}
configCommand, err := getConfigCommand(clientFactory)
if err != nil {
return err
context := Context{
clientFactory: clientFactory,
}
// Implement the same flags at the upper CLI, but do nothing with them.
// This is so that the usage text is the same
fakeFlags := []cli.Flag{
cli.StringFlag{
Name: "gui-address",
Usage: "Override GUI address to `URL` (e.g. \"192.0.2.42:8443\")",
},
cli.StringFlag{
Name: "gui-apikey",
Usage: "Override GUI API key to `API-KEY`",
},
cli.StringFlag{
Name: "home",
Usage: "Set configuration and data directory to `PATH`",
},
cli.StringFlag{
Name: "config",
Usage: "Set configuration directory (config and keys) to `PATH`",
},
cli.StringFlag{
Name: "data",
Usage: "Set data directory (database and logs) to `PATH`",
},
}
// Construct the actual CLI
app := cli.NewApp()
app.Author = "The Syncthing Authors"
app.Metadata = map[string]interface{}{
"clientFactory": clientFactory,
}
app.Commands = []cli.Command{{
Name: "cli",
Usage: "Syncthing command line interface",
Flags: fakeFlags,
Subcommands: []cli.Command{
configCommand,
showCommand,
operationCommand,
errorsCommand,
debugCommand,
{
Name: "-",
HideHelp: true,
Usage: "Read commands from stdin",
Action: func(ctx *cli.Context) error {
if ctx.NArg() > 0 {
return errors.New("command does not expect any arguments")
}
// Drop the `-` not to recurse into self.
args := make([]string, len(cliArgs)-1)
copy(args, cliArgs)
fmt.Println("Reading commands from stdin...", args)
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
input, err := shlex.Split(scanner.Text())
if err != nil {
return fmt.Errorf("parsing input: %w", err)
}
if len(input) == 0 {
continue
}
err = app.Run(append(args, input...))
if err != nil {
return err
}
}
return scanner.Err()
},
},
},
}}
return app.Run(cliArgs)
kongCtx.Bind(context)
return nil
}
func parseFlags(c *preCli) error {
// kong only needs to parse the global arguments after "cli" and before the
// subcommand (if any).
if len(os.Args) <= 2 {
return nil
}
return parseFlagsWithArgs(os.Args[2:], c)
}
type stdinCommand struct{}
func parseFlagsWithArgs(args []string, c *preCli) error {
for i := 0; i < len(args); i++ {
if !strings.HasPrefix(args[i], "--") {
args = args[:i]
break
func (*stdinCommand) Run() error {
// Drop the `-` not to recurse into self.
args := make([]string, len(os.Args)-1)
copy(args, os.Args)
fmt.Println("Reading commands from stdin...", args)
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
input, err := shellquote.Split(scanner.Text())
if err != nil {
return fmt.Errorf("parsing input: %w", err)
}
if !strings.Contains(args[i], "=") {
i++
if len(input) == 0 {
continue
}
var cli CLI
p, err := kong.New(&cli)
if err != nil {
// can't happen, really
return fmt.Errorf("creating parser: %w", err)
}
ctx, err := p.Parse(input)
if err != nil {
fmt.Println("Error:", err)
continue
}
if err := ctx.Run(); err != nil {
fmt.Println("Error:", err)
continue
}
}
// We don't want kong to print anything nor os.Exit (e.g. on -h)
parser, err := kong.New(c, kong.Writers(io.Discard, io.Discard), kong.Exit(func(int) {}))
if err != nil {
return err
}
_, err = parser.Parse(args)
return err
return scanner.Err()
}

View File

@@ -12,48 +12,43 @@ import (
"fmt"
"path/filepath"
"github.com/alecthomas/kong"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/fs"
"github.com/urfave/cli"
)
var operationCommand = cli.Command{
Name: "operations",
HideHelp: true,
Usage: "Operation command group",
Subcommands: []cli.Command{
{
Name: "restart",
Usage: "Restart syncthing",
Action: expects(0, emptyPost("system/restart")),
},
{
Name: "shutdown",
Usage: "Shutdown syncthing",
Action: expects(0, emptyPost("system/shutdown")),
},
{
Name: "upgrade",
Usage: "Upgrade syncthing (if a newer version is available)",
Action: expects(0, emptyPost("system/upgrade")),
},
{
Name: "folder-override",
Usage: "Override changes on folder (remote for sendonly, local for receiveonly). WARNING: Destructive - deletes/changes your data.",
ArgsUsage: "FOLDER-ID",
Action: expects(1, foldersOverride),
},
{
Name: "default-ignores",
Usage: "Set the default ignores (config) from a file",
ArgsUsage: "PATH",
Action: expects(1, setDefaultIgnores),
},
},
type folderOverrideCommand struct {
FolderID string `arg:""`
}
func foldersOverride(c *cli.Context) error {
client, err := getClientFactory(c).getClient()
type defaultIgnoresCommand struct {
Path string `arg:""`
}
type operationCommand struct {
Restart struct{} `cmd:"" help:"Restart syncthing"`
Shutdown struct{} `cmd:"" help:"Shutdown syncthing"`
Upgrade struct{} `cmd:"" help:"Upgrade syncthing (if a newer version is available)"`
FolderOverride folderOverrideCommand `cmd:"" help:"Override changes on folder (remote for sendonly, local for receiveonly). WARNING: Destructive - deletes/changes your data"`
DefaultIgnores defaultIgnoresCommand `cmd:"" help:"Set the default ignores (config) from a file"`
}
func (*operationCommand) Run(ctx Context, kongCtx *kong.Context) error {
f := ctx.clientFactory
switch kongCtx.Selected().Name {
case "restart":
return emptyPost("system/restart", f)
case "shutdown":
return emptyPost("system/shutdown", f)
case "upgrade":
return emptyPost("system/upgrade", f)
}
return nil
}
func (f *folderOverrideCommand) Run(ctx Context) error {
client, err := ctx.clientFactory.getClient()
if err != nil {
return err
}
@@ -61,7 +56,7 @@ func foldersOverride(c *cli.Context) error {
if err != nil {
return err
}
rid := c.Args()[0]
rid := f.FolderID
for _, folder := range cfg.Folders {
if folder.ID == rid {
response, err := client.Post("db/override", "")
@@ -86,12 +81,12 @@ func foldersOverride(c *cli.Context) error {
return fmt.Errorf("Folder %q not found", rid)
}
func setDefaultIgnores(c *cli.Context) error {
client, err := getClientFactory(c).getClient()
func (d *defaultIgnoresCommand) Run(ctx Context) error {
client, err := ctx.clientFactory.getClient()
if err != nil {
return err
}
dir, file := filepath.Split(c.Args()[0])
dir, file := filepath.Split(d.Path)
filesystem := fs.NewFilesystem(fs.FilesystemTypeBasic, dir)
fd, err := filesystem.Open(file)

View File

@@ -9,37 +9,30 @@ package cli
import (
"net/url"
"github.com/urfave/cli"
"github.com/alecthomas/kong"
)
var pendingCommand = cli.Command{
Name: "pending",
HideHelp: true,
Usage: "Pending subcommand group",
Subcommands: []cli.Command{
{
Name: "devices",
Usage: "Show pending devices",
Action: expects(0, indexDumpOutput("cluster/pending/devices")),
},
{
Name: "folders",
Usage: "Show pending folders",
Flags: []cli.Flag{
cli.StringFlag{Name: "device", Usage: "Show pending folders offered by given device"},
},
Action: expects(0, folders()),
},
},
type pendingCommand struct {
Devices struct{} `cmd:"" help:"Show pending devices"`
Folders struct {
Device string `help:"Show pending folders offered by given device"`
} `cmd:"" help:"Show pending folders"`
}
func folders() cli.ActionFunc {
return func(c *cli.Context) error {
if c.String("device") != "" {
func (p *pendingCommand) Run(ctx Context, kongCtx *kong.Context) error {
indexDumpOutput := indexDumpOutputWrapper(ctx.clientFactory)
switch kongCtx.Selected().Name {
case "devices":
return indexDumpOutput("cluster/pending/devices")
case "folders":
if p.Folders.Device != "" {
query := make(url.Values)
query.Set("device", c.String("device"))
return indexDumpOutput("cluster/pending/folders?" + query.Encode())(c)
query.Set("device", p.Folders.Device)
return indexDumpOutput("cluster/pending/folders?" + query.Encode())
}
return indexDumpOutput("cluster/pending/folders")(c)
return indexDumpOutput("cluster/pending/folders")
}
return nil
}

View File

@@ -7,44 +7,36 @@
package cli
import (
"github.com/urfave/cli"
"github.com/alecthomas/kong"
)
var showCommand = cli.Command{
Name: "show",
HideHelp: true,
Usage: "Show command group",
Subcommands: []cli.Command{
{
Name: "version",
Usage: "Show syncthing client version",
Action: expects(0, indexDumpOutput("system/version")),
},
{
Name: "config-status",
Usage: "Show configuration status, whether or not a restart is required for changes to take effect",
Action: expects(0, indexDumpOutput("config/restart-required")),
},
{
Name: "system",
Usage: "Show system status",
Action: expects(0, indexDumpOutput("system/status")),
},
{
Name: "connections",
Usage: "Report about connections to other devices",
Action: expects(0, indexDumpOutput("system/connections")),
},
{
Name: "discovery",
Usage: "Show the discovered addresses of remote devices (from cache of the running syncthing instance)",
Action: expects(0, indexDumpOutput("system/discovery")),
},
pendingCommand,
{
Name: "usage",
Usage: "Show usage report",
Action: expects(0, indexDumpOutput("svc/report")),
},
},
type showCommand struct {
Version struct{} `cmd:"" help:"Show syncthing client version"`
ConfigStatus struct{} `cmd:"" help:"Show configuration status, whether or not a restart is required for changes to take effect"`
System struct{} `cmd:"" help:"Show system status"`
Connections struct{} `cmd:"" help:"Report about connections to other devices"`
Discovery struct{} `cmd:"" help:"Show the discovered addresses of remote devices (from cache of the running syncthing instance)"`
Usage struct{} `cmd:"" help:"Show usage report"`
Pending pendingCommand `cmd:"" help:"Pending subcommand group"`
}
func (*showCommand) Run(ctx Context, kongCtx *kong.Context) error {
indexDumpOutput := indexDumpOutputWrapper(ctx.clientFactory)
switch kongCtx.Selected().Name {
case "version":
return indexDumpOutput("system/version")
case "config-status":
return indexDumpOutput("config/restart-required")
case "system":
return indexDumpOutput("system/status")
case "connections":
return indexDumpOutput("system/connections")
case "discovery":
return indexDumpOutput("system/discovery")
case "usage":
return indexDumpOutput("svc/report")
}
return nil
}

View File

@@ -19,7 +19,6 @@ import (
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/db/backend"
"github.com/syncthing/syncthing/lib/locations"
"github.com/urfave/cli"
)
func responseToBArray(response *http.Response) ([]byte, error) {
@@ -30,68 +29,72 @@ func responseToBArray(response *http.Response) ([]byte, error) {
return bytes, response.Body.Close()
}
func emptyPost(url string) cli.ActionFunc {
return func(c *cli.Context) error {
client, err := getClientFactory(c).getClient()
if err != nil {
return err
}
_, err = client.Post(url, "")
func emptyPost(url string, apiClientFactory *apiClientFactory) error {
client, err := apiClientFactory.getClient()
if err != nil {
return err
}
_, err = client.Post(url, "")
return err
}
func indexDumpOutputWrapper(apiClientFactory *apiClientFactory) func(url string) error {
return func(url string) error {
return indexDumpOutput(url, apiClientFactory)
}
}
func indexDumpOutput(url string) cli.ActionFunc {
return func(c *cli.Context) error {
client, err := getClientFactory(c).getClient()
if err != nil {
return err
}
response, err := client.Get(url)
if errors.Is(err, errNotFound) {
return errors.New("not found (folder/file not in database)")
}
if err != nil {
return err
}
return prettyPrintResponse(response)
}
}
func saveToFile(url string) cli.ActionFunc {
return func(c *cli.Context) error {
client, err := getClientFactory(c).getClient()
if err != nil {
return err
}
response, err := client.Get(url)
if err != nil {
return err
}
_, params, err := mime.ParseMediaType(response.Header.Get("Content-Disposition"))
if err != nil {
return err
}
filename := params["filename"]
if filename == "" {
return errors.New("Missing filename in response")
}
bs, err := responseToBArray(response)
if err != nil {
return err
}
f, err := os.Create(filename)
if err != nil {
return err
}
defer f.Close()
_, err = f.Write(bs)
if err != nil {
return err
}
fmt.Println("Wrote results to", filename)
func indexDumpOutput(url string, apiClientFactory *apiClientFactory) error {
client, err := apiClientFactory.getClient()
if err != nil {
return err
}
response, err := client.Get(url)
if errors.Is(err, errNotFound) {
return errors.New("not found (folder/file not in database)")
}
if err != nil {
return err
}
return prettyPrintResponse(response)
}
func saveToFile(url string, apiClientFactory *apiClientFactory) error {
client, err := apiClientFactory.getClient()
if err != nil {
return err
}
response, err := client.Get(url)
if err != nil {
return err
}
_, params, err := mime.ParseMediaType(response.Header.Get("Content-Disposition"))
if err != nil {
return err
}
filename := params["filename"]
if filename == "" {
return errors.New("Missing filename in response")
}
bs, err := responseToBArray(response)
if err != nil {
return err
}
f, err := os.Create(filename)
if err != nil {
return err
}
_, err = f.Write(bs)
if err != nil {
_ = f.Close()
return err
}
err = f.Close()
if err != nil {
return err
}
fmt.Println("Wrote results to", filename)
return nil
}
func getConfig(c APIClient) (config.Configuration, error) {
@@ -111,19 +114,6 @@ func getConfig(c APIClient) (config.Configuration, error) {
return cfg, nil
}
func expects(n int, actionFunc cli.ActionFunc) cli.ActionFunc {
return func(ctx *cli.Context) error {
if ctx.NArg() != n {
plural := ""
if n != 1 {
plural = "s"
}
return fmt.Errorf("expected %d argument%s, got %d", n, plural, ctx.NArg())
}
return actionFunc(ctx)
}
}
func prettyPrintJSON(data interface{}) error {
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
@@ -159,7 +149,3 @@ func nulString(bs []byte) string {
func normalizePath(path string) string {
return filepath.ToSlash(filepath.Clean(path))
}
func getClientFactory(c *cli.Context) *apiClientFactory {
return c.App.Metadata["clientFactory"].(*apiClientFactory)
}

Some files were not shown because too many files have changed in this diff Show More