* Show proper subcommand prefix in generated config CLI.
* Remove useless author info and copy command group description.
* Really accept (implicit) -h and --help flags.
These were disabled by HideHelp, leading to an error message in every
usage output. This way, the flags get documented as well.
* Override AppHelpTemplate to better match Kong's style.
* Override (Sub)commandHelpTemplate to better match Kong's style.
* Use <command> and [flags] like Kong.
Signed-off-by: André Colomb <src@andre.colomb.de>
Based on user requests from Weblate:
* `@miryusifrahimov` for Azerbaijani
* `@halbast` für Kurdish (Central)
Both seem to be legit and have previously contributed translations on
Weblate.
Signed-off-by: André Colomb <src@andre.colomb.de>
In #9701 there was a change that put the mutex used for `getExpireAdd` directly in `defaultRealCaser`, which is erroneous because multiple filesystems can share the same `caseCache`.
### Purpose
Fixes#9836 and [Slow sync sending files from Android](https://forum.syncthing.net/t/slow-sync-sending-files-from-android/24208?u=marbens). There may be other issues caused by `getExpireAdd` conflicting with itself, though.
### Testing
Unit tests pass and the case cache and conflict detection _seem_ to behave correctly.
Signed-off-by: Marcus B Spencer <marcus@marcusspencer.us>
This replaces `allow_contributor` with `allow_non_author_contributor`,
because the former allows authors to approve their own pull requests.
From https://github.com/syncthing/syncthing/pull/9818#issue-2651431707:
> This adds `allow_contributor: true` which allows approvals by contributors to the PR (*but still not the author themself*, which is a different thing).
This statement conflicts with [the policybot README](c013552248/README.md), which says:
> If true, the approvals of someone who has committed to the pull request are
> considered when calculating the status.
> *The pull request author is considered a contributor.*
Signed-off-by: Marcus B Spencer <marcus@marcusspencer.us>
Due to a thinko, this optimisation was wildly incorrect and would read
to lack of block reuse when syncing files.
(We do not insert a blocklist per device, but only a single one. We
can't use the fact of whether the insert happened as a criteria for
inserting blocks.)
Signed-off-by: Jakob Borg <jakob@kastelo.net>
IMHO the logic here was inverted. The only use for the report data is to
show a preview when we ask the user whether they want to participate in
usage reporting. However, the GUI would first load the report data and
then consider whether we wanted to show that dialog or not. Instead,
only load if it we're going to show the dialog.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
This adds a new field to the file information we keep, the "previous
blocks hash". This is the hash of the file contents as it was in its
previous incarnation. That is, every scan that updates the blocks hash
will move the current hash to the "previous" field.
This enables an addition to the conflict detection algorithm: if the
file to be synced is in conflict with the current file on disk
(version-counter wise), but it indicates that it was based on the
precise contents we have (new.prevBlocksHash == current.blocksHash),
then it's not really a conflict.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
This adds a rule where a maintainer can merge a PR on their own even in
non-trivial cases, which we (I) already do as-is today, but by breaking
the rules instead of following them. This just codifies that behavior.
If we get to a point where it's no longer necessary, that'd be cool.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
This changes the files table to use normalisation for the names and
versions. The idea is that these are often common between all remote
devices, and repeating an integer is more efficient than repeating a
long string. A new benchmark bears this out; for a database with 100k
files shared between 31 devices, with some worst case assumption on
version vector size, the database is reduced in size by 50% and the test
finishes quicker:
Current:
db_bench_test.go:322: Total size: 6263.70 MiB
--- PASS: TestBenchmarkSizeManyFilesRemotes (1084.89s)
New:
db_bench_test.go:326: Total size: 3049.95 MiB
--- PASS: TestBenchmarkSizeManyFilesRemotes (776.97s)
The other benchmarks end up about the same within the margin of
variability, with one possible exception being that RemoteNeed seems to
be a little slower on average:
old files/s new files/s
Update/n=RemoteNeed/size=1000-8 5.051k 4.654k
Update/n=RemoteNeed/size=2000-8 5.201k 4.384k
Update/n=RemoteNeed/size=4000-8 4.943k 4.242k
Update/n=RemoteNeed/size=8000-8 5.099k 3.527k
Update/n=RemoteNeed/size=16000-8 3.686k 3.847k
Update/n=RemoteNeed/size=30000-8 4.456k 3.482k
I'm not sure why, possibly that query can be optimised anyhow.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
We have a slightly naive io.ReadAll on the authentication handler, which
can result in unlimited memory consumption from an unauthenticated API
endpoint. Add a reasonable limit there.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
Remove the migrated v0.14.0 database format after two weeks. Remove a
few old patterns that are no longer relevant. Ensure the cleanup runs in
both the config and database directories.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
Store the sequence number of the last GC sweep in a KV. Next time, if it
matches we can just skip GC because nothing has been added or removed.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
There are some return statements in between, but putting back the values
isn't my motivation (hardly ever happens), I just find this more
readable. Same with moving `hashLength`: Placed next to the pool the
connection with `sha256.New()` is closer.
Followup to:
chore(scanner): reduce memory pressure by using pools inside hasher #102226e26fab3a0
Signed-off-by: Simon Frei <freisim93@gmail.com>
Periodic garbage collection can take a long time on large folders. The worst
step is the one for blocks, which are typically orders of magnitude more
numerous than files or block lists.
This improves the situation in by running blocks GC in a number of smaller
range chunks, in random order, and stopping after a time limit. At most ten
minutes per run will be spent garbage collecting blocklists and blocks.
With this, we're not guaranteed to complete a full GC on every run, but
we'll make some progress and get there eventually.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
No migration on this as it has no practical impact, just a slight
cleanup for new installations.
Also a refactor of how we declare single column primary keys, for
consistency.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
This adds several options for configuring the log format of timestamps
and severity levels, making it more suitable for integration with log
systems like systemd.
--log-format-timestamp="2006-01-02 15:04:05"
Format for timestamp, set to empty to disable timestamps ($STLOGFORMATTIMESTAMP)
--[no-]log-format-level-string
Whether to include level string in log line ($STLOGFORMATLEVELSTRING)
--[no-]log-format-level-syslog
Whether to include level as syslog prefix in log line ($STLOGFORMATLEVELSYSLOG)
So, to get a timestamp suitable for systemd (syslog prefix, no level
string, no timestamp) we can pass `--log-format-timestamp=""
--no-log-format-level-string --log-format-level-syslog` or,
equivalently, set `STLOGFORMATTIMESTAMP="" STLOGFORMATLEVELSTRING=false
STLOGFORMATLEVELSYSLOG=true`.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
No practical effect, just a tiny bit of fun to stamp the database files
with an application ID that identifies them.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
While we're figuring out optimal defaults, reduce the page cache size to
the compiled-in default. In my computer this makes no difference in
benchmarks. In forum threads, it solved the problem of massive memory
usage during initial scan.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
* fix(api): redact device encryption passwords in support bundle config
Signed-off-by: Tommy van der Vorst <tommy@pixelspark.nl>
* Update lib/api/support_bundle.go
Signed-off-by: Jakob Borg <jakob@kastelo.net>
---------
Signed-off-by: Tommy van der Vorst <tommy@pixelspark.nl>
Signed-off-by: Jakob Borg <jakob@kastelo.net>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
Since #10332 we'd create the temp file when closing out the puller state
for a file, but this is inappropriate if the reason we're bailing out is
that there isn't space for it to begin with. Instead, do the
free space check before we even start copying/pulling.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
Removes the chitter-chatter of folder state changes from the info level,
while adding the error state at warning level and a corresponding
clearing of the error state at info level.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
This reduces database migration memory usage in my test scenario from
3.8 GB to 440 MB. In principle I don't think we're causing many temp
tables to be generated anyway in normal usage, but if we do and someone
can benchmark a performance difference, we can add a tunable. I ran the
database benchmark before and after and didn't see a difference above
the noise level.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
This copies the relevant parts of the contribution guidelines in the
docs, for the purpose of keeping them in a single place. The in-docs
contribution guidelines can become a link to this document.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
When handling files that consist only of power-of-two-sized blocks of
zero we'd know we have nothing to write, and when using sparse files
we'd never even create the temp file. Hence the sync would fail.
Signed-off-by: Jakob Borg <jakob@kastelo.net>
Just to be entirely sure that if the migration succeeds the schema
version is always also updated. Currently if a migration succeeds but a
later migration doesn't, the changes of the migration apply but the
version stays - if the migration is breaking/non-idempotent, it will
fail when it tries to rerun it next time (otherwise it's just a
pointless re-execution).
Unfortunately with the current `db.runScripts` it wasn't that easy to
do, so I had to do quite a bit of refactoring. I am also ensuring the
right order of transactions now, though I assume that was already the
case lexicographically - can't hurt to be safe.
Currently, the input field has no step defined, meaning that it can be
increased with the arrow keys by the default value of "1". Considering
the fact that the default value is "3600" (seconds or one hour), it is
unlikely that the user wants to change it with such minimal steps.
For this reason, change the default step to "3600" (one hour). If the
user needs more granual control, they can still input the value
in seconds manually.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Currently, the bandwidth limit input fields have no step defined, and as
such they use the default value of "1". Taking into account the fact
that these fields use KiB as their measurements, it makes more sense to
use larger steps, such as "1024" (1 MiB), as in most cases, it is very
unlikely that the user needs to have byte-level control over the limits.
Note that these steps only apply to increasing the values by using the
arrow keys, and the user is still allowed to input any value they want
manually.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This just removes an unnecessary foreign key constraint, where we
already do the garbage collection manually in the database service.
However, as part of getting here I tried a couple of other variants
along the way:
- Changing the order of the primary key from `(hash, blocklist_hash,
idx)` to `(blocklist_hash, idx, hash)` so that inserts would be
naturally ordered. However this requires a new index `on blocks (hash)`
so that we can still look up blocks by hash, and turns out to be
strictly worse than what we already have.
- Removing the primary key entirely and the `WITHOUT ROWID` to make it a
rowid table without any required order, and an index as above. This is
faster when the table is small, but becomes slower when it's large (due
to dual indexes I guess).
These are the benchmark results from current `main`, the second
alternative below ("Index(hash)") and this proposal that retains the
combined primary key ("combined"). Overall it ends up being about 65%
faster.
<img width="764" height="452" alt="Screenshot 2025-08-29 at 14 36 28"
src="https://github.com/user-attachments/assets/bff3f9d1-916a-485f-91b7-b54b477f5aac"
/>
Ref #10264
Currently, the number of hashers is always set to 1 on interactive
operating systems, which are defined as Windows, macOS, iOS, and
Android. However, with modern multicore CPUs, it does not make much
sense to limit performance so much.
For example, without this fix, a CPU with 16 cores / 32 threads is
still limited to using just a single thread to hash files per folder,
which may severely affect its performance.
For this reason, instead of using a fixed value, calculate the number
dynamically, so that it equals one-fourth of the total number of CPU
cores. This way, the value of hashes will still end up being just 1 on
a slower 4-thread CPU, but it will be allowed to take larger values when
the number of threads is higher, increasing hashing performance in the
process.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
Currently, the number of hashers, with the exception of some specific
operating systems or when defined manually, equals the number of CPU
cores divided by the overall number of folders, and it does not take
into account the value of MaxFolderConcurrency at all. This leads to
artificial performance limits even when MaxFolderConcurrency is set to
values lower than the number of cores.
For example, let's say that the number of folders is 50 and
MaxFolderConcurrency is set a value of 4 on a 16-core CPU. With the old
calculation, the number of hashers would still end up being just 1 due
to the large number of folders. However, with the new calculation, the
number of hashers in this case will be 4, leading to better hashing
performance per folder.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This adds a cleanup stage to remove database files for folders that no
longer exist on startup. Folder database files were already removed when
dropping a folder, assuming that the folder database had been opened at
that point. This won't be the case though when a folder is removed from
the config when Syncthing isn't running, or when a folder is dropped and
re-migrated in a restarted migration.
This adds a temporary GUI/API server during the database migration. It
responds with 200 OK and some log output for every request. This serves
two purposes:
- Primarily, for deployments that use the API as a health check, it
gives them something positive to accept during the migration, reducing
the risk of the migration getting killed halfway through and restarted,
thus never completing.
- Secondarily, it gives humans who happen to try to load the GUI some
sort of indication of what's going on.
Obviously, anything that expects a well-formed API response at this
stage is still going to fail. They were already failing though, as we
didn't even listen at this point before.
Two things:
- We could run into a write error, which would block the progress
forever without an error. This because the writer routine exited, while
the reader was just blocked on sending to it.
- After a failed migration, inserts could fail with unique index
constraint errors because we are reusing the sequence numbers from the
original database. Add a drop folder to the start of migration to handle
this.
Additionally, the drop folder will clear out broken database files due
to killed migrations.
### Purpose
This was lost / replaced when introducing the "version" command.
However, the documentation still lists the flag - actually under the
serve command, but that can be omitted. Common convention for CLI
programs is to accept it as a flag.
### Testing
```
$ bin/syncthing --help
Usage: syncthing <command> [flags]
Flags:
-h, --help Show context-sensitive help.
-C, --config=PATH Set configuration directory (config and keys) ($STCONFDIR)
-D, --data=PATH Set data directory (database and logs) ($STDATADIR)
-H, --home=PATH Set configuration and data directory ($STHOMEDIR)
--version Show current version, then exit
Commands:
serve Run Syncthing (default)
cli Command line interface for Syncthing
browser Open GUI in browser, then exit
decrypt Decrypt or verify an encrypted folder
device-id Show device ID, then exit
generate Generate key and config, then exit
paths Show configuration paths, then exit
upgrade Perform or check for upgrade, then exit
version Show current version, then exit
debug Various debugging commands
install-completions Print commands to install shell completions
Run "syncthing <command> --help" for more information on a command.
```
```
$ bin/syncthing --version
syncthing v2.0.3-dev.2.g0f47e944-restore-version-flag "Hafnium Hornet" (go1.24.0 linux-amd64) acolomb@riddo 2025-08-18 19:25:31 UTC
```
### Documentation
Already / *still* listed in the docs under Command Line Operation.
Avoid:
/_/GOROOT/src/os/user/cgo_lookup_cgo.go:45:(.text+0x54): warning: Using
'getgrgid_r' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
and
/tmp/go-build/cgo-gcc-prolog:60:(.text+0x40): warning: Using
'getaddrinfo' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
I added these tags as part of the big database PR, but I forget why. I
think it came from an attempt at a static binary using the Go-based
SQLite packages, but that's not the primary build anymore anyway. We can
remove this and go back to the standard resolvers, which gives better
support for primarily Windows and macOS special resolution methods...
chore(gui): remove redundant "authenticated" conditions from Actions
menu (#10235)
Due to previous code changes, the whole Actions menu is only available
when the user is logged in. As such, there is no reason to have the same
ng-if="authenticated" condition repeated in other items belonging to it.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This removes the `debugging` bool under GUI configuration, and two no
longer relevant development endpoints: `httpmetrics` (which I can't
imagine anyone using for anything -- if we need such metrics today, the
right place is the Prometheus exported metrics) and the `peerCompletion`
endpoint (previously used by integration tests).
The debugging bool initially enabled just those two endpoints, which are
not for end users. Then we added profiling and support bundles, which
are very useful indeed for end users to access, and they were hidden
behind the same debug flag. I don't see any reason for keeping that flag
now that these methods are more generally useful.
https://github.com/syncthing/docs/pull/949
This updates our logging framework from legacy freetext strings using
the `log` package to structured log entries using `log/slog`. I have
updated all INFO or higher level entries, but not yet DEBUG (😓)... So,
at a high level:
There is a slight change in log levels, effectively adding a new warning
level:
- DEBUG is still debug (ideally not for users but developers, though
this is something we need to work on)
- INFO is still info, though I've added more data here, effectively
making Syncthing more verbose by default (more on this below)
- WARNING is a new log level that is different from the _old_ WARNING
(more below)
- ERROR is what was WARNING before -- problems that must be dealt with,
and also bubbled as a popup in the GUI.
A new feature is that the logging level can be set per package to
something other than just debug or info, and hence I feel that we can
add a bit more things into INFO while moving some (in fact, most)
current INFO level warnings into WARNING. For example, I think it's
justified to get a log of synced files in INFO and sync failures in
WARNING. These are things that have historically been tricky to debug
properly, and having more information by default will be useful to many,
while still making it possible get close to told level of inscrutability
by setting the log level to WARNING. I'd like to get to a stage where
DEBUG is never necessary to just figure out what's going on, as opposed
to trying to narrow down a likely bug.
Code wise:
- Our logging object, generally known as `l` in each package, is now a
new adapter object that provides the old API on top of the newer one.
(This should go away once all old log entries are migrated.) This is
only for `l.Debugln` and `l.Debugf`.
- There is a new level tracker that keeps the log level for each
package.
- There is a nested setup of handlers, since the structure mandated by
`log/slog` is slightly convoluted (imho). We do this because we need to
do formatting at a "medium" level internally so we can buffer log lines
in text format but with separate timestamp and log level for the API/GUI
to consume.
- The `debug` API call becomes a `loglevels` API call, which can set the
log level to `DEBUG`, `INFO`, `WARNING` or `ERROR` per package. The GUI
is updated to handle this.
- Our custom `sync` package provided some debugging of mutexes quite
strongly integrated into the old logging framework, only turned on when
`STTRACE` was set to certain values at startup, etc. It's been a long
time since this has been useful; I removed it.
- The `STTRACE` env var remains and can be used the same way as before,
while additionally permitting specific log levels to be specified,
`STTRACE=model:WARN,scanner:DEBUG`.
- There is a new command line option `--log-level=INFO` to set the
default log level.
- The command line options `--log-flags` and `--verbose` go away, but
are currently retained as hidden & ignored options since we set them by
default in some of our startup examples and Syncthing would otherwise
fail to start.
Sample format messages:
```
2009-02-13 23:31:30 INF A basic info line (attr1="val with spaces" attr2=2 attr3="val\"quote" a=a log.pkg=slogutil)
2009-02-13 23:31:30 INF An info line with grouped values (attr1=val1 foo.attr2=2 foo.bar.attr3=3 a=a log.pkg=slogutil)
2009-02-13 23:31:30 INF An info line with grouped values via logger (foo.attr1=val1 foo.attr2=2 a=a log.pkg=slogutil)
2009-02-13 23:31:30 INF An info line with nested grouped values via logger (bar.foo.attr1=val1 bar.foo.attr2=2 a=a log.pkg=slogutil)
2009-02-13 23:31:30 WRN A warning entry (a=a log.pkg=slogutil)
2009-02-13 23:31:30 ERR An error (a=a log.pkg=slogutil)
```
---------
Co-authored-by: Ross Smith II <ross@smithii.com>
Based on the discussion in
https://forum.syncthing.net/t/towards-syncthing-2-0/24072/35 This PR
adds the ability for Windows users to use the pipe character (|) to
escape the metacharacters *, ?, [, and { in .stignore files.
Additionally, this PR adds the ability for the user to set the escape
character to backslash, or any character they want, by adding a line in
the form:
#escape=X
(where X is any single rune), to the top of an .stignore file.
This would allow users to use the same .stignore file across platforms,
by simply adding
#escape=\
to the top of the file.
### Testing
All tests pass in CI.
### Documentation
See https://github.com/syncthing/docs/pull/919Fixes#10057: Support escaping in .stignore files on Windows
Fixes#7547: Ignore pattern with \[ and \] does not work
Prior to this fix, the folder would only get marked as "syncing" once we
started downloading data from the network. However in some cases there
will be a lot of data that can be reused locally and we spend
significant time copying blocks before downloading anything; in that
case, the folder would appear as "preparing to sync" while it was in
fact moving lots of data.
This fixes that, making it "syncing" as soon as it begins either copying
or downloading data.
### Purpose
Uses recommended pattern for slice pools to avoid copying the slice
struct, suggested by the linter and actually used in the go stdlib, for
example in `net/http/h2_bundle.go`.
Currently, the Revert Local Changes button for Receive Only folders, and
the Delete Unexpected Items button for Receive Encrypted folders buttons
are shown even when the folder is already performing other operations.
Because of the above, pressing the button seems to have no effect, as
its operation can only proceed after the previous operations have
completed. This confuses the user, who then may keep trying to press the
buttons again and again with no visible result.
Therefore, show the two buttons only when the folders are actually idle,
without performing other operations at the same time. This change makes
them behave similarly to the Override Changes button, which is also only
displayed for Send Only folders when they are idle.
Signed-off-by: Tomasz Wilczyński twilczynski@naver.com
Update the jQuery Fancytree Plugin to the newest version. Apart from
keeping it up-to-date out of principle, this may also help with further
investigation of issue #10155, which is related to the plugin.
Signed-off-by: Tomasz Wilczyński twilczynski@naver.com
chore(gui): Fix "Shut Down" spelling in Actions
Currently, the word is written as "Shutdown". However, similarly to
"Log Out", it is used as a verb here, thus it should be written as two
separate words, i.e. "Shut Down".
Signed-off-by: Tomasz Wilczyński twilczynski@naver.com
Add a wrapper that uses anet on Android, but net on other platforms.
### Purpose
Fixes
https://forum.syncthing.net/t/workaround-for-android-local-discovery/20403/12
### Testing
Run two Syncthing instances with Global Discovery disabled. Pair them
with each other, don't hardcode their addresses, and verify they
connect.
We've always, since the introduction of conflicts, had the policy that
deletes lose against any other change, for safety's sake. This is a
problem, however, because it means the sort order of versions is not a
total order.
That is, given two versions `A` and `B` that are currently in conflict,
we will sort them in a given order (let's say `A, B`, so `A < B` for
ordering purposes: we say "A wins over B" or "A is newer than B") and
consider the first in the list the winner. The loser (who has `B` on
disk) will process the conflict at some point and move the file to a
conflict copy and announce `A'` as the resolved conflict. The winner
(with `A` on disk) doesn't do anything.
However, if `A` is deleted the ordering changes. We still have `A < B`
and, of course, `Adel < A` (this is not even a conflict, just linear
order). In most sane systems this would imply the ordering `Adel < A <
B`, however in our case we in fact have `B < Adel` because any version
wins over a deleted one, so there is no logical ordering at all of the
files at this point. `Adel < A < B < Adel ???` In practice the deleted
version may end up at the head or the tail of the list, depending on the
order we do the compares.
Hence, at this point, "whatever" happens and it's not guaranteed to make
any sense. 😬
I propose that we resolve this my simply letting deletes be versions
like anything else and maintain a total ordering based on just version
vectors with the existing tie breakers like always. That means a delete
can win in a conflict situation, and the result should be that the file
is moved to a conflict copy on the losing device. I think this retains
the data safety to almost the same degree as previously, while removing
probably an entire class of strange out of sync bugs...
---
(A potential wrinkle here is that, ideally, we wouldn't even create the
conflict copy when the delete and the losing version represent the same
data -- same as when we handle normal modification conflicts. However,
the deleted FileInfo doesn't carry any information on what the contents
were, so we can't do that right now. A possible future extension would
be to carry the block list hash of the deleted data in the deleted
FileInfo and use that for this purpose, but I don't want to complicate
this PR with that. The block list hash itself also isn't a
protocol-defined thing at the moment, it's something implementation
dependent that we just use locally.)
### Purpose
As discussed on the forum:
https://forum.syncthing.net/t/reviving-nat-pmp-in-v2-x-on-android-14/24554
TL;DR
Android 14+ only lets java code get the gateway IPv4 address which is
used in SyncthingNative’s NAT-PMP feature.
So I’ve added the java code to the wrapper, got the router IP address
and feeded it to SyncthingNative by setting the env var
“FALLBACK_NET_GATEWAY_IPV4”.
This revives the NAT feature:
> [Z36WU] INFO: Detected 1 NAT service
### Testing
Local build and test via Android emulator (AVD 15).
This makes a couple of backwards compatible changes to the
ClusterConfig:
- Remove the `ignore_permissions` and `ignore_delete` booleans which
we've never read or used for anything
- Remove the `disable_temp_indexes` boolean and option entirely. We did
use this one, and about 1% of users have set the option. The only thing
it does is inhibits sending of periodical DownloadProgress messages
while downloading data, which is a minuscule bandwidth optimisation
given that we're already sending data at the time.
- Change the `read_only` boolean (which indicated send-only folders) to
an enum `FolderType`, where the values zero and one match the existing
usage. Again, we don't actually use this value, but I can see that we
might want to and then it makes more sense for it to be more
comprehensive.
- Change the `paused` boolean to an enum `StopReason`, where zero
indicates not stopped and one indicates paused, exactly the same wire
representation as previously but leaves space for additional stop
reasons (errors etc).
### Purpose
Filesystem watcher errors didnt have any whitespace between the share
name and the error message, making it hard to read. A simple colon and
whitespace solves this issue
I lately wanted some photos on my phone, and watched them sync
excrutiatingly slowly. I am used to android being slow, but not that
slow. This restriction caught my eye and I increased it beyond the
limit (didn't spot it at first), and I did see a clear improvement. Of
course as always with such a one-off test, I might also have
hallucinated it, but it seems plausible with the slow thing in android
being some layer between the actual filesystem and apps.
Also increase the max limit, mostly just because I don't see any reason
to restrict it that low - not that I have a particular reason to want
more.
I also changed the xml default to 0: The `prepare` code will change it
to the actual default - no need to change that anymore if we change the
default in the future.
Before:
- Local discovery on Android 10+ is broken. The phone receives local
discovery packets from other devices running Syncthing on the same
network, e.g. a computer. But it doesn't send its own local discovery
packets.
- Startup of the beacon/broadcast.go and beacon/multicast.go "services"
subsequently fail, see the log entries of "service.go" with "2 of 2
failures, backing off".
Root cause:
- Android 10+ restricts determining the network interfaces for privacy
reasons. The interfaces and IP addresses cannot be determined.
- There's a bug in the go "net" library. I can actually get the
interfaces, but the fix was not implemented by the go team.
Workaround:
- The "community" found a workaround by creating a light wrapper around
"net" called "anet" library.
- "anet" adjusts the behaviour on Android 10+ and gets the interfaces
plus their IP addresses, as required by Syncthing.
After:
- By using the "anet" lib, Syncthing is able to get the interface ip
addresses and put them into the "AllAddresses" string array.
- The "AllAddresses" string array is then announced on the local
discovery multicast and broadcast packets, if enabled in Syncthing's
config.
- By correctly getting the interfaces and IP addresses using "anet" in
"beacon/broadcast.go" and "beacon/multicast.go", the services start up
fine again.
Verification:
- I've built "libSyncthingNative.so" with this PR applied for Android
and put it into Syncthing-Fork v1.29.7.5 for testing. My two phones,
Android 10 and Android 15 (arm64-v8a) immediately discovered each other
using local discovery.
- I can see the "sent XX bytes" and "recv XX bytes" on both phones in
the log filtering for "SyncthingNativeCode" :-).
Personal note:
- Please go light on me, and, if it's not demanded too much of your
time, please help me on this. I am no go programmer. Most things you
think are easy or common sense aren't part of my knowledge set. I'd just
like to help and hope we somehow can drive this home together to fix the
problem.
----
ref: https://github.com/Catfriend1/syncthing-android/pull/1501
ref: https://github.com/Catfriend1/syncthing-android/issues/1500
ref: https://github.com/wlynxg/anet/blob/main/interface.go &
https://github.com/wlynxg/anet/blob/main/interface_android.go
With that fix, I can see the broadcast/multicast lines again and my
phone can be discovered by other phones running the Syncthing app which
wasn't possible before on Android 10+.
```
[ET76H] .346892 broadcast.go:107: DEBUG: sent 185 bytes to 192.168.x.255:21027
[ET76H] .347114 multicast.go:86: DEBUG: sent 185 bytes to [ff12::8384]:21027 on wlan0
```
---------
Co-authored-by: Marcus B Spencer <marcus@marcusspencer.us>
Fixes#7403.
Tested by enabling UPnP on the router, and checking on the router page
that the external ports of the UDP mappings match what is shown in the
logs and the internal ports matching the QUIC listening port.
Only Require either matching UID & GID OR matching Names.
If the 2 devices have a different Name => UID mapping, they can never be
totaly equal. Therefore when syncing we try matching the Name and fall
back to the UID. However when scanning for changes we currently require
both the Name & UID to match. This leads to forever having out of sync
files back and forth, or local additions when receive only.
This patch does not change the sending behavoir. It only change what we
decide is equal for exisiting files with mismapped Name => UID,
The added testcases show the change: Test 1,5,6 are the same as current.
Test 2,3 Are what change with this patch (from false to true). Test 4 is
a subset of test 2 they is currently special cased as true, which does
not chnage.
Co-authored-by: Jakob Borg <jakob@kastelo.net>
While it doesn't hurt, it's unnecessary since the big protobuf
modernisation, that also introduced types separate from the generated
ones for internal use. Those fields are already dropped when converting
to the wire in protocol.
This updates our key generation to use Ed25519 keys/certificates for
sync connections. Certificates for browser use remain ECDSA for wider
compatibility.
Ed25519 is more modern and has fewer concerns for the future than the
ECDSA curves we used previously. It is supported from Go 1.13 and
forwards, which is Syncthing 1.3.0 (October 2019).
### Purpose
Locally, on Windows 11, and on the windows-2025 GitHub runner (go 1.23
and 1.24), the `TestCopyRange` test is failing with `The request is not
supported.`
On windows-2022 and windows-2019:
```go
err == syscall.ENOTSUP
```
worked, but on Windows 11 and windows-2025, we need:
```go
errors.Is(err, errors.ErrUnsupported)
```
### Testing
Tested on Windows 11, windows-2019, windows-2022, and
[windows-2025](https://github.com/rasa/syncthing/actions/runs/15525123437/job/43703630634#step:7:2811).
* v2: (62 commits)
build: add dependency for next-version script
docs: add release note mention of platforms no longer built
build: streamline gathering of facts, checkouts (#10158)
build: more resilient pushes to releases
chore(etc): add option dash to upstart config
chore(fs): linter complaints
chore(model): the easier linter complaints
chore(internal): linter complaints
chore(sqlite): linter complaints
build: allow v2 into APT candidate channel
docs: link to Docker image, APT, in release notes
build: refactor builds for forks/PRs
build: use same GitHub token for releases as for translation etc pushes
refactor(sqlite): move deleted flag into logical order in schema
feat(config): enable multiple connections by default (#10151)
docs: mention subcommands in release notes, use for all 2.0 releases
docs: adjust release notes for v2.0.0
docs: add relnotes for v2.0.0
build: upgrade setup-zig action (#10134)
fix(versioner): correct fs creation in test
...
* main:
docs: link to Docker image, APT, in release notes
build: also create relaysrv and discosrv releases
fix(stupgrades): return latest stable & pre for each major
fix(syncthing): avoid writing panic log to nil fd (#10154)
### Purpose
This change fixes a logical bug in the panic log writing where we could
end up writing to a uninitialized file descriptor.
On the very first iteration, `panicFd` is nil. We enter the if `panicFd
== nil { … }` block, check for “panic:” or “fatal error:”, and if
neither matches, we skip instantiating `panicFd` altogether. However,
immediately after, still within `if panicFd == nil { … }`, we call
`panicFd.WriteString("Panic at ...")`. But `panicFd` would in this case
be `nil`, which will cause a run‐time panic.
It's not clear to me why panicFd is only initialized if the lines start
with "panic:" or "fatal error:" so I've left that logic untouched. With
this change we at least avoid the risk of writing to a nil
filedescriptor.
## Authorship
Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
* main:
feat(config): expose folder and device info as metrics (fixes#9519) (#10148)
chore: add issue types to GitHub issue templates
build: remove schedule from PR metadata job
chore(protocol): only allow enc. password changes on cluster config (#10145)
chore(protocol): don't start connection routines a second time (#10146)
Tihs makes it easier to use metrics based on device and folder labels,
names, and other attributes. Other metrics which are based on folder or
device ID can be joined with these info metrics to enrich their label
sets.
```
# HELP syncthing_config_device_info Provides additional information labels on devices
# TYPE syncthing_config_device_info gauge
syncthing_config_device_info{device="I6KAH76-66SLLLB-5PFXSOA-UFJCDZC-YAOMLEK-CP2GB32-BV5RQST-3PSROAU",introducer="false",name="s1",paused="false",untrusted="false"} 1
# HELP syncthing_config_folder_info Provides additional information labels on folders
# TYPE syncthing_config_folder_info gauge
syncthing_config_folder_info{folder="default",label="The default folder",path="s2",paused="false",type="sendreceive"} 1
```
With this you can e.g. query for
```
syncthing_connections_active * on(device) group_left syncthing_config_device_info
```
Fixes#9519Closes#10074Closes#10147
This changes the default number of connections from one to three (one
metadata + two data connections). This should give some advantages of
multiple connections, while also not being an overwhelming change for
larger installations. (Though those may need to tweak their settings
anyway, as always.)
In practice we already always call SetPassword and ClusterConfig
together. However it's not just "sensible" to do that, it's required: If
the passwords change, the remote device needs to know about that to
check that the enc. setup is valid/consistent (e.g. tokens match,
folder-type is appropriate, ...).
And with the passwords set later, there's no point in adding them as
part of creating a new connection.
This is a "followup" (if one can call it that 4 years later :) ) to
resp. fix for the following commit:
924b96856f
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This adds a file that will be prepended to release notes (tag messages,
GitHub releases, forum posts) for v1 releases. I'd like there to be
something there to flag that things are going to change.
* main:
refactor: use slices package for sorting (#10136)
build: handle multiple general release notes
build: no need to build on the branches that just trigger tags
* main:
build: use specific token for pushing release tags
fix(gui): update `uncamel()` to handle strings like 'IDs' (fixes#10128) (#10131)
refactor: use slices package for sort (#10132)
build: process for automatic release tags (#10133)
chore(gui, man, authors): update docs, translations, and contributors
> ⚠️ resubmission targeting `main` instead of `v2`
### Purpose
Updates `uncamel()` function in
[uncamelFilter.js](https://github.com/syncthing/syncthing/blob/v2/gui/default/syncthing/core/uncamelFilter.js)
to fix camelCase conversion edge cases, see #10128
This adds an array called `reservedStrings` which will be printed as-is,
e.g. `IDs`, `LAN` etc. I pre-populated this with what I believe makes
sense, but of course this is easily updated.
### Testing
I compiled all the config variables I could find in
`syncthing/lib/config/*configuration.go` and tested this new function
against them. Everything seemed to pass.
### Screenshot
The sort package is still used in places that were not trivial to
change. Since Go 1.21 slices package can be uswed for sort. See
https://go.dev/doc/go1.21#slices
### Purpose
Make some progress with the migration to a more up-to-date syntax.
* main:
fix(syncthing): ensure both config and data dirs exist at startup (fixes#10126) (#10127)
fix(versioner): fix perms of created folders (fixes#9626) (#10105)
refactor: use slices.Contains to simplify code (#10121)
As suggested in the linked issue, I've updated the versioner code to use
the permissions of the corresponding directory in the synced folder,
when creating the folder in the versions directory
### Testing
- Some tests are included with the PR. Happy to add more if you think
there are some edge-cases that we're missing.
- I've tested manually on linux to confirm the permissions of the
created directories.
- I haven't tested on Windows or OSX (I don't have access to these OS)
The copier routine refactor resulted in bad buffer pool handling,
putting a buffer back into the pool twice. This simplifies and removes
the danger prone Upgrade() method.
### Testing
Change the `auditEnabled` option and you should get a prompt in the Web
GUI.
Restart and change the `auditFile` option, and you should get that same
prompt.
The prompt you should get is shown in the screenshots below.
### Screenshots
Co-authored-by: Jakob Borg <jakob@kastelo.net>
### Purpose
Setting default configuration was not working properly since the
defaults struct is not deeply copied.
### Testing
Try running commands to change default configuration and either inspect
`config.xml` or `/rest/config` result to see the applied changed.
Example:
```
./syncthing cli config defaults folder versioning params set keep 5
```
* main:
chore(gui, man, authors): update docs, translations, and contributors
feat(gui): close a modal when pressing ESC after switching modal tabs (fixes#9489) (#10092)
Flattened the copier code more. Also removing and moving some
parameters/return values to simplify things. Generally rely less on
return values, e.g. by handling errors right away and using `state` to
do the right thing (e.g. abort on failure).
Supposed to be a refactor without any behaviour changes, except for
fixing a tiny regression on folder order: We used to try copying from
the same folder first, but lost that property at some point (also sent a
PR fixing only that, I'd merge that first making this refactor only).
### Purpose
As stated in #9489 after clicking on a tab link to switch tabs in a
modal you can no longer close the modal through clicking the ESC key
unless you click anywhere on the modal to focus on it again.
### Testing
- Click on a modal that has tabs like "Settings" or "Add Folder" and
switch tabs then click on ESC.
- Check if clicking outside of a modal in the backdrop should still
close the modal.
### Demo
https://github.com/user-attachments/assets/a010db9a-72f7-4160-a7db-ddfebffb4834
Where `folderFilesystems` and `folders` is built, there's a comment
spelling out the purpose: To have the same folder first, as that's the
most likely to get hits. Plus a copy is possibly more efficient than
from another folder, e.g. if that's on a different filesystem. We lost
that behaviour during some unrelated change.
(Also sneaking in a comment fix on yesterdays change.)
This is a draft because I haven't adjusted all the tests yet, I'd like
to get feedback on the change overall first, before spending time on
that.
In my opinion the main win of this change is in it's lower complexity
resp. fewer moving parts. It should also be faster as it only does one
query instead of two, but I have no idea if that's practically
relevant.
This also mirrors the v1 DB, where a block map key had the name
appended. Not that this is an argument for the change, it was mostly
reassuring me that I might not be missing something key here
conceptually (I might still be of course, please tell me :) ).
And the change isn't mainly intrinsically motivated, instead it came
up while fixing a bug in the copier. And the nested nature of that code
makes the fix harder, and "un-nesting" it required me to understand
what's happening. This change fell out of that.
After opening the database, we performed some checks, such as whether
the migration had already been successfully completed. If so, the
function returned immediately, and the database was not closed.
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
* main:
fix(strelaysrv): make the session limiter session-dependent (fixes#10072) (#10073)
build: artifact uploads destination OCI
chore(gui, man, authors): update docs, translations, and contributors
chore(gui): use go list --deps for dependency list (#10071)
### Purpose
Fix https://github.com/syncthing/syncthing/issues/9336
The `emitLoginAttempt` function now checks for the presence of an
`X-Forwarded-For` header. The IP from this header is only used if the
connecting host is either on loopback or on the same LAN.
In the case of a host pretending to be a proxy, we'd still have both IPs
in the logs, which should make this much less critical from a security
standpoint.
### Testing
1. directly via localhost
2. via proxy an localhost
#### Logs
```
[3JPXJ] 2025/04/11 15:00:40 INFO: Wrong credentials supplied during API authorization from 127.0.0.1
[3JPXJ] 2025/04/11 15:03:04 INFO: Wrong credentials supplied during API authorization from 192.168.178.5 proxied by 127.0.0.1
```
#### Event API
```
{
"id": 23,
"globalID": 23,
"time": "2025-04-11T15:00:40.578577402+02:00",
"type": "LoginAttempt",
"data": {
"remoteAddress": "127.0.0.1",
"success": false,
"username": "sdfsd"
}
},
{
"id": 24,
"globalID": 24,
"time": "2025-04-11T15:03:04.423403976+02:00",
"type": "LoginAttempt",
"data": {
"proxy": "127.0.0.1",
"remoteAddress": "192.168.178.5",
"success": false,
"username": "sdfsd"
}
}
```
### Documentation
https://github.com/syncthing/docs/pull/907
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This cleans up the option to allow old TLS 1.2 sync connections. The
flag existed for compatibility with old Syncthing versions that don't
support TLS 1.3, which is approximately Syncthing 1.2.2 (September 2019)
and older. ("Approximately" because it depends on the Go version it's
built with and that's when we switched to building with Go 1.13.)
Ref #10062 because it reminded me this exists.
* infrastructure:
feat(stdiscosrv): configurable desired not-found rate
chore(blobs): generalised blob storage
chore(stdiscosrv): path style s3
feat(ursv): add os/arch/distribution metric
chore(strelaypoolsrv): limit number of returned relays
build(infra): run in Docker environment for pushes
chore(stupgrades): expose latest release as a metric
* main:
build: push artifacts to Azure (#10044)
fix(syncthing): use separate lock file instead of locking the certificate (fixes#10053) (#10054)
fix(syncthing): use separate lock file instead of locking the certificate (fixes#10053) (#10054)
* release-1.29.5:
build: push artifacts to Azure (#10044)
fix(syncthing): use separate lock file instead of locking the certificate (fixes#10053) (#10054)
* main:
fix(gui): fix previous commit
fix(gui): mark unseen disconnected devices as inactive (#10048)
fix(strings): differentiate setup(n) and set(v) up (#10024)
chore(fs): changes to allow Filesystem to be implemented externally (#10040)
chore(config): resolve primary STUN servers via SRV record (fixes#10029) (#10031)
build: push artifacts to Azure (#10044)
chore(gui, man, authors): update docs, translations, and contributors
Currently, the "Disconnected (Inactive)" status is only given to devices
that have not been seen for 7 days or longer. However, this is not the
case when adding a new device, or after resetting the database. Those
devices are only marked as "Disconnected", and they will stay like that
even if a long time passes without any connectivity. Moreover, the lack
of an "Inactive" status may confuse the user to believe that their
disconnect is only temporary.
For this reason, always mark devices that have not been seen yet as
"Disconnected (Inactive)".
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
### Purpose
The `fs.Filesystem` interface contains two parts that cannot be
implemented externally because they are private:
* `filesystemWrapperType`: this PR changes `unwrapFilesystem` to
downcast to a specific concrete type
* `underlying`: this PR simply moves it to an unexported interface
### Testing
Regular tests pass.
This adds a simple delay to the process for starting the pull, by
default one second. In practice this means we're likely to wait for
initial index transfer, or multiple messages sent as part of a larger
change. This is better because we're more likely to have the whole
change for the purpose of handling renames etc, and also it's more
efficient to do one larger puller iteration instead of multiple while
also processing changes.
It does however introduce a certain amount of delay into the sync
process, so it can be tuned down or turned off entirely.
This changes the database structure to use one database per folder, with
a small main database to coordinate. Reverts the prior change to buffer
all files in memory when pulling, meaning there is now a phase where the
WAL file will grow significantly, at least for initial sync of folders
with many directories.
---------
Co-authored-by: bt90 <btom1990@googlemail.com>
* main:
fix(config): properly apply defaults when reading folder configuration (#10034)
chore(model): add metric for total number of conflicts (#10037)
build: replace underscore in Debian version (#10032)
The workflow building Debian packages chokes on branches containing
underscores:
```
{:timestamp=>"2025-04-03T10:31:46.749835+0000", :message=>"Invalid package configuration: The version looks invalid for Debian packages. Debian version field must contain only alphanumerics and . (period), + (plus), - (hyphen) or ~ (tilde). I have '1.29.5~dev.13.ga38df11f~srv_stun' which which isn't valid.", :level=>:error}
```
This replaces the offending `_` with a `~` which should yield a valid
version.
This reduces the number of file entries we carry in the database,
sometimes significantly. The downside is that if a file is deleted while
a device is offline, and that device comes back more than the cutoff
interval (six months) later, those files will get resurrected at some
point.
The current limit is far too low for our workloads. Perhaps we should
aim even higher than the 64MiB this patch proposes?
Citing the sqlite docs:
> [...] after committing a transaction the rollback journal file may
remain in the file-system. **This increases performance for subsequent
transactions since overwriting an existing file is faster than append to
a file**, but it also consumes file-system space.
tl;dr: if the limit is too low, we're shooting ourselves in the foot in
terms of performance.
(v2 change)
This cleans up the command line parsing a little:
- Remove the hack for supporting legacy single-dash long options (e.g.
`-home`), thus enabling actual short options
- Move legacy imperative flags from under the serve command into
separate commands, e.g. `syncthing serve --paths` to see the paths list
is now `syncthing paths`, `syncthing --upgrade-check` is now `syncthing
upgrade --check`
- Add environment variable support for all remaining flags for the
`serve` command (with one exception, left for the reader to discover),
as these are now all modifiers and not imperative
```
% syncthing --help
Usage: syncthing <command>
Flags:
-h, --help Show context-sensitive help.
Commands:
serve Run Syncthing (default)
cli Command line interface for Syncthing
browser Open GUI in browser, then exit
decrypt Decrypt or verify an encrypted folder
device-id Show device ID, then exit
generate Generate key and config, then exit
paths Show configuration paths, then exit
upgrade Perform or check for upgrade, then exit
version Show current version, then exit
debug Various debugging commands
install-completions Print commands to install shell completions
Run "syncthing <command> --help" for more information on a command.
```
```
% syncthing serve --help
Usage: syncthing serve [flags]
Run Syncthing (default)
Flags:
-h, --help Show context-sensitive help.
-C, --config=PATH Set configuration directory (config and keys) ($STCONFDIR)
-D, --data=PATH Set data directory (database and logs) ($STDATADIR)
-H, --home=PATH Set configuration and data directory ($STHOMEDIR)
--allow-newer-config Allow loading newer than current config version ($STALLOWNEWERCONFIG)
--audit Write events to audit file ($STAUDIT)
--auditfile=PATH Specify audit file (use "-" for stdout, "--" for stderr) ($STAUDITFILE)
--db-maintenance-interval=8h Database maintenance interval ($STDBMAINTINTERVAL)
--gui-address=URL Override GUI address (e.g. "http://192.0.2.42:8443") ($STGUIADDRESS)
--gui-apikey=API-KEY Override GUI API key ($STGUIAPIKEY)
--no-console Hide console window ($STHIDECONSOLE)
--logfile=PATH Log file name (see below) ($STLOGFILE)
--logflags=BITS Select information in log line prefix (see below) ($STLOGFLAGS)
--log-max-old-files=N Number of old files to keep (zero to keep only current) ($STNUMLOGFILES)
--log-max-size=BYTES Maximum size of any file (zero to disable log rotation) ($STLOGMAXSIZE)
--no-browser Do not start browser ($STNOBROWSER)
--no-default-folder Don't create the "default" folder on first startup ($STNODEFAULTFOLDER)
--no-port-probing Don't try to find free ports for GUI and listen addresses on first startup ($STNOPORTPROBING)
--no-restart Do not restart Syncthing when exiting due to API/GUI command, upgrade, or crash ($STNORESTART)
--no-upgrade Disable automatic upgrades ($STNOUPGRADE)
--paused Start with all devices and folders paused ($STPAUSED)
--unpaused Start with all devices and folders unpaused ($STUNPAUSED)
--verbose Print verbose log output ($STVERBOSE)
--debug-gui-assets-dir=PATH Directory to load GUI assets from ($STGUIASSETS)
--debug-perf-stats Write running performance statistics to perf-$pid.csv (Unix only) ($STPERFSTATS)
--debug-profile-block Write block profiles to block-$pid-$timestamp.pprof every 20 seconds ($STBLOCKPROFILE)
--debug-profile-cpu Write a CPU profile to cpu-$pid.pprof on exit ($STCPUPROFILE)
--debug-profile-heap Write heap profiles to heap-$pid-$timestamp.pprof each time heap usage increases ($STHEAPPROFILE)
--debug-profiler-listen=ADDR Network profiler listen address ($STPROFILER)
--debug-reset-delta-idxs Reset delta index IDs, forcing a full index exchange
...
```
Similarly to #10009, we will remove some discontinued STUN servers,
except instead of being the official primary server, it's some
unofficial secondary STUN servers.
### Testing
Use a STUN client (like [`pystun3`](https://pypi.org/project/pystun3))
to probe that the removed STUN servers are inactive.
### Documentation
syncthing/docs#902
The mechanism for primary STUN servers, is still intact, in case this
gets retried with a different domain.
### Purpose
As seen in [stun.syncthing.net doesn’t resolve
anymore](https://forum.syncthing.net/t/stun-syncthing-net-doesnt-resolve-anymore/24075/2?u=marbens)
on the forums, stun.syncthing.net has been shut down, so I think it's
probably a good idea to remove it.
### Testing
1. Have two or more devices
2. Disable Relaying
3. Have no Internet ports open on either end for incoming connections
trigger STUN)
4. Enable the `stun` debugging facility in the Actions -> Logs ->
Debugging Facilities
5. Verify that it doesn't output something like this within a few
seconds:
```
2025-03-30 05:51:32 Enabled debug data for "stun"
2025-03-30 05:51:47 Starting stun for Stun@udp://[::]:22000
2025-03-30 05:51:47 Running stun for Stun@udp://[::]:22000 via stun.syncthing.net:3478
2025-03-30 05:51:47 Stun@udp://[::]:22000 stun addr resolution on stun.syncthing.net:3478: lookup stun.syncthing.net: no such host
```
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
### Purpose
In the GUI, the device ID validation was case-sensitive and didn’t
account for dash variations, which allowed users to enter an existing
device ID without receiving proper feedback.
This fix ensures the ID is validated in its canonical form, thus
preventing the user from submitting the request if the device ID already
exists.
### Testing
To test this change, try adding a new device with an ID that matches an
existing device, but with a different case or dashes.
Switch the database from LevelDB to SQLite, for greater stability and
simpler code.
Co-authored-by: Tommy van der Vorst <tommy@pixelspark.nl>
Co-authored-by: bt90 <btom1990@googlemail.com>
We've had weak/rolling hashing in the code for quite a while. It was a
popular request for a while, based on the belief that rsync does this
and we should too. However, the benefit is quite small; we save on
average about 0.8% of transferred blocks over the population as a whole:
<img width="974" alt="Screenshot 2025-03-28 at 17 09 02"
src="https://github.com/user-attachments/assets/bbe10dea-f85e-4043-9823-7cef1220b4a2"
/>
This would be fine if the cost was comparably low, however the downside
of attempting rolling hash matching is that we (by default) do a
complete file read on the destination in order to look for matches
before we starting pulling blocks for the file. For any larger file this
means a sometimes long, I/O-intensive pause before the file starts
syncing, for usually no benefit.
I propose we simply rip off the bandaid and save the effort.
Currently, some options are automatically enabled or disabled depending
on the folder type. However, there is no explanation in the GUI on why
the options are like that. Thus, add short explanatory notes to each
case, where the option is either disabled or enabled according to the
current folder type.
### Purpose
This exposes four methods from `Model` through `Internals`. It allows
apps like Synctrain to obtain information about local/remote need and
sync progress.
### Testing
No testing seems necessary, functions are exported verbatim.
### Screenshots
N/a
### Documentation
Not public API, I am aware this interface may change at any time.
## Authorship
OK.
Co-authored-by: Ross Smith II <ross@smithii.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This allows users to easily disable nightly builds in their forks,
simply by disabling the
build-nightly action.
### Testing
I tested it in my fork, and it works.
### Purpose
Path autocompletion wasn't working when using `~` as a shortcut for the
home directory. The issue occurred because the tilde was expanded to
/home/user, which caused the suggestion to no longer match the input
(thus preventing the autocompletion from appearing in the suggestion
list).
To fix this, I replaced the custom `parentAndBase` function, which
handled path splitting in a more complex way, with `filepath.Split` from
the standard `path/filepath` package. This prevents tilde expansion
while keeping the expected behavior for path splitting.
### Testing
The issue has been tested manually on Linux.
### Screenshots
This is extracted from PR #9175. This deduplicates `SetPassword` calls
and makes `postAdjustGui` a single place where PR #9175 can add
another adjustment step for sanitizing changes to WebAuthn
credentials.
This also adds tests to validate that the refactored logic was not
broken.
These refactorizations were made in [PR #9175][1] to accommodate a few
new variants of authentication method and body content. On request from
reviewers, this PR extracts it as a smaller refactorization to review in
isolation.
### Purpose
This extracts a shared `httpRequest` base function from `httpGet` and
`httpPost`, which will be used in PR #9175 for new helper functions
`httpGetCsrf` (hiding all optional parameters except the CSRF token),
`httpPostCsrf` (same) and `httpPostCsrfAuth` (hiding basic auth
parameters). A `getSessionCookie` function is also extracted from
`hasSessionCookie` and will be used to test that concurrent WebAuthn
authentications result in separate sessions (indicated by different
session cookies).
### Purpose
On iOS, the FSEvents API for watching files (also used on macOS) is not
available, but `kqueue` is. This PR ensures `kqueue` support is built on
iOS instead of the FSEvents based watcher implementation.
Before this PR, you could already use the `kqueue` build option to force
its usage. Unfortunately `gomobile`, the tool that I use to build
Syncthing for iOS and macOS for Synctrain, does not support setting
different build flags for iOS and macOS (unless I build separately for
each, which is a bit of a hassle because XCode nonsense). I am assuming
there are good reasons to support FSEvents even though `kqueue` is also
available on macOS (but I'm not sure why?). I do know FSEvents has been
working fine for me on macOS so it seems best to use FSEvents on macOS
and kqueue on iOS.
Note that this also requires https://github.com/syncthing/notify/pull/4
to be merged in `synchting/notify` (until that is done, this PR will
fail to build on iOS due to `notify` still trying to link to `fsevents`
stuff when the `kqueue` build flag is not set).
### Testing
I compiled both `syncthing/notify` and syncthing with this PR applied,
and used that to successfully build the Synctrain iOS app, which after
this PR works fine and should follow up file changes a bit quicker.
### Screenshots
n/a
### Documentation
n/a
## Authorship
Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
### Purpose
This is a [new function](https://pkg.go.dev/slices@go1.21.0#Contains)
added in the go1.21 standard library, which can make the code more
concise and easy to read.
### Testing
Describe what testing has been done, and how the reviewer can test the
change
if new tests are not included.
### Screenshots
If this is a GUI change, include screenshots of the change. If not,
please
feel free to just delete this section.
### Documentation
If this is a user visible change (including API and protocol changes),
add a link here
to the corresponding pull request on https://github.com/syncthing/docs
or describe
the documentation changes necessary.
## Authorship
Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
Signed-off-by: dashangcun <907225865@qq.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
The requirements for Windows code signing changed in 2023, so that newly
generated certificates can only be stored in hardware modules. Luckily,
I managed to snag a three year certificate before that so it hasn't
affected us so much. Now though, it does, because our cert is expiring
in March.
This changes the code signing process for Windows to use a cloud
service, Azure Trusted Signing. This appears to work equally well and
outsources the problem entirely, while also being cheaper than the
actual certificate was to begin with. 🤷
The signing entity will be Kastelo AB and not the Syncthing Foundation,
because the latter is almost impossible to get a certificate for as it's
not a normal corporate entity whose existence can be verified, etc. This
is also how it was prior to the latest certificate; it's not ideal, but
I think it's acceptable under the circumstances.
Currently, this just results in a very ambiguous `setting metadata: lookup
failed` while it could report what it's looking up and why it failed
(not found, etc).
Marshal() was called with the read lock held, and in turn called
Created() which also takes the read lock. This is fine by itself, but
there is a risk of deadlock if another call to lock the mutex happens
concurrently, as the lock call will block the inner rlock and the outer
rlock can never become unlocked.
It's an easy fix as marshalling is guaranteed to be called with a read
lock and does not need to call any methods that read lock themselves.
This was broken in the Protobuf refactor. The reason is that with the
previous library, generated types would have a JSON marshalling method
that would automatically return strings for enums. In the current
library you need to use the jsonpb marshaller for that, but these are
hand crafted structs so we can't do that. The easy solution is to just
use strings directly, since this is an API-only type anyway.
Commit dee920a840 was merged into
Weblate's repository, but later replaced on main by
2167ce9656. This led to a merge conflict,
which this commit fixes cleanly. After merging, we can unlock Weblate
again.
The GitHub checkout action does weird stuff with tags which breaks `git
describe`. This works around that so we get proper release builds on tag
pushes.
We must skip unix sockets, fifos, etc when scanning as these are not
filetypes we can handle. Currently we return a "bug" error, which
results in the walk being aborted and the rest of the tree being
essentially invisible to Syncthing. Instead, just ignore these files and
continue onwards.
This might well be #9859 as well but I can't confirm.
When creating an initial default config, we usually probe for a free
TCP port. But when a UNIX socket is specified via the `STGUIADDRESS=`
override or the `--gui-address=unix:///...` command line syntax, parsing
that option will fail during port probing.
The solution is to just skip the port probing when the address is
determined to specify something other than a TCP socket.
### Testing
Start with a fresh home directory each time.
1. Specify a UNIX socket for the GUI (works with this PR):
TMPHOME=$(mktemp -d); ./syncthing --home=$TMPHOME
--gui-address=unix://$TMPHOME/socket
2. Specify no GUI address (probes for a free port if default is taken,
as before):
TMPHOME=$(mktemp -d); ./syncthing --home=$TMPHOME
3. Specify a TCP GUI address (probes whether the given port is taken,
as before):
TMPHOME=$(mktemp -d); ./syncthing --home=$TMPHOME
--gui-address=127.0.0.1:8385
At a high level, this is what I've done and why:
- I'm moving the protobuf generation for the `protocol`, `discovery` and
`db` packages to the modern alternatives, and using `buf` to generate
because it's nice and simple.
- After trying various approaches on how to integrate the new types with
the existing code, I opted for splitting off our own data model types
from the on-the-wire generated types. This means we can have a
`FileInfo` type with nicer ergonomics and lots of methods, while the
protobuf generated type stays clean and close to the wire protocol. It
does mean copying between the two when required, which certainly adds a
small amount of inefficiency. If we want to walk this back in the future
and use the raw generated type throughout, that's possible, this however
makes the refactor smaller (!) as it doesn't change everything about the
type for everyone at the same time.
- I have simply removed in cold blood a significant number of old
database migrations. These depended on previous generations of generated
messages of various kinds and were annoying to support in the new
fashion. The oldest supported database version now is the one from
Syncthing 1.9.0 from Sep 7, 2020.
- I changed config structs to be regular manually defined structs.
For the sake of discussion, some things I tried that turned out not to
work...
### Embedding / wrapping
Embedding the protobuf generated structs in our existing types as a data
container and keeping our methods and stuff:
```
package protocol
type FileInfo struct {
*generated.FileInfo
}
```
This generates a lot of problems because the internal shape of the
generated struct is quite different (different names, different types,
more pointers), because initializing it doesn't work like you'd expect
(i.e., you end up with an embedded nil pointer and a panic), and because
the types of child types don't get wrapped. That is, even if we also
have a similar wrapper around a `Vector`, that's not the type you get
when accessing `someFileInfo.Version`, you get the `*generated.Vector`
that doesn't have methods, etc.
### Aliasing
```
package protocol
type FileInfo = generated.FileInfo
```
Doesn't help because you can't attach methods to it, plus all the above.
### Generating the types into the target package like we do now and
attaching methods
This fails because of the different shape of the generated type (as in
the embedding case above) plus the generated struct already has a bunch
of methods that we can't necessarily override properly (like `String()`
and a bunch of getters).
### Methods to functions
I considered just moving all the methods we attach to functions in a
specific package, so that for example
```
package protocol
func (f FileInfo) Equal(other FileInfo) bool
```
would become
```
package fileinfos
func Equal(a, b *generated.FileInfo) bool
```
and this would mostly work, but becomes quite verbose and cumbersome,
and somewhat limits discoverability (you can't see what methods are
available on the type in auto completions, etc). In the end I did this
in some cases, like in the database layer where a lot of things like
`func (fv *FileVersion) IsEmpty() bool` becomes `func fvIsEmpty(fv
*generated.FileVersion)` because they were anyway just internal methods.
Fixes#8247
Each section in the advanced settings dialog has similar code to insert
repeated input fields for each option. But only the first section (GUI
options) was adjusted in #9743 to avoid calling the `inputTypeFor()`
function repeatedly.
Apply the same caching to a locally scoped variable for each ng-repeat
entry by defining it in an ng-init directive.
I came accross this in another context and didn't investigate fully, but
literally ten lines above this code, in another method, we say that
filesets _must_ be created under the lock. It's either one or the other
and I'm taking the safer route here.
---------
Co-authored-by: Simon Frei <freisim93@gmail.com>
Encrypted-to-encrypted connections (i.e., ones where both sides set a
password) used to work but were broken in the 1.28.0 release. The
culprit is the 5342bec1b refactor which slightly changed how the request
was constructed, resulting in a bad block hash field.
Co-authored-by: Simon Frei <freisim93@gmail.com>
### Purpose
When generating a new `config.xml` file with default options, the GUI
address is populated with a hard-coded default value of
`127.0.0.1:8384`, except for a random free port if that default one is
occupied. This is independent from the GUI configuration default address
defined in the protobuf description. More importantly, it ignores any
`STGUIADDRESS` override given via environment variable or command-line
option, thus probing for the default port instead of the one specified
via override.
The `ProbeFreePorts()` function now respects the override, by reading
the `GUIConfiguration.Address()` method instead of using hard-coded
defaults.
When not calling `ProbeFreePorts()`, the override should still be
persisted rather than the default address. This happens only when
generating a fresh default `config.xml`, never on an existing one.
This adds `allow_contributor: true` which allows approvals by
contributors to the PR (but still not the author themself, which is a
different thing). This allows things like pushing minor fixups while
also approving.
The `ignore_update_merges: true` option makes it so that someone is not
considered a "contributor" just because they push the merge button to
update the branch. In principle this is not needed given the above, but
I like it for clarity.
### Purpose
This closes#9400 by always expanding tildes when parent/subdir checks
are done.
### Testing
I tested this by creating folders with paths to parent or subdirectories
of the default folder that include a tilde in their path as shown in the
attached screenshots.
With this change, overlap will be detected regardless of wether or not
tildes are used in other folder paths.
### Screenshots
Default Folder:
Newly created folder (parent directory in this case)
---------
Signed-off-by: tobifroe <froeltob@pm.me>
### Purpose
As discussed in #9686
Syncthing currently does not check folderstate on remote device before
pulling. If no devices have a valid folderstate (i.e all devices have
the folder paused) it will still attempt to pull. On large folders this
will cause a hanging "Syncing" status.
This checks whether at least one connected device has the file available
and has a valid folderstate.
### Testing
Tested locally on multiple devices.
We're new to Go (all our stuff is Python) so please bear with!
Interested if there may be a better place to slot this in.
Thanks,
Jon
---------
Co-authored-by: Simon Frei <freisim93@gmail.com>
Currently, the "Restart" and "Shutdown" buttons are displayed in the
middle of the Actions menu. On the other hand, the "Log Out" button is
displayed at the very bottom. However, in other cases, e.g. the menus in
operating systems like Windows or macOS, these kind of buttons are
usually grouped together.
Therefore, move the "Restart" and "Shutdown" buttons down, so that they
are listed together with the "Log Out" button. Also, change the order,
so that it goes from the least impactful ("Log Out") to the most
impactful ("Shutdown").
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
### Screenshots
#### Before
#### After
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
### Purpose
This fixes#9775. I also improved the comments as they were lacking.
My apologies for introducing this bug. In summary, the bug was
```
mode = mode ^ (ModeSymlink | ModeIrregular)
```
didn't correctly reset those bits. This correctly resets them:
```
mode = mode &^ ModeSymlink &^ ModeIrregular
```
Tested and working in Windows 11 version 10.0.22631.4317. I didn't test
in other versions, but I'm sure this is the only issue.
Currently we log on every single one of 10 retries deep in the upnp
stack. However we also return the failure as an error, which is bubbled
up a while until it's logged at debug level. Switch that around, such
that the repeat logging happens at debug level but the top-level happens
at info. There's some chance that this will newly log errors from
nat-pmp that were previously hidden in debug level - I hope those are
useful and not too numerous.
Also potentially this can even close#9324, my (very limited)
understanding of the reports/discussion there is that there's likely no
problem with syncthing beyond the excessive logging, it's some weird
router behaviour.
These CSS overrides address issues that are already present on wider
screens, so apply it there. Some experiments show we might even want to
up the limit more, but I am chicken and lazy, so I propose to use the
existing 470px media block.
Supersedes another PR after not getting any reaction to feedback there:
https://github.com/syncthing/syncthing/pull/9591#issuecomment-2212586134
Co-authored-by: Jakob Borg <jakob@kastelo.net>
As discussed in
https://github.com/syncthing/syncthing/pull/9175#discussion_r1730431703,
entries in advanced settings are unusable if they are comprised of a
list of objects. It just displays `[object Object], [object Object],
[object Object]`, e.g. for the devices a folder is shared with.
Filter out these config elements by detecting an array whose members are
not all strings or numbers, and setting them to `skip` type.
Fix some unnecessary repetition in calling `inputTypeFor()`, since it is
already cached in the `ng-init` directive.
### Purpose
Fixes#9776 by tweaking the text/background colours of disabled checkbox
panels when dark mode is enabled.
It was [noted on that
issue](https://github.com/syncthing/syncthing/issues/9776#issuecomment-2424828520)
that there's a bigger issue around the correctness of using the
`disabled` attribute on a `<div>` in the first place, but this PR does
not attempt to change that.
### Testing
I've hooked up the GUI files against a release build as suggested below.
### Screenshots
Using the dark theme, or the default theme with a system dark scheme:
Using the black theme:
These borrow the colours from dark theme text inputs and black theme
tabs for a consistent look (initially I tried the text colour of
disabled text inputs, but that produced some poor contrast).
Skipping these makes the sequence numbering inconcistent; we've received
a file and suppsedly added it to the database, but if you check the
sequence number afterwards it didn't increase, i.e., we trigger [this
failure
condition](47f48faed7/lib/model/indexhandler.go (L447-L459))
and, similarly, a future update will look like there was a hole in the
numbering.
I propose to at least temporarily remove this optimisation in order for
things to make more sense. Is there a reason to keep this beyond saving
some database operations?
This should prevent the panic that occurred in this test run:
https://github.com/syncthing/syncthing/actions/runs/11095876010/job/30825046810
```
2024-09-29T21:01:53.5425372Z === RUN TestIssue4357
2024-09-29T21:01:53.5505943Z panic: runtime error: integer divide by zero [recovered]
2024-09-29T21:01:53.5512200Z panic: runtime error: integer divide by zero
2024-09-29T21:01:53.5516633Z
2024-09-29T21:01:53.5523018Z goroutine 2655 [running]:
2024-09-29T21:01:53.5524157Z github.com/thejerf/suture/v4.(*Supervisor).runService.func2.2()
2024-09-29T21:01:53.5527176Z /home/runner/go/pkg/mod/github.com/thejerf/suture/v4@v4.0.5/supervisor.go:563 +0xd0
2024-09-29T21:01:53.5530556Z panic({0x1080d20?, 0x1851290?})
2024-09-29T21:01:53.5564723Z /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.1.linux-amd64/src/runtime/panic.go:785 +0x132
2024-09-29T21:01:53.5566616Z github.com/syncthing/syncthing/lib/model.(*model).numHashers(0xc0006f6180, {0x117dc1a, 0x7})
2024-09-29T21:01:53.5568061Z /home/runner/work/syncthing/syncthing/lib/model/model.go:2581 +0x210
2024-09-29T21:01:53.5569912Z github.com/syncthing/syncthing/lib/model.(*folder).scanSubdirsChangedAndNew(0xc00c38c808, {0x0, 0x0, 0x0}, 0xc0003fc060)
2024-09-29T21:01:53.5571612Z /home/runner/work/syncthing/syncthing/lib/model/folder.go:653 +0x250
2024-09-29T21:01:53.5573010Z github.com/syncthing/syncthing/lib/model.(*folder).scanSubdirs(0xc00c38c808, {0x0, 0x0, 0x0})
2024-09-29T21:01:53.5574447Z /home/runner/work/syncthing/syncthing/lib/model/folder.go:512 +0xd0f
2024-09-29T21:01:53.5576011Z github.com/syncthing/syncthing/lib/model.(*folder).scanTimerFired(0xc00c38c808)
2024-09-29T21:01:53.5577367Z /home/runner/work/syncthing/syncthing/lib/model/folder.go:916 +0x46
2024-09-29T21:01:53.5579010Z github.com/syncthing/syncthing/lib/model.(*folder).Serve(0xc00c38c808, {0x1307650, 0xc0006a0910})
2024-09-29T21:01:53.5580428Z /home/runner/work/syncthing/syncthing/lib/model/folder.go:205 +0xd7e
2024-09-29T21:01:53.5581624Z github.com/thejerf/suture/v4.(*Supervisor).runService.func2()
2024-09-29T21:01:53.5582978Z /home/runner/go/pkg/mod/github.com/thejerf/suture/v4@v4.0.5/supervisor.go:567 +0x249
2024-09-29T21:01:53.5584400Z created by github.com/thejerf/suture/v4.(*Supervisor).runService in goroutine 2651
2024-09-29T21:01:53.5585872Z /home/runner/go/pkg/mod/github.com/thejerf/suture/v4@v4.0.5/supervisor.go:541 +0x32a
2024-09-29T21:01:53.5661413Z FAIL github.com/syncthing/syncthing/lib/model 5.510s
```
### Testing
I have not been able to reproduce the panic throughout a few minutes of
continuously running the test without this fix, but judging by the
traceback it seems to only happen if the test happens to delete the
folder from config at the same time `scanTimerFired` triggers.
### Purpose
Syncthing had a healthcheck API for a while, and the example Dockerfile
for it has it in the form of:
HEALTHCHECK --interval=1m --timeout=10s \
CMD curl -fkLsS -m 2 127.0.0.1:8384/rest/noauth/health | grep -o
--color=never OK || exit 1
Let's add it to the docker-compose as well
### Testing
I use this docker-compose.yml file to deploy via ansible (using
community.docker.docker_compose_v2) to my machine with success, using
`wait: true` in ansible for it to use `docker compose up --wait`.
```yml
- name: Enable syncthing docker
community.docker.docker_compose_v2:
project_src: /srv/syncthing
wait: true
wait_timeout: 90
```
In ignores, normalize the input when parsing it.
When scanning, normalize earlier such that the path is already
normalized when checking ignores. This requires splitting normalization
of the string from normalization of the file, as we don't want to
attempt the latter if the file is ignored.
Closes#9598
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
There was a bug that the unique ID was not set when reporting was
enabled, and thus the reports where rejected by the server. The unique
ID got set only on startup, so next time Syncthing restarted.
This makes sure to set the unique ID when blank.
I can see already in our Sentry data that there are a fair amount of
these warnings, and mostly the shape of it. Asking users to report them
will likely cause a lot of reporting effort to fairly little additional
value. We can do that when/if we have something more targeted to ask
for.
This codifies a review policy which is closer to what I always
envisioned, but which isn't expressible using the normal checks in the
GitHub GUI. It would move the commit approval check from GitHub into the
policy-bot check which is already present to enforce the
conventional-commits standard. Approvals in general would still work the
same -- it's just that the bot picks it up and toggles the status
accordingly. From a GitHub side when this is enabled we'd remove the
requires-review check from there and let the bot decide that part. We
would still require builds and tests to pass of course.
There are a couple of relexations from the current policy, details in
the code but briefly:
- Changes to translations or dependencies by a trusted person don't
require review
- Trivial changes by a trusted person, explicitly marked as such, don't
require review
This enables less bureaucracy for things like adding new translated
languages and updating dependencies, and enables the trivial-change
workflow to a larger audience than, like, me, who could always just
bypass the rules by way of being admin.
### Purpose
Since #8757, the Syncthing GUI now has an unauthenticated state. One
consequence of this is that `$scope.versionBase()` is not initialized
while unauthenticated, which causes the `docsURL` function to truncate
links to just `https://docs.syncthing.net`/, discarding the section
path. This currently affects at least the "Help > Introduction" link
reachable both while logged in and not. The issue is exacerbated in
https://github.com/syncthing/syncthing/pull/9175 where we sometimes want
to show additional contextual help links from the login page to
particular sections of the docs.
I don't think it's any worse to try to preserve the section path even
without an explicit version tag, than to fall back to just the host and
lose all context the link was attempting to provide.
### Testing
- On commit b1ed2802fb (before):
- Open the GUI, set a username and log out.
- Open the "Help" drop-down. The "Introduction" item links to:
https://docs.syncthing.net/
- Log in.
- Open the "Help" drop-down. The "Introduction" item links to:
https://docs.syncthing.net/v1.27.10/intro/gui
- On commit 44fef31780 (after):
- Open the GUI, set a username and log out.
- Open the "Help" drop-down. The "Introduction" item links to:
https://docs.syncthing.net/intro/gui
- Log in.
- Open the "Help" drop-down. The "Introduction" item links to:
https://docs.syncthing.net/v1.27.10/intro/gui
### Screenshots
This is a GUI change, but affecting only URLs in the markup with no
visual changes.
### Drawbacks
If a `docsURL` call generates a versionless link to a docs page that
doesn't exist on https://docs.syncthing.net - presumably because
Syncthing is not the latest version and links to a deleted page? - then
this will lead to GitHub's generic 404 page with no link to the
Syncthing docs root. Before, any versionless link would also be a
pathless link, leading to the Syncthing docs root instead of a 404 page.
### Purpose
The primary aim of this change is to minimize log clutter in production
environments. There are many lines in the logs coming from an expected
race condition when two devices connect `already connected to this
device`. These messages do not indicate errors and can overwhelm the log
files with unnecessary noise.
By lowering the logging level, we enhance the usability of the logs,
making it easier for users and developers to identify actual issues
without being distracted
### Testing
1. Build syncthing locally
2. Start two Syncthing instances
```bash
./syncthing -no-browser -home=~/.config/syncthing1
./syncthing -no-browser -home=~/.config/syncthing2
```
3. Enable the DEBUG logs from UI for `connections` package
4. Connect the synching instances by adding remote devices from the UI
5. Observe the logs for the message `XXXX already connected to this
device`
### Screenshots
## Authorship
Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
Move infrastructure related commands to under `cmd/infra` and
development stuff to `cmd/dev`. The default build command builds the
regular user facing binaries: syncthing, stdiscosrv, and strelaysrv.
Reasoning in comments. The main motivation is to avoid all the case
checks when walking the filesystem.
"again" as we already tried once, but it caused a major issue ragarding
mtimefs layer. The root of this problem has been fixed in the meantime
in ac8b3342a
* infrastructure:
chore(stdiscosrv): ensure incoming addresses are sorted and unique
chore(stdiscosrv): use zero-allocation merge in the common case
chore(stdiscosrv): properly clean out old addresses from memory
chore(stdiscosrv): calculate IPv6 GUA
The read/write loops may keep going for a while on a closing connection
with lots of read/write activity, as it's random which select case is
chosen. And if the connection is slow or even broken, a single
read/write
can take a long time/until timeout. Add initial non-blocking selects
with only the cases relevant to closing, to prioritize those.
This would have addressed a recent issue that arose when re-ordering our
"filesystem layers". Specifically moving the caseFilesystem to the
outermost layer. The previous cache included the filesystem, and as such
all the layers below. This isn't desirable (to put it mildly), as you
can create different variants of filesystems with different layers for
the same path and options. Concretely this did happen with the mtime
layer, which isn't always present. A test for the mtime related breakage
was added in #9687, and I intend to redo the caseFilesystem reordering
after this.
Ref: #9677
Followup to: #9687
This is to add the generation of `compat.json` as a release artifact. It
describes the runtime requirements of the release in question. The next
step is to have the upgrade server use this information to filter
releases provided to clients. This is per the discussion in #9656
---------
Co-authored-by: Ross Smith II <ross@smithii.com>
This changes the two remaining instances where we use insecure HTTPS to
use standard HTTPS certificate verification.
When we introduced these things, almost a decade ago, HTTPS certificates
were expensive and annoying to get, much of the web was still HTTP, and
many devices seemed to not have up-to-date CA bundles.
Nowadays _all_ of the web is HTTPS and I'm skeptical that any device can
work well without understanding LetsEncrypt certificates in particular.
Our current discovery servers use hardcoded certificates which has
several issues:
- Not great for security if it leaks as there is no way to rotate it
- Not great for infrastructure flexibility as we can't use many load
balancer or TLS termination services
- The certificate is a very oddball ECDSA-SHA384 type certificate which
has higher CPU cost than a more regular certificate, which has real
effects on our infrastructure
Using normal TLS certificates here improves these things.
I expect there will be some very few devices out there for which this
doesn't work. For the foreseeable future they can simply change the
config to use the old URLs and parameters -- it'll be years before we
can retire those entirely.
For the upgrade client this simply seems like better hygiene. While our
releases are signed anyway, protecting the metadata exchange is _better_
and, again, I doubt many clients will fail this today.
The test is quite odd and specific, but it does reproduce the issue that
caused #9677, so I'd propose to add it to have a simple regression test
for the basic scenario. Also the option to the fakefs might come handy
for other scenarios where you want to quickly test some behaviour on a
filesystem without nanosecond precision, without actually needing access
to one.
gui: Actually filter scope ID out of IPv6 address when using Remote GUI
(ref #8084)
The current code does not work, because it uses a string in the
replace() method instead of regex. Thus, change it to a proper regex.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The preference for languages in the Accept-Language header field
should not be deduced from the listed order, but from the passed
"quality values", according to the HTTP specification:
https://httpwg.org/specs/rfc9110.html#field.accept-language
This implements the parsing of q=values and ordering within the API
backend, to not complicate things further in the GUI code. Entries
with invalid (unparseable) quality values are discarded completely.
* gui: Fix API endpoint in comment.
gui: Fix incorrect UI language auto detection (fixes#9668)
Currently, the code only checks whether the detected language partially
matches one of the available languages. This means that if the detected
language is "fi" but among the available languages there is only "fil"
and no "fi", then it will match "fi" with "fil", even though the two are
completely different languages.
With this change, the matching is only done when there is a hyphen in
the language code, e.g. "en" will match with "en-US".
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds a header with the operating system version, verbatim in
whatever format the operating system reports it, to the upgrade check.
The intention is that the upgrade server can use this information to
filter out (or maybe just mark) potentially unsupported upgrades.
This adds a header with the operating system version, verbatim in
whatever format the operating system reports it, to the upgrade check.
The intention is that the upgrade server can use this information to
filter out (or maybe just mark) potentially unsupported upgrades.
Based on user request from Weblate, user `@aindriu80`.
Looks promising based on the profile:
https://hosted.weblate.org/user/aindriu80/ Not sure whether almost
30.000 translations in about one month is realistic for a human though.
### Purpose
Wrap access to Model for users that use the syncthing Go package. See
discussion:
https://github.com/syncthing/syncthing/pull/9619#pullrequestreview-2212484910
### Testing
It works with the iOS app. Other than that, there are no current users
of this API (to my knowledge) as Model was only exposed recently form
the iOS app.
### Purpose
This PR contains the set of changes needed to make Syncthing work on iOS
for [my iOS app for
Syncthing](https://github.com/pixelspark/sushitrain).
Most changes originate from [the Mobius Sync
fork](http://github.com/MobiusSync/syncthing/tree/ios). I have removed
the changes from their fork that are not strictly needed for my app
(i.e. their changes to the GUI and command line utilities, for instance)
and squashed it all in a single commit.
In summary, the changes are:
* Resolve non-absolute paths to the 'Documents' folder (basically the
only one an app can/should write user data to by default on iOS)
* Tweaking of build flags/conditions for iOS (i.e. determine which
basicfs_watch, ignoreresult variant to build for iOS)
* Disable upgrade mechanism on iOS
* Make `RequestGlobal` and `PullerProgress` public symbols
* Expose syncthing.app's Model instance (app.M)
* Add no-op stub for SetLowPriority on iOS
I would very much appreciate these changes to be (eventually) merged to
mainline syncthing, as this would allow my iOS app to track the mainline
source code directly and removes the need (for me at least) for
maintaining a separate fork. Perhaps the Mobius folks can also benefit
from this (although as noted this branch does not contain their changes
to e.g. the GUI).
### Testing
This branch has been tested with the iOS app and appears to work fine.
The full set of MobiusSync changes has been used before with success.
### Screenshots
n/a
### Documentation
There should be no visible changes for users due to this set of changes.
---------
Co-authored-by: Simon Pickup <simon@pickupinfinity.com>
Tiny cleanup I noticed while trying to fix/test another issue
(https://github.com/syncthing/syncthing/pull/9600). I shortly tried to
figure out what it was used for in the past, but gave up without
results.
Previously we queried cache with backslashes, and stored entries with
slashes. As in no cache hits ever for non-toplevel files. I also
eventually remembered that cache is disabled by default, so this is a
bit pointless, but still right :P
### Purpose
Avoid the issue where the folder marker is deleted by overzealous
cleanup tools because it's just a useless, empty directory.
We create a small file containing a an admonishment to not delete the
directory, and some metadata that is just for human consumption at the
moment. (But it would parse as a valid yaml file if we wanted to read
this, at some point.)
This will only apply when _creating_ a folder marker, that is, existing
setups will not gain the file automatically. Obviously, when using a
custom folder marker none of this applies.
Also, slightly adjust the permission bits for the folder marker directory and file on Unixes, making sure the group & write bits are unset.
### Testing
I've created and deleted a few folders and it appears to behave as I
expect.
### Screenshots
```
jb@ok:~/somefolder % ls -la
total 0
drwxr-xr-x 3 jb staff 96 May 1 08:52 ./
drwx------ 12 jb staff 384 May 1 08:52 ../
drwxr-xr-x 3 jb staff 96 May 1 08:52 .stfolder/
jb@ok:~/somefolder % ls -l .stfolder
total 8
-rw-r--r-- 1 jb staff 122 May 1 08:52 syncthing-folder-39a4b0.txt
jb@ok:~/somefolder % cat .stfolder/syncthing-folder-39a4b0.txt
# This directory is a Syncthing folder marker.
# Do not delete.
folderID: xtdca-cudyf
created: 2024-05-01T08:52:49+02:00
```
Currently the maximum delay is always derived automatically from the
initial delay. This is fine in most cases, but for some use cases (large
files that take a long time to write) we need to be able to set a longer
max delay than the computed value (e.g., 15s delay with 10min timeout).
This adds a small package `geoip` which knows how to download and manage
the Maxmind GeoLite2 database we use. This removes the need for various
scripts to download and manage the geoip database, something that today
happens on Docker startup for the relay pool server and using various
hand written hacks for the usage reporting server.
The database is downloaded when needed and then refreshed on a
best-effort basis weekly.
Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go)
from 0.42.0 to 0.43.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/quic-go/quic-go/releases">github.com/quic-go/quic-go's
releases</a>.</em></p>
<blockquote>
<h2>v0.43.0</h2>
<h2><em>quic-go.net</em>: Launching a new Documentation Site</h2>
<p>With this release, we're launching a new documentation site for the
quic-go projects (quic-go itself, HTTP/3, webtransport-go, and soon,
masque-go): <a href="https://quic-go.net">quic-go.net</a>.</p>
<p>The documentation site aims to explain QUIC concepts and how they are
made accessible using quic-go's API. This site replaces the wiki, and
the ever-growing README files.</p>
<p>A lot of work has gone into the documentation already, but we're by
no means done yet. The entire source is public in <a
href="https://github.com/quic-go/docs/">https://github.com/quic-go/docs/</a>,
and we're happy about community contributions.</p>
<h2>HTTP Datagrams (RFC 9297)</h2>
<p>This release adds support for HTTP Datagrams (<a
href="https://datatracker.ietf.org/doc/html/rfc9297">RFC 9297</a>), both
on the client and on the server side (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4452">#4452</a>).
HTTP Datagrams are used in WebTransport in CONNECT-UDP (<a
href="https://datatracker.ietf.org/doc/html/rfc9298">RFC 9298</a>),
among others.</p>
<p>The new API for HTTP Datagrams is described on the new documentation
page: <a href="https://quic-go.net/docs/http3/datagrams/">HTTP
Datagrams</a>. The integration of HTTP Datagram support necessitated a
comprehensive refactor of the HTTP/3 package, resulting in several
breaking API changes listed below.</p>
<h2>Breaking Changes</h2>
<ul>
<li>quicvarint: functions now return an <code>int</code> instead the
internal <code>protocol.ByteCount</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4365">#4365</a>)</li>
<li>http3: <code>Server.SetQuicHeaders</code> was renamed to
<code>SetQUICHeaders</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4377">#4377</a>)</li>
<li>http3: <code>Server.QuicConfig</code> was renamed to
<code>QUICConfig</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4384">#4384</a>)</li>
<li>http3: <code>RoundTripper.QuicConfig</code> was renamed to
<code>QUICConfig</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4385">#4385</a>)</li>
<li>http3: <code>RoundTripOpt.CheckSettings</code> was removed (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4416">#4416</a>).
Use the new<code>SingleDestinationRoundTripper</code> API instead.</li>
<li>http3: the <code>HTTPStreamer</code> interface is now implemented by
the <code>http.ResponseWriter</code> (and not the
<code>http.Request.Body</code>) (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4469">#4469</a>)</li>
<li>include the maximum payload size in the
<code>DatagramTooLargeError</code> (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4470">#4470</a>)</li>
</ul>
<h2>Other Notable Changes</h2>
<ul>
<li>GSO and ECN is disabled on kernel versions older than 5 (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4456">#4456</a>)</li>
<li>http3: logging can be controlled using an <code>slog.Logger</code>
(<a
href="https://redirect.github.com/quic-go/quic-go/issues/4449">#4449</a>)</li>
<li>http3: HEAD requests can now be sent in 0-RTT (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4378">#4378</a>)</li>
<li>http3: duplicate QPACK encoder and decoder streams are not rejected
as required by the RFC (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4388">#4388</a>)</li>
<li>http3: Extended CONNECT are blocked until the server's SETTINGS are
received, as required by the RFC (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4450">#4450</a>)</li>
<li>http3: HTTP/3 client connections aren't removed if
<code>RoundTrip</code> errors due to a cancelled context (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4448">#4448</a>).
Thanks to <a
href="https://github.com/GeorgeMac"><code>@GeorgeMac</code></a>!</li>
<li>http3: sniff Content-Type when flushing the ResponseWriter (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4412">#4412</a>).
Thanks to <a
href="https://github.com/WeidiDeng"><code>@WeidiDeng</code></a>!</li>
<li>The <code>Context</code> exposed on the <code>quic.Stream</code> is
now derived from the connection's context (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4414">#4414</a>)</li>
<li>The UDP send and receive buffer size was increased to 7 MiB (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4455">#4455</a>).
Thanks to <a
href="https://github.com/bt90"><code>@bt90</code></a>!</li>
</ul>
<h2>Clarifications on the QUIC Stream State Machine</h2>
<h3>Calling CancelWrite after Close</h3>
<p>After a long and fruitful discussion (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4404">#4404</a>),
we decided to clarify that calling <code>CancelWrite</code> after
<code>Close</code> on a <code>SendStream</code> (or a bidirectional
stream) should cause a state transition from the "Data Sent"
to the "Reset Sent" state, as described in <a
href="https://datatracker.ietf.org/doc/html/rfc9000#section-3.1">section
3.1 of RFC 9000</a>. This matches the current behavior of quic-go,
however, it didn't match the API documentation (fixed in <a
href="https://redirect.github.com/quic-go/quic-go/issues/4419">#4419</a>).</p>
<p>This means that stream data will not be delivered reliably if
<code>CancelWrite</code> is called, and that this applies even if
<code>Close</code> was called before.</p>
<h3>Garbage Collection of Streams</h3>
<p>This release also changes the way streams are garbage-collected (and
the peer is granted additional limit to open a new stream), once they're
not needed anymore, in a subtle way:</p>
<ul>
<li>for the send direction of streams: <a
href="https://redirect.github.com/quic-go/quic-go/issues/4445">#4445</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="93c4785521"><code>93c4785</code></a>
http3: sniff Content-Type when flushing the ResponseWriter (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4412">#4412</a>)</li>
<li><a
href="c0250ce824"><code>c0250ce</code></a>
include the maximum payload size in the DatagramTooLargeError (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4470">#4470</a>)</li>
<li><a
href="34f4d1443f"><code>34f4d14</code></a>
http3: implement on the HTTPStreamer on the ResponseWriter, flush header
(<a
href="https://redirect.github.com/quic-go/quic-go/issues/4469">#4469</a>)</li>
<li><a
href="083ceb42f2"><code>083ceb4</code></a>
http3: rename Settings.EnableDatagram to EnableDatagrams (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4466">#4466</a>)</li>
<li><a
href="e1e5b6294d"><code>e1e5b62</code></a>
README: link to the new documentation site (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4464">#4464</a>)</li>
<li><a
href="2a37c53143"><code>2a37c53</code></a>
http3: add support for HTTP Datagrams (RFC 9297) (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4452">#4452</a>)</li>
<li><a
href="11b11594b2"><code>11b1159</code></a>
http3: fix race condition in client unit test (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4463">#4463</a>)</li>
<li><a
href="4b87539b1e"><code>4b87539</code></a>
delay completion of the receive stream until the reset error was read
(<a
href="https://redirect.github.com/quic-go/quic-go/issues/4460">#4460</a>)</li>
<li><a
href="bff131e546"><code>bff131e</code></a>
delay completion of the send stream until the reset error was delivered
(<a
href="https://redirect.github.com/quic-go/quic-go/issues/4445">#4445</a>)</li>
<li><a
href="12aa63824c"><code>12aa638</code></a>
disable GSO and ECN on kernels older than version 5 (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4456">#4456</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/quic-go/quic-go/compare/v0.42.0...v0.43.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
### Purpose
Adds a new metric `syncthing_connections_active` which equals to the
amount of active connections per device.
Fixes#9527
<!--
Describe the purpose of this change. If there is an existing issue that
is
resolved by this pull request, ensure that the commit subject is on the
form
`Some short description (fixes#1234)` where 1234 is the issue number.
-->
### Testing
I've manually tested it by running syncthing with these changes locally
and examining the returned metrics from `/metrics`.
I've done the following things:
- Connect & disconnect a device
- Increase & decrease the number of connections and verify that the
value of the metric matches with the amount displayed in the GUI.
### Documentation
https://github.com/syncthing/docs/blob/main/includes/metrics-list.rst
needs to be regenerated with
[find-metrics.go](https://github.com/syncthing/docs/blob/main/_script/find-metrics/find-metrics.go)
## Authorship
Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
### Purpose
Firefox uses the last specified favicon link for bookmarks, but only if
it is available on initial page load.
Remove the second link and use ng-href to change the icon instead.
I'm not really familiar with AngularJS, feel free to offer suggestions
for improvements.
### Testing
Briefly tested on Firefox 124.0.2 and Chrome 123.0.6312.105.
### Purpose
Resend our indexes since we fixed that index-sending issue.
I made a new thing to only drop the non-local-device index IDs, i.e.,
those for other devices. This means we will see a mismatch and resend
all indexes, but they will not. This is somewhat cleaner as it avoids
resending everything twice when two devices are upgraded, and in any
case, we have no reason to force a resend of incoming indexes here.
### Testing
It happens on my computer...
The commit 7e4e65ebf5 added links to
devices listed in the Shared With list in the folder info. However, it
only added them to those that had no superscript next to them.
With this change, the links are added to all devices regardless of
whether they have the superscript next to their names or not. The commit
also simplifies the code by using anchors directly instead of wrapping
them in spans.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
### Purpose
Bountysource no longer exists and the readme link 404s. This removes the
Bountysource link and corresponding readme section.
Perhaps the section should instead be replaced by other instructions for
voting on features.
This is an extract from PR #9175, which can be reviewed in isolation to
reduce the volume of changes to review all at once in #9175. There are
about to be several services and API handlers that read and set cookies
and session state, so this abstraction will prove helpful.
In particular a motivating cause for this is that with the current
architecture in PR #9175, in `api.go` the [`webauthnService` needs to
access the
session](https://github.com/syncthing/syncthing/pull/9175/files#diff-e2e14f22d818b8e635572ef0ee7718dee875c365e07225d760a6faae8be7772dR309-R310)
for authentication purposes but needs to be instantiated before the
`configMuxBuilder` for config purposes, because the WebAuthn additions
to config management need to perform WebAuthn registration ceremonies,
but currently the session management is embedded in the
`basicAuthAndSessionMiddleware` which is [instantiated much
later](https://github.com/syncthing/syncthing/pull/9175/files#diff-e2e14f22d818b8e635572ef0ee7718dee875c365e07225d760a6faae8be7772dL371-R380)
and only if authentication is enabled in `guiCfg`. This refactorization
extracts the session management out from `basicAuthAndSessionMiddleware`
so that `basicAuthAndSessionMiddleware` and `webauthnService` can both
use the same shared session management service to perform session
management logic.
### Testing
This is a refactorization intended to not change any externally
observable behaviour, so existing tests (e.g., `api_auth_test.go`)
should cover this where appropriate. I have manually verified that:
- Appending `+ "foo"` to the cookie name in `createSession` causes
`TestHtmlFormLogin/invalid_URL_returns_403_before_auth_and_404_after_auth`
and `TestHtmlFormLogin/UTF-8_auth_works` to fail
- Inverting the return value of `hasValidSession` cases a whole bunch of
tests in `TestHTTPLogin` and `TestHtmlFormLogin` to fail
- (Fixed) Changing the cookie to `MaxAge: 1000` in `destroySession` does
NOT cause any tests to fail!
- Added tests `TestHtmlFormLogin/Logout_removes_the_session_cookie`,
`TestHTTPLogin/*/Logout_removes_the_session_cookie`,
`TestHtmlFormLogin/Session_cookie_is_invalid_after_logout` and
`TestHTTPLogin/200_path#01/Session_cookie_is_invalid_after_logout` to
cover this.
- Manually verified that these tests pass both before and after the
changes in this PR, and that changing the cookie to `MaxAge: 1000` or
not calling `m.tokens.Delete(cookie.Value)` in `destroySession` makes
the respective pair of tests fail.
This re-implements the stalled enhancement from #8808. Thanks @Craeckie
for the idea and first implementation draft!
If a folder is shared to a device with encryption, add a lock icon in
front of the device name under "Shared With" in the folder details
panel. Be careful not to add whitespace caused by line wraps in HTML
source code, which would defeat the purpose of keeping the icon glued to
the name by a non-breaking space.
Apply the same lock icon for the list of folders shared with a device.
This change was split off from #9355 as an independent clean-up / fix.
See that PR for review discussion, testing, and screenshots.
Improve the wrapping of folder labels / device names by going back to
word-wrapping, but making sure other spans, such as the trailing comma,
do not get separated from the label span.
* Avoid adding whitespace caused by line wraps in HTML source code.
The different cases within the ng-switch block are separated by
newlines for readability, but that gets parsed as whitespace. For
wrapping purposes, this should not happen, because then there is no
way to keep other HTML parts glued to the name / label in each list
entry.
* Simplify redundant conditional comma code.
The separating comma after a device name or folder label (all but the
last) should always stick to it. Use the HTML comment trick to avoid
whitespace and therefore a wrapping opportunity caused by the code
formatting newline. Thus the conditional comma only needs to be
defined once, not in each ng-switch case.
* Wrap at word boundaries and only break up words if necessary.
Use the overflow-wrap: break-word; style instead of word-break:
break-all;. While the latter is suitable for longish paths, breaking
device names or folder labels arbitrarily within words is ugly.
This also makes the the <sup> numbers actually stay glued to their
respective neighboring words.
Include legacy CSS alias "word-wrap" in the class definition.
* Fix indentation (unrelated).
Adds a bool flag to `scanIfItemChanged()` to indicate when the scan was initiated from a delete function, and if so, tell `IsEquivalentOptional()` to ignore Xattrs and Ownership regardless of the global setting.
I tested this with my sledgehammer and it seems to pass.
Go is not cgroup aware and by default will set GOMAXPROCS to the number
of available threads, regardless of whether it is within the allocated
quota. This behaviour causes high amount of CPU throttling and degraded
application performance.
Add support for setting umask value in the Docker `entrypoint.sh`
script. This is useful when
not syncing permissions and working with groups, and needing umask
values like `002` instead of `022`.
The changes to go.mod in latest Go 1.21/1.22 are not fully understood by
older Go that might be pre-installed on builds, so make sure we always
have a modern one in place even for running small release scripts etc.
Somewhere along the way, the non-parallel test became parallel, and at
that point, timeouts occurred. Parallel is better, so increase the
timeout on the offending call a bit...
The new test has a flakiness factor on slow platforms, where the close
on the sending connection races with the last index message, potentially
messing up the count. This adds a wait to ensure that all sent messages
are received, or the test will eventually fail with a timeout.
Currently `IsTraced("xyz")` will return true for
any inclusion of "xyz" in string.
This change splits `STTRACE` using `','`, `' '` and `';'`
as delimiters. That makes facilities separation
more clear.
This makes a couple of small improvements to the folder summary
mechanism:
- The folder summary includes the local and remote sequence numbers in
clear text, rather than some odd sum that I'm not sure what it was
intended to represent.
- The folder summary event is generated when appropriate, regardless of
whether there is an event listener. We did this before because
generating it was expensive, and we wanted to avoid doing it
unnecessarily. Nowadays, however, it's mostly just reading out
pre-calculated metadata, and anyway, it's nice if it shows up reliably
when running with -verbose.
The point of all this is to make it easier to use these events to judge
when devices are, in fact, in sync. As-is, if I'm looking at two
devices, it's very difficult to reliably determine if they are in sync
or not. The reason is that while we can ask device A if it thinks it's
in sync, we can't see if the answer is "yes" because it has processed
all changes from B, or if it just doesn't know about the changes from B
yet. With proper sequence numbers in the event we can compare the two
and determine the truth. This makes testing a lot easier.
This is a refactor of the protocol/model interface to take the actual
message as the parameter, instead of the broken-out fields:
```diff
type Model interface {
// An index was received from the peer device
- Index(conn Connection, folder string, files []FileInfo) error
+ Index(conn Connection, idx *Index) error
// An index update was received from the peer device
- IndexUpdate(conn Connection, folder string, files []FileInfo) error
+ IndexUpdate(conn Connection, idxUp *IndexUpdate) error
// A request was made by the peer device
- Request(conn Connection, folder, name string, blockNo, size int32, offset int64, hash []byte, weakHash uint32, fromTemporary bool) (RequestResponse, error)
+ Request(conn Connection, req *Request) (RequestResponse, error)
// A cluster configuration message was received
- ClusterConfig(conn Connection, config ClusterConfig) error
+ ClusterConfig(conn Connection, config *ClusterConfig) error
// The peer device closed the connection or an error occurred
Closed(conn Connection, err error)
// The peer device sent progress updates for the files it is currently downloading
- DownloadProgress(conn Connection, folder string, updates []FileDownloadProgressUpdate) error
+ DownloadProgress(conn Connection, p *DownloadProgress) error
}
```
(and changing the `ClusterConfig` to `*ClusterConfig` for symmetry;
we'll be forced to use all pointers everywhere at some point anyway...)
The reason for this is that I have another thing cooking which is a
small troubleshooting change to check index consistency during transfer.
This required adding a field or two to the index/indexupdate messages,
and plumbing the extra parameters in umpteen changes is almost as big a
diff as this is. I figured let's do it once and avoid having to do that
in the future again...
The rest of the diff falls out of the change above, much of it being in
test code where we run these methods manually...
This improves the ignore handling so that directories can be fully
ignored (skipped in the watcher) in more cases. Specifically, where the
previous rule was that any complex `!`-pattern would disable skipping
directories, the new rule is that only matches on patterns *after* such
a `!`-pattern disable skipping. That is, the following now does the
intuitive thing:
```
/foo
/bar
!whatever
*
```
- `/foo/**` and `/bar/**` are completely skipped, since there is no
chance anything underneath them could ever be not-ignored
- `!whatever` toggles the "can't skip directories any more" flag
- Anything that matches `*` can't skip directories, because it's
possible we can have `whatever` match something deeper.
To enable this, some refactoring was necessary:
- The "can skip dirs" flag is now a property of the match result, not of
the pattern set as a whole.
- That meant returning a boolean is not good enough, we need to actually
return the entire `Result` (or, like, two booleans but that seemed
uglier and more annoying to use)
- `ShouldIgnore(string) boolean` went away with
`Match(string).IsIgnored()` being the obvious replacement (API
simplification!)
- The watcher then needed to import the `ignore` package (for the
`Result` type), but `fs` imports the watcher and `ignore` imports `fs`.
That's a cycle, so I broke out `Result` into a package of its own so
that it can be safely imported everywhere in things like `type Matcher
interface { Match(string) result.Result }`. There's a fair amount of
stuttering in `result.Result` and maybe we should go with something like
`ignoreresult.R` or so, leaving this open for discussion.
Tests refactored to suit, I think this change is in fact quite well
covered by the existing ones...
Also some noise because a few of the changed files were quite old and
got the `gofumpt` treatment by my editor. Sorry not sorry.
---------
Co-authored-by: Simon Frei <freisim93@gmail.com>
### Purpose
Fix#9241 by expanding tildes in version paths.
When creating the versioner file system, first try to expand any leading
tildes to the user's home directory before handling relative paths. This
makes a version path `"~/p"` expand to `"$HOME/p"` instead of
`"/folder/~/p"`.
### Testing
Added a test to lib/versioner that exercises this code path. Also
manually tested with local syncthing instances.
### Purpose
This PR changes behaviour of syncthing related to `receive-only`
folders, which I believe to be a bug since I wouldn't expect the current
behaviour. With the current syncthing codebase, a file of a
`receive-only` folder that is only modified locally can cause the
creation of a `.sync-conflict` file.
### Testing
Consider this szenario: Setup two paired clients that sync a folder with
a given file (e.g. `Test.txt`). One of the clients configures the folder
to be `receive-only`. Now, change the contents of the file for the
receive-only client **_twice_**.
With the current syncthing codebase, this leads to the creation of a
`.sync-conflict` file that contains the modified contents, while the
regular `Test.txt` file is reset to the cluster's provided contents.
This is due to a `protocol.FileInfo#ShouldConflict` check, that is
succeeding on the locally modified file.
This PR changes this behaviour to not reset the file and not cause the
creation of a `.sync-conflict`. Instead, the second content update is
treated the same as the first content update.
This PR also contains a test that fails on the current codebase and
succeeds with the changes introduced in this PR.
### Screenshots
This is not a GUI change
### Documentation
This is not a user visible change.
## Authorship
Your name and email will be added automatically to the AUTHORS file
based on the commit metadata.
#### Thanks to all the syncthing folks for this awesome piece of
software!
### Purpose
This implements CLI completion using the Kongplete module. As a side
effect a CLI structure for syncthing/cli was created for kongplete to be
able to parse and implement CLI completion.
### Testing
I've tested the autocompletion manually, and it had worked, but I hadn't
added any tests so as to test it automatically. Additionally, I ran `go
run build.go test` with all tests passing.
This adds a "token manager" which handles storing and checking expired
tokens, used for both sessions and CSRF tokens. It removes the old,
corresponding functionality for CSRFs which saved things in a file. The
result is less crap in the state directory, and active login sessions
now survive a Syncthing restart (this really annoyed me).
It also adds a boolean on login to create a longer-lived session cookie,
which is now possible and useful. Thus we can remain logged in over
browser restarts, which was also annoying... :)
<img width="1001" alt="Screenshot 2023-12-12 at 09 56 34"
src="https://github.com/syncthing/syncthing/assets/125426/55cb20c8-78fc-453e-825d-655b94c8623b">
Best viewed with whitespace-insensitive diff, as a bunch of the auth
functions became methods instead of closures which changed indentation.
```
% export GOTOOLCHAIN=go1.20.7
% go list -m all | cut -d ' ' -f 1 | xargs go get -u
% go mod tidy
```
Except:
- github.com/jackpal/gateway now requires Go 1.21
- github.com/shirou/gopsutil breaks linux-mips in latest version
Currently, with a large number of versioned files, there is a delay
between the Restore Versions modal showing up on the screen and
initialisation of the actual versions tree. This leads to a situation,
where the modal is initially empty, which confuses the user, making
them think that something is not working correctly.
To avoid the above, always show the loading data information. The string
is displayed using static HTML first, and then replaced with the exact
same content once the tree has been initialised. Both elements use the
same style and position, so there is no visual shift between the two.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Bumps
[actions/download-artifact](https://github.com/actions/download-artifact)
from 3 to 4.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/download-artifact/releases">actions/download-artifact's
releases</a>.</em></p>
<blockquote>
<h2>v4.0.0</h2>
<h2>What's Changed</h2>
<p>The release of upload-artifact@v4 and download-artifact@v4 are major
changes to the backend architecture of Artifacts. They have numerous
performance and behavioral improvements.</p>
<p>For more information, see the <a
href="https://github.com/actions/toolkit/tree/main/packages/artifact"><code>@actions/artifact</code></a>
documentation.</p>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/bflad"><code>@bflad</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/download-artifact/pull/194">actions/download-artifact#194</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/download-artifact/compare/v3...v4.0.0">https://github.com/actions/download-artifact/compare/v3...v4.0.0</a></p>
<h2>v3.0.2</h2>
<ul>
<li>Bump <code>@actions/artifact</code> to v1.1.1 - <a
href="https://redirect.github.com/actions/download-artifact/pull/195">actions/download-artifact#195</a></li>
<li>Fixed a bug in Node16 where if an HTTP download finished too quickly
(<1ms, e.g. when it's mocked) we attempt to delete a temp file that
has not been created yet <a
href="hhttps://redirect.github.com/actions/toolkit/pull/1278">actions/toolkit#1278</a></li>
</ul>
<h2>v3.0.1</h2>
<ul>
<li><a
href="https://redirect.github.com/actions/download-artifact/pull/178">Bump
<code>@actions/core</code> to 1.10.0</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="7a1cd3216c"><code>7a1cd32</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/246">#246</a>
from actions/v4-beta</li>
<li><a
href="8f32874a49"><code>8f32874</code></a>
licensed cache</li>
<li><a
href="b5ff8444b1"><code>b5ff844</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/245">#245</a>
from actions/robherley/v4-documentation</li>
<li><a
href="f07a0f73f5"><code>f07a0f7</code></a>
Update README.md</li>
<li><a
href="7226129829"><code>7226129</code></a>
update test workflow to use different artifact names for matrix</li>
<li><a
href="ada9446619"><code>ada9446</code></a>
update docs and bump <code>@actions/artifact</code></li>
<li><a
href="7eafc8b729"><code>7eafc8b</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/244">#244</a>
from actions/robherley/bump-toolkit</li>
<li><a
href="3132d12662"><code>3132d12</code></a>
consume latest toolkit</li>
<li><a
href="5be1d38671"><code>5be1d38</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/243">#243</a>
from actions/robherley/v4-beta-updates</li>
<li><a
href="465b526e63"><code>465b526</code></a>
consume latest <code>@actions/toolkit</code></li>
<li>Additional commits viewable in <a
href="https://github.com/actions/download-artifact/compare/v3...v4">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
If syncOwnership is enabled and the remote uses for example a dockerized
Syncthing it can't fetch the ownername and groupname of the local
instance. Without this patch this led to an endless cycle of detected
changes on the remote and failing re-sync attempts.
This patch skips comparing the ownername and groupname if they zare empty
on one side.
See https://github.com/syncthing/syncthing/issues/9039 for details.
### Testing
Proposed by @calmh in
https://github.com/syncthing/syncthing/issues/9039#issuecomment-1870584783
and tested locally in my setup,
Setup PC 1:
- Syncthing is run in Docker as user `root` and has none of the users
configured that synchronize their files
Setup PC 2:
- this PC has all users locally setup
- Syncthing runs as `systemd` service as user `syncthing` and has
multiple capabilities set to set the correct owner and permissions
Setup PC 3:
- same as PC 2
Handling:
- `PC 1` is send & receive and uses just the `UID` and `GID` identifiers
to store the files
- `PC 2` and `PC 3` synchronize their files over `PC 1` but not directly
to each other
Outcome:
- `PC 2` and `PC 3` should send and receive their files with the correct
ownership and groups from `PC 1`
### Purpose
Instead of hardcoding `SigningKey` as text use `go:embed`. Fixes#9247.
### Testing
* Building syncthing
* Trying to upgrade (signature verification)
On kqueue-systems, folders listen for folder summaries to (be able to)
warn for potential high resource usage. However, it listened for any
folder summary and not for the summary which matches the folder it's
about. This could cause that an unwatched folder causes a folder summary
containing more files than the threshold (10k), and the listening folder
(with the watcher enabled) triggers the warning.
This makes sure that only the folder summaries which are relevant to the
specific folder are being handled.
### Testing
- Fire up some kqueue-system (freebsd, I used).
- add folder A, disable the watcher, add 10001 files
- add folder B with the watcher enabled, no files are needed here
Before the change:
- add an item to folder A, trigger a rescan to speed up the process
- wait some seconds...warning triggered by folder B's
summarySubscription
After the change:
- Only a warning is triggered if the received folder summary matches the
folder which listens for the summaries
Cleanup after #9275.
This renames `fmut` -> `mut`, removes the deadlock detector and
associated plumbing, renames some things from `...PRLocked` to
`...RLocked` and similar, and updates comments.
Apart from the removal of the deadlock detection machinery, no
functional code changes... i.e. almost 100% diff noise, have fun
reviewing.
I'm tired of the fmut/pmut shenanigans. This consolidates both under one
lock; I'm not convinced there are any significant performance
differences with this approach since we're literally just protecting map
juggling...
- The locking goes away when we were already under an appropriate fmut
lock.
- Where we had fmut.RLock()+pmut.Lock() it gets upgraded to an
fmut.Lock().
- Otherwise s/pmut/fmut/.
In order to avoid diff noise for an important change I did not do the
following cleanups, which will be filed in a PR after this one, if
accepted:
- Renaming fmut to just mut
- Renaming methods that refer to being "PRLocked" and stuff like that
- Removing the no longer relevant deadlock detector
- Comments referring to pmut and locking sequences...
`syncthing cli` subcommand was using urfave/cli as the command parser.
This PR replace it with kong, which the main command uses.
Some help texts and error message format are changed. Other than that,
all the command usage and logic remains unchanged.
There's only one place which still uses urfave/cli, which is `syncthing
cli config`, because it uses some magic to dynamically build commands
from struct reflects. I used kong's `passthrough:""` tag to pass any
argument following `syncthing cli config` to urfave/cli parser.
This PR also fixes#9041
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This pull request allows syncthing to request an IPv6
[pinhole](https://en.wikipedia.org/wiki/Firewall_pinhole), addressing
issue #7406. This helps users who prefer to use IPv6 for hosting their
services or are forced to do so because of
[CGNAT](https://en.wikipedia.org/wiki/Carrier-grade_NAT). Otherwise,
such users would have to configure their firewall manually to allow
syncthing traffic to pass through while IPv4 users can use UPnP to take
care of network configuration already.
### Testing
I have tested this in a virtual machine setup with miniupnpd running on
the virtualized router. It successfully added an IPv6 pinhole when used
with IPv6 only, an IPv4 port mapping when used with IPv4 only and both
when dual-stack (IPv4 and IPv6) is used.
Automated tests could be added for SOAP responses from the router but
automatically testing this with a real network is likely infeasible.
### Documentation
https://docs.syncthing.net/users/firewall.html could be updated to
mention the fact that UPnP now works with IPv6, although this change is
more "behind the scenes".
---------
Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
Co-authored-by: André Colomb <github.com@andre.colomb.de>
gui: Show folder/device status on small screens
On larger screens, folder and device status is shown in a textual form
directly next to folder and device titles. However, on small screens,
only icons are currently shown, which may be ambiguous to new users, who
cannot possibly know what a specific icon means (see [1]). Thus, on
small screens only, display a new entry in folder/device info that
contains the same textual information that is shown in the title on
larger screens.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: André Colomb <src@andre.colomb.de>
Before introducing the service map and using it for folder runners, the
entries in folderCfgs and folderRunners for the same key/folder were
removed under a single lock. Stopping the folder happens separately
before that with just the read lock. Now with the service map stopping
the folder and removing it from the map is a single operation. And that
still happens with just a read-lock. However even with a full lock it’s
still problematic: After the folder stopped, the runner isn’t present
anymore while the folder-config still is and sais the folder isn't
paused.
The index handler in turn looks at the folder config that is not paused,
thus assumes the runner has to be present -> nil deref on the runner.
A better solution might be to push most of these fmut maps into the
folder - they anyway are needed in there. Then there's just a single
map/source of info that's necessarily consistent. That's quite a bit of
work though, and probably/likely there will be corner cases there too.
This reverts commit e477777f49.
In principle, we could have stayed with `~1.21.1`, but `check-latest:
true` apparently checks some cache/manifest/something that is only
periodically refreshed and isn't aware of 1.21.5 yet. So update the
constraints to force an upgrade.
Also the infrastructure images weren't actually using the constraint
since there was no `setup-go` action...
This reduces allocations, in number and in size, while getting extended
attributes. This is mostly noticable when there is a large number of new
files to scan and we're running with the default scanProgressInterval --
then a queue of files is built in-memory, and this queue includes
extended attributes as part of file metadata. (Arguable it shouldn't,
but that's a more difficult and involved change.)
With 1M files to scan, each with one extended attribute, current peak
memory usage looks like this:
Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total
Dropped 1435 nodes (cum <= 7.26MB)
Showing top 10 nodes out of 54
flat flat% sum% cum cum%
976.56MB 67.27% 67.27% 976.56MB 67.27%
github.com/syncthing/syncthing/lib/fs.getXattr
305.44MB 21.04% 88.31% 305.44MB 21.04%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.78MB 3.15% 91.47% 1045.23MB 72.00%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
22.89MB 1.58% 93.04% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 1.58% 94.62% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
16MB 1.10% 95.72% 16.01MB 1.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
After the change, it's this:
Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total
Dropped 1400 nodes (cum <= 2.62MB)
Showing top 10 nodes out of 91
flat flat% sum% cum cum%
305.43MB 58.19% 58.19% 305.43MB 58.19%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.79MB 8.72% 66.91% 68.68MB 13.09%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
32MB 6.10% 73.01% 32.01MB 6.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
22.89MB 4.36% 77.37% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 4.36% 81.73% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
15.35MB 2.92% 84.66% 15.36MB 2.93%
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
15.28MB 2.91% 87.57% 15.28MB 2.91% strings.(*Builder).grow
(The usage for xattrs is reduced from 976 MB to 68 MB)
LastSeen for a device was only updated when they connected. This now
updates it when they disconnect, so that we remember the last time we
actually saw them. When asking the API for current stats, currently
connected devices get a last seen value of the current time.
The string is currently hard-coded in English, so allow to translate it
into other languages. It appears mainly in the browser title bar before
logging in into the GUI or when the GUI is still loading.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds our short device ID to the basic auth realm. This has at least
two consequences:
- It is different from what's presented by another device on the same
address (e.g., if I use SSH forwards to different dives on the same
local address), preventing credentials for one from being sent to
another.
- It is different from what we did previously, meaning we avoid cached
credentials from old versions interfering with the new login flow.
I don't *think* there should be things that depend on our precise realm
string, so this shouldn't break any existing setups...
Sneakily this also changes the session cookie and CSRF name, because I
think `id.Short().String()` is nicer than `id.String()[:5]` and the
short ID is two characters longer. That's also not a problem...
### Purpose
The OCI image spec specifies well-defined
[annotations](https://github.com/opencontainers/image-spec/blob/main/annotations.md)
that can be added to images.
Theses annotations can then be used by other tools to gather more
information of an image.
This PR adds the `org.opencontainers.image.source` to allow tools such
as [renovate](https://github.com/renovatebot/renovate) to find the
release notes of a give version.
~~I've only done this change for `Dockerfile`. Should I also add the
label to the other dockerfiles?~~
I've now added the source annotations to all `Dockerfile`s & action
workflows.
### Testing
None, change was done by following the [renovate
documentation](https://docs.renovatebot.com/modules/datasource/docker/).
Following up on #9192, this makes use of the new mechanism for the theme
names.
The dummy string added for testing is removed again here. All
translations are updated to the new nested syntax, except Chinese
(zh-HK) where the string weren't actually translated.
Some translations, especially single words or other short
labels for buttons and the like, may not be transferable between
contexts even if they happen to be equal in English. In these cases,
setting an explicit translation ID is important for context separation.
Angular Translate also supports nested JSON in translation tables,
addressed using `.` as namespace separator; this enhancement makes use
of this when extracting translation with an explicit translation ID.
We use `env.GO_VERSION` as cache key for the build cache, but this is
nowadays typically something like `~1.21.1` which doesn't change when
1.21.2, 1.21.3 etc are released, making the cache fairly useless as
everything gets rebuilt. This re-sets the `GO_VERSION` variable after
installing Go so that it contains the actual installed version.
### Purpose
Treat X-Forwarded-For as a comma-separated string to prevent nil IP being returned by the Discovery Server
### Testing
Unit Tests implemented
Testing with a Discovery Client can be done as follows:
```
A simple example to replicate this entails running Discovery with HTTP, use Nginx as a reverse proxy and hardcode (as an example) a list of IPs in the X-Forwarded-For header.
1. Send an Announcement with tcp://0.0.0.0:<some-port>
2. Query the DeviceID
3. Observe the returned IP Address is no longer nil; i.e. `tcp://<nil>:<some-port>`
```
Add an id attribute to the submit button shown on the login form. This
allows my password manager's form filling function to interact with the
button after filling in username and password (which already have the id
attribute in place).
This makes the new default $XDG_STATE_HOME/syncthing or
~/.local/state/syncthing, while still looking in legacy locations first
for existing installs.
Note that this does not *move* existing installs, and nor should we.
Existing paths will continue to be used as-is, but the user can move the
dir into the new place if they want to use it (as they could prior to
this change as well, for that matter).
### Documentation
Needs update to the config docs about our default locations.
lib/fs: Fix conflicts on Android due to fluctuating inode change time
[1] added inode change time to file info in order to support syncing
extended attributes. However, in the case of Android, this inode change
time fluctuates, leading to unexpected conflicts even when the user has
not even touched the files on the Android device itself. Thus, in order
to prevent those conflicts from happening, do not write inode change
time on Android.
[1] 6cac308bcd
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
### Purpose
Deduplicated files are apparently considered 'irregular' under the hood,
this causes them to simply be ignored by Syncthing. This change is more
of a workaround than a proper fix, as the fix should probably happen in
the underlying libraries? - which may take some time. In the meanwhile,
this change should make deduplicated files be treated as regular files
and be indexed and synced as they should.
### Testing
Create some volume where deduplication is turned on (see the relevant
issue for details, including a proper description of how to reproduce
it). Prior to this change, the deduplicated files were simply ignored
(even by the indexer). After this change, the deduplicated files are
being index and synced properly.
This is motivated by the Android app:
https://github.com/syncthing/syncthing-android/pull/1982#issuecomment-1752042554
The planned fix in response to basic auth behaviour changing in #8757
was to add the `Authorization` header when opening the WebView, but it
turns out the function used only applies the header to the initial page
load, not any subsequent script loads or AJAX calls. The
`basicAuthAndSessionMiddleware` checks for no-auth exceptions before
checking the `Authorization` header, so the header has no effect on the
initial page load since the `/` path is a no-auth exception. Thus the
Android app fails to log in when opening the WebView.
This changes the order of checks in `basicAuthAndSessionMiddleware` so
that the `Authorization` header is always checked if present, and a
session cookie is set if it is valid. Only after that does the
middleware fall back to checking for no-auth exceptions.
`api_test.go` has been expanded with additional checks:
- Check that a session cookie is set whenever correct basic auth is
provided.
- Check that a session cookie is not set when basic auth is incorrect.
- Check that a session cookie is not set when authenticating with an API
token (either via `X-Api-Key` or `Authorization: Bearer`).
And an additional test case:
- Check that requests to `/` always succeed, but receive a session
cookie when correct basic auth is provided.
I have manually verified that
- The new assertions fail if the `createSession` call is removed in
`basicAuthAndSessionMiddleware`.
- The new test cases in e6e4df4d70 fail
before the change in 0e47d37e73 is
applied.
improve parsing of gui-address overrides
make checks for whether the gui-address is overridden consistent by
checking whether the environment variable is set and not an empty
string. the `Network()` function however checked for the inclusion of
a slash instead of the presence of any characters. If the config file's
gui address was set to a unix socket and the gui override to a tcp
address, then the function would have wrongly returned "unix".
the `URL()` function always returned the config file's gui address if a
unix socket was configured, even if an override was specified.
the `URL()` function wrongly formatted unix addresses. the http(s)
protocol was used as the sheme and the path was percent escaped. because
of the previous bug, this could only be triggered if the config file's
gui address was tcp and an unix socket override was given.
simplify the `useTLS()` function's codepath for overrides.
Co-authored-by: digital <didev@dinid.net>
Add a new entry to the unfolded device info to inform the user that the
device has been marked as "untrusted" and all folders shared with it
have to be password-protected or already Receive Encrypted.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Add a new entry to the unfolded device info to inform the user that the
device has Auto Accept enabled.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Opening and hiding multiple modals at the same time as well as opening a
modal before fully hiding the previous one can lead to the body padding
infinitely increasing by the scrollbar width each time, with the only
way to fix it being refreshing the GUI.
Therefore, always try to ensure to open and hide multiple modals one by
one, and also that the previous modal has fully been hidden before
proceeding to open the next one. The most common case when this problem
happens is when saving config changes which displays a GUI blocking
modal that overlaps, e.g. with folder or device modals that have not
been hidden yet.
Ref: https://github.com/twbs/bootstrap/issues/3902#issuecomment-1547187799
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Because $scope is missing, there are JavaScript errors when ticking and
unticking the "Untrusted" checkbox in the Advanced tab of the Edit
Device modal.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
I don't really understand under what circumstances, but sometimes these
calls panic with a "panic: counter cannot decrease in value" because the
value passed to Add() was negative.
Currently, the UI is always blocked from modifications when changes are
being saved, even if the save process takes very little time. This leads
to a situation where showing and closing the blocking modal can take
more time than is actually required to perform the whole operation. The
modal opening and closing very quickly can also cause the screen to
flash for a brief moment, leading to visual discomfort.
Because of this, wait for at least 200 ms and only show the blocking
modal if the changes have not been saved until then yet. The value of
200 ms is loosely based on [1] which states that 'a delay of 0.2–1.0
seconds does mean that users notice the delay and thus feel the computer
is "working" on the command, as opposed to having the command be a
direct effect of the users' actions.' Additionally, the delay must not
be too long, because the main purpose of the blocking modal is to
prevent the user from making further changes, and a longer delay would
possibly allow to do so in that brief amount of time as long as the user
is quick enough with their input.
[1] https://nngroup.com/articles/response-times-3-important-limits
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
In principle a connection can close while it's in progress with
starting, and then it's undefined if we wait for goroutines to exit etc.
With this change, we will wait for start to complete before starting to
stop everything.
gui: Remove unused hard-coded styles from globalChangesModalView modal
Currently, the globalChangesModalView modal has hardcoded th and td
styles. However, they are not even used in the modal itself, because
Bootstrap overrides them with its own styles for these elements in the
same modal. Yet, when hard-coded like that, these styles can conflict
with other table elements in the GUI. Thus, remove them completely.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds the ability to have multiple concurrent connections to a single device. This is primarily useful when the network has multiple physical links for aggregated bandwidth. A single connection will never see a higher rate than a single link can give, but multiple connections are load-balanced over multiple links.
It is also incidentally useful for older multi-core CPUs, where bandwidth could be limited by the TLS performance of a single CPU core -- using multiple connections achieves concurrency in the required crypto calculations...
Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: tomasz1986 <twilczynski@naver.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
Instead of separately tracking the token.
Also changes serviceMap to have a channel version of RemoveAndWait, so
that it's possible to do the removal under a lock but wait outside of
the lock. And changed where we do that in connection close, reversing
the change that happened when I added the serviceMap in 40b3b9ad1.
Currently, some of the information for folders and devices displayed in
the GUI relies on arbitrary values that come pre-set as defaults on a
fresh Syncthing installation, i.e. if the value matches the default, it
is hidden, and if does not, then it is displayed.
With this change, the GUI always displays all information regardless
of their value, making the overall experience more consistent and
predictable.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Currently, different icons are used for File Versioning when displayed
in the unfolded folder info in the main part of the GUI, and the icon
used in the Edit Folder modal. This changes the main GUI icon to match
the icon used in the modal.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The Logs icon was changed in [1] in the header, however the icon used in
the modal was left out. This changes it, so that the header and the
modal icons match.
[1] 2abba1dfb0
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Summarize platforms that fail to build, without overloading the build
log with errors that we anyway ignore. (Currently freebsd/riscv64 fails
to build.)
* gui: Remove footer and move links to header (fixes#5607)
Currently, the footer is always present and takes space at the bottom of
the GUI. However, the links listed there are not part of everyday user
interaction, and as such, they unnecessarily clutter the page, reducing
the usable screen space. Thus, transform the current Help link in the
header into a Help dropdown menu, and move the links from the footer
into it.
Also apply the following tweaks:
1. Move the About dialog from Actions to Help.
2. Add an Introduction (to the GUI) link to Help.
3. Change the Support icon from a question mark to a group of people.
4. Change the Changelog and About icons to a filled version to match the
other icons better.
5. Use a source code icon for Source Code instead of a wrench icon, and
move the wrench icon to Logs. This is done to prevent Changelog and
Logs from using the same icon.
6. Update all dropdown icons' Fork Awesome styles to "fa fa-fw <icon>".
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* a few more Fork Awesome style updates
---------
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The CSS method to select device IDs on click was added in [1]. However,
it was later mistakenly overwritten by [2]. This commit fixes the
regression and also applies the same behaviour to the Edit Device modal
which was omitted in the original commit.
[1] 5baf5fedb5
[2] 5e384c9185
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
By creating the http.Transport and tls.Configuration ourselves we
override some default behavior and end up with a client that speaks only
HTTP/1.1.
This adds a call to http.ConfigureTransport to do the relevant magic to
enable HTTP/2.
Also tweaks the keepalive settings to be a little kinder to the
server(s).
The allowed IPv4 ranges are the same as before. But we now also accept IPv6 addresses in the ULA range FC00::/7. These addresses don't require an interface identifier and are roughly equivalent to the IPv4 private ranges.
Typical usecases:
VPN interface IPs: Wireguard, OpenVPN, Tailscale, ...
fixed IPv6 LAN addressing while the provider assigns a dynamic prefix. e.g used by pihole
https://cs.opensource.google/go/go/+/refs/tags/go1.21.0:src/net/ip.go;l=146
* lib/versioner: Factor out DefaultPath constant.
Replace several instances where .stversions is named literally to all
use the same definition in the versioner package. Exceptions are the
packages where a cyclic dependency on versioner is impossible, or some
tests which combine the versions base path with other components.
* lib/versioner: Fix comment about trash can in simple versioner.
* lib/versioner: Fix wrong versioning type string in error message.
The error message shows the folder type instead of the versioning
type, although the correct field is used in the comparison.
Safety check added in v1.23.6 introduced bug. Bug unshares folders with untrusted devices if folder does not have an encryption password set, regardless of whether the folder is shared with the untrusted device as encrypted or not. Prevents sharing with untrusted devices in some cases where sharing would be encrypted.
Patch preserves safety check but permits sharing folders with untrusted devices if they are shared as encrypted.
Signed-off-by: kewiha <keithh@protonmail.com>
Currently, the versions filter is case-sensitive regardless of the
underlying OS. With this change, the filter becomes case-insensitive
everywhere, which is more user-friendly and makes it easier to search
for files whose exact case the user may not remember.
In addition, forward and backslashes are no longer distinguished,
whether used as path separators or as part of a file / directory
name (which is unlikely but possible on some platforms).
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
As per Bootstrap recommendation, buttons with tooltips inside button
groups require to have container: 'body' set. This prevents tooltips
from causing the buttons to jump on hover and also allows the tooltips
to be wider instead of wrapping on every space.
Ref: https://getbootstrap.com/docs/3.3/components/#btn-groups
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Currently, historically, we look for the `X-API-Key` header to
authenticate with an API key. There's nothing wrong with this, but in
some scenarios it's easier to produce an `Authorization` header with a
`Bearer $token` content, which is nowadays more common. This change adds
support for both, so that we will accept an API key either in our custom
header or as a bearer token.
Currently, because of devices with unset RTC clock, the 100% percentile
for Uptime on [1] is calculated since the Unix epoch which is useless as
far as usage statistics are concerned. Thus, if the Syncthing start time
is set to a past date, assume that the clock is wrong and do not even
try to report the uptime.
[1] https://data.syncthing.net
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
refactor: replace empty slice literal with `var`
An empty slice can be represented by `nil` or an empty slice literal. They are
functionally equivalent — their `len` and `cap` are both zero — but the `nil`
slice is the preferred style. For more information about empty slices,
see [Declaring Empty Slices](https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices).
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
refactor: fix unused method receiver
Methods with unused receivers can be a symptom of unfinished refactoring or a bug. To keep
the same method signature, omit the receiver name or '_' as it is unused.
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
refactor: unused parameter should be replaced by underscore
Unused parameters in functions or methods should be replaced with `_`
(underscore) or removed.
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
This adds an environment variable STVERSIONEXTRA that, when set, gets
added to the version information in the API and GUI.
The purpose of all this is to be able to communicate something about the
bundling or packaging, through the log & GUI and the end user, to the
potential person supporting it -- i.e., us. :) A wrapper can set this
variable to indicate that Syncthing is being run via `SyncTrayzor`,
`Syncthing-macOS`, etc., and thus indicate to the end user that the GUI
they are looking at is perhaps not the only source of truth and
management for this instance.
With this change, error messages include the offending characters or
name parts. Examples:
nul.txt: name is invalid, contains Windows reserved name: "nul"
foo>bar.txt: name is invalid, contains Windows reserved character: ">"
foo \bar.txt: name is invalid, must not end in space or period on Windows
This adds builds for the discovery and relay servers in the same manner
as Syncthing, so that Docker images are kept up to date with releases.
Inspired by and closes#8834.
Co-authored-by: Migelo <miha@filetki.si>
This adds builds for the discovery and relay servers in the same manner
as Syncthing, so that Docker images are kept up to date with releases.
Inspired by and closes#8834.
Co-authored-by: Migelo <miha@filetki.si>
This removes the tilde expansion we had in the GUI. I'm not sure why we
had it, but there are valid reasons to have a folder path with tilde in
it in the config, and it was previously impossible to enter one of
those.
This allows environment overrides for our directories. This is
advantageous because, apart from the obvious, it means we can set it in
the Docker file and not add command line options there. Having the
command line option as we did meant that it was impossible to use the
Docker image for other commands than `serve` (because that is implied
when we see other options on the command line).
This prevents combining untrusted with introducer and auto-accept, and
also verifies that folders shared with untrusted devices have passwords
at config loading time.
Co-authored-by: Simon Frei <freisim93@gmail.com>
We usually want to ensure that our own device is present. However if the
given device ID is the empty ID, we shouldn't do that. This is a
legimate (though way too non-obvious) use-case when opening the config
without knowing/caring about the device ID.
* Platform data (ownership, xattrs, etc.) is now set correctly for newly-received folders, even if the received folder has the NoPermissions flag.
* Call setPlatformData on receivers that have ignorePerms set to true.
This fixes various test issues with Go 1.20.
- Most tests rewritten to use fakefs where possible
- Some tests that were already skipped, or dubious (invasive,
unmaintainable, unclear what they even tested) have been removed
- Some actual code rewritten to better support testing in fakefs
Co-authored-by: Eric P <eric@kastelo.net>
In the sequence of loading ignores, the error File Does Not Exist is not being considered a fatal error, since the .stignore file is allowed to not exist. However, included ignore files also tossed that same error in case those do not exist while in those cases it's considered an error and it should lead to the folder stopping. Changing the error when opening an included ignore file to something other than the regular does fix this issue, as in it now works again as described in the Documentation.
This makes the various protocol priorities configurable among the other
options. With this, it's possible to prefer QUIC over TCP for WAN
connections, for example. Both sides need to be similarly configured for
this to work properly.
The default priority order remains the same as previously (TCP, QUIC,
Relay, with LAN better than WAN).
To make this happen I made each dialer & listener more priority aware,
and moved the check for whether a connection is LAN or not into the
dialer / listener -- this is the new "lanChecker" type that's passed
around.
In the original fix in #8563 I simply forgot this. Which meant #8556
wasn't actually fixed, as the trialer size would have been 0 (default),
and thus we would have still sent the inflated size to encrypted peers.
lib/model: Fix file size inconsisency due to enc. trailer
Fixes a regression due to PR #8563, while arguable the bug was actually
introduced in a much older PR #7155, but didn't have any bad effects so
far:
We account for the encryption trailer in the db updater routine,
calculating the file-info size there. However there's no guarantee that
the file-info at this point is still the exact same as when it was
written. It was before, but isn't anymore since introducing the new
EncryptedTrailerSize field.
Fix: Adjust the size in the info at the same place where the trailer is
written, i.e. we definitely have the actual size on disk.
The problem was that a statistics/cleanup run is triggered when the
database started and runs concurrently with the test. That cleanup run
removes old entries without valid addresses, and one of the test objects
matched this. The test object would thus randomly be removed in the
middle of the test, causing a failure. This fixes it so the object looks
recent when the cleaner-upper looks, and also uses a RAM database
(faster).
* gui: Import latest state from Weblate.
Downloaded 2023-03-23 at 20:38:53.
* gui: Remove untranslated strings from JSON files.
The GUI code will fall back to the English string translation anyway,
so keeping it identical inside the other language files does not
help. It makes the filter handling on Weblate easier though.
Current state retreived with these filter settings:
state:>=translated OR (state:needs-editing AND NOT check:same)
The layout of the request differs based on whether it comes from an
untrusted device or a trusted device with encrypted enabled. Handle
both.
Closes#8819.
Allow the watcher delay to take fractional values, effectively allowing
for much shorter delays. The minimum value is limited at 0.01, which
effectively translates to 10ms. This is required in order to guarantee
that there is still enough time to aggregate multiple single change
events.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds a cache to the expensive key generation operations. It's fixes
size LRU/MRU stuff to keep memory usage bounded under absurd conditions.
Also closes#8600.
Currently, the name and date filters in the Restore Versions modal are
always enabled, even if there are no versioned files present. With this
change, they are enabled only when there are no errors and versioned
files actually exist.
In addition to disabling the filters, also completely skip date picker
generation, as it serves no function while still displaying a non-sense
date range, which starts today and ends in the past.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds the BlocksHash field from the FileInfo to our API output. It
can be useful for debugging, or for external tools. I'm intentionally
leaving it as an opaque base64 string because no meaning should be
derived from it: it's just a string.
The reference comes from fd0a6225aa,
but the file itself was removed in the process of working on the pull
request, yet the reference to it was still left in the index.html.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
gui: Remove duplicate Spanish (Spain) translation.
Currently not among the set of "valid" (sufficiently complete)
languages and no longer synced with the tranlation platform, Weblate.
* Normalize EOL before EOF.
* Import current Transifex state, only translated.
Skip duplicated and completely empty languages.
* Re-add untranslated strings to JSON files.
These are now treated correctly as untranslated, but present with the
original (English) translation key.
* Update valid language codes and names.
* Refer to Weblate instead of Transifex for translations.
* lang-es: Copy untranslated strings from es-ES.
* Integrate translation changes from Weblate.
Translated using Weblate (Polish)
Currently translated at 100.0% (512 of 512 strings)
Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/pl/
Translated using Weblate (Danish)
Currently translated at 98.6% (505 of 512 strings)
Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/da/
Translated using Weblate (Lithuanian)
Currently translated at 85.4% (441 of 516 strings)
Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/lt/
Translated using Weblate (French)
Currently translated at 100.0% (516 of 516 strings)
Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/fr/
Translated using Weblate (Danish)
Currently translated at 98.6% (509 of 516 strings)
Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/da/
Translated using Weblate (Spanish)
Currently translated at 100.0% (516 of 516 strings)
Translation: Syncthing/GUI strings
Translate-URL: https://hosted.weblate.org/projects/syncthing/gui/es/
* translation update uses weblate
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This makes sure the service manager doesn't interpret timeout errors, or any other error, as a signal to stop the service instead of restarting it.
I added it directly to our service utility function, as it may help catch other instances of the same problem... We would typically want timeouts etc to be a retryable error, unless it is the top level context that has timed out and we check for that specifically.
Currently, action and type are both displayed only in English. This
commit makes it possible to translate both of them.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The current code assumes that lastSeenDays is always set which is not
the case when device reports its lastSeen as equal to the Unix epoch.
Thus, make sure that lastSeenDays is set before proceeding with the
disconnected-inactive status check to avoid JS errors in the browser.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds a word to the version string when running containerized. The
purpose is mostly to facilitate troubleshooting via screenshot by
"leaking" this rather important aspect of the setup. Additionally, the
version row gets "no-overflow-ellipsis" treatment so that the whole
thing is actually visible in the GUI and the (now useless) tooltip is
removed. In production releases this won't make a difference as the
whole thing will typically fit, but in odd setups it provides more info
up front.
The authorship script didn't pick up people who were only ever
"co-authors" of a commit, such as when they wrote stuff which was later
included in a PR by someone else, or added code during code review.
This modified the script to look closer in the commit bodies for
"Co-authored-by:"-lines and adds those found to the set of authors.
gui: Remove unmaintained language variant nl-BE.
This has long been removed from Transifex and from the valid-langs.js
list, but somehow the translation file stayed in the repo. There is
already a generic Dutch variant (nl) available as replacement.
On systems with safe umasks (`umask 077`), the entrypoint as copied from
the host may not be executable by other users. Ensure that it is set to
be within the Dockerfile.
* lib/connections: Cache isLAN decision for later external access.
The check whether a remote device's address is on a local network
currently happens when handling the Hello message, to configure the
limiters. Save the result to the ConnectionInfo and pass it out as
part of the model's ConnectionInfo struct in ConnectionStats().
* gui: Use provided connection attribute to distinguish LAN / WAN.
Replace the dumb IP address check which didn't catch common cases and
actually could contradict what the backend decided. That could have
been confusing if the GUI says WAN, but the limiter is not actually
applied because the backend thinks it's a LAN.
Add strings for QUIC and relay connections to also differentiate
between LAN and WAN.
* gui: Redefine reception level icons for all connection types.
Move the mapping to the JS code, as it is much easier to handle
multiple switch cases by fall-through there.
QUIC is regarded no less than TCP anymore. LAN and WAN make the
difference between levels 4 / 3 and 2 / 1:
{TCP,QUIC} LAN --> {TCP,QUIC} WAN --> Relay LAN --> Relay WAN -->
Disconnected.
Without this, we tag the build as made by some random user account on some random host name which is not useful.
(And minor bug in the cache key which has no effect on the build itself.)
We had some unholy mix of our own logger and the stdlib logger, probably
because for historical reasons we wanted the device ID to stdout and the
rest to stderr? But that's not the case any more, and the mix of formats
is weird. Ideally I think the generate command should be silent and just
print the device ID and nothing else, but that's tricky to accomplish
since we have other methods do logging on their own. Hence this just
harmonizes it so that we at least use the same logger with the same
format and target...
gui: Add copy to clipboard, share by email, and share by SMS buttons to device IDs (fixes#2771, ref #3868)
Add buttons to allow for simpler sharing device IDs with others. The
first one copies the ID to clipboard (trying to use three different
methods, depending on the browser). The second one triggers a mailto
link with prefilled subject and body. The third one triggers an sms link
with prefilled body. The short description of Syncthing included in the
latter part of the body is a direct copy from the description at the
official website https://syncthing.net.
Issue #3868 is referred here, because the copy to clipboard button
offers an alternative method for IE11 users to actually be able to copy
device IDs without having to select it manually (which doesn't work).
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This is sort of a proof of concept, but since our current Windows
builder is down this might solve the problem. It includes a change for
easier code signing (taking the certificate in a secret/env var rather
than existing already on disk), but otherwise mirrors precisely what we
already do in the build server.
The connection type icon comes from Bootstrap. As such, it does not
follow the same dimensions as the other GUI icons, which come from Fork
Awesome. Thus, add left and right margin to make its width roughly the
same as the other GUI icons, which fixes its alignment in relation to
text.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Currently, a disconnected device is marked as "Disconnected" without any
consideration whether the state is just temporary or rather it has been
like that for a long time, potentially requiring user intervention.
This commit adds a new state called "Disconnected (Inactive)" which is
shown for devices that have been disconnected for longer than a week. It
also changes the "Last seen" information, so that "Never" is used only
when the device hasn't connected at all, and the exact date is displayed
otherwise.
Additionally, when the date is older than 1 week, a note about that is
displayed below the date. Furthermore, when the date becomes older than
1 month, the note text colour changes to orange, and when it exceeds 1
year, the colour changes again to red. This is done to provide the user
with a visual clue that something may potentially be wrong and requires
a manual fix.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: André Colomb <src@andre.colomb.de>
Right now, the Out of Sync Items list for a specific remote device is
available only when the device is online and currently synchronising.
However, the list itself is being created by Syncthing all the time, so
there is no need to hide it from the user even when the remote device is
offline or paused. This way they can always easily check which files
exactly have not been synchronised yet.
In addition, the Out of Sync Items entry already includes file size next
to it, so there is no need to display it again next to the Sync Status
entry. Thus, remove out-of-sync file size from Sync Status, leaving only
percentage next to it.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This is sort of a proof of concept, but since our current Windows
builder is down this might solve the problem. It includes a change for
easier code signing (taking the certificate in a secret/env var rather
than existing already on disk), but otherwise mirrors precisely what we
already do in the build server.
In large setups, it is currently impossible to know the exact number of
folders and remote devices without counting them manually, either in the
GUI or in config.xml. Thus, to provide this information to the user, add
a specific number right next to both Folders and Remote Devices headers
in the Web GUI. The numbers are only displayed when two or more folders
or devices are present.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Previous debug input didn't really give enough info to show what was
happening, while it also printed full block lists which are enormously
verbose. Now it consistently prints 1. what it sees on disk, 2. what it
got from CurrentFile (without blocks), 3. the action taken on that file.
Some WebKit browsers select more than needed when using double click to
select device IDs, e.g. new lines and white space. This commit adds a
prefixed version of user-select in CSS in order to add support for those
browsers and allow them to select just device IDs automatically.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
There are some situations where an upgrade wouldn't be supported, even though the noUpgrade bool isn't set. So when handling the errors that are caused by this, when attempting an upgrade, it shouldn't lead to some sort of offline-message/restart/warning/etc...
I added some checks on specific errors related to this and return a 501 (Not Implemented) response instead, in case of an "UpgradeUnsupported"-error. Additionally, on the GUI-side, the 501-response is now not to be considered an error to act upon.
* implement authentication via token for relaysrv
Make replaysrv check for a token before allowing clients to
join. The token can be set via the replay-uri.
* fix formatting
* key composite literal
* do not error out if auth material is provided but not needed
* remove unused method receiver
* clean up unused parameter in functions
* cleaner token handling, disable joining the pool if token is set.
* Keep backwards compatibility with older clients.
In prior versions of the protocol JoinRelayRequest did not have a
token field. Trying to unmarshal such a request will result in
an error. Return an empty JoinRelayRequest, that is a request
without token, instead.
Co-authored-by: entity0xfe <entity0xfe@my.domain>
Currently, the link to remote GUI uses device addresses as they are
advertised from the router. Because of this, they may end up having a
scope ID attached to them. The problem is that browsers do not support
such addresses, leaving the user with a non-working URL.
Because of the above, use regex to simply filter out the scope ID from
the address before using it for Remote GUI.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Currently, the QR code image has no size specified in the HTML. This
causes other elements of the modal to jump around when opening it for
the first time and waiting for the QR image to generate. The jumping is
especially noticeable when the GUI is accessed remotely and there is a
delay while loading the elements due to slow connection.
In addition, capitalise "QR" in the alt text. This is necessary for
accessibility reasons, e.g. when using a screen reader, which may read
upper- and lowercase characters differently.
Lastly, allow to translate the alt text itself.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Currently, a custom JS script is used to select the whole device ID on
click. However, the current script isn't compatible with all browsers
(and in IE in particular), making it impossible to select the ID in them
at all. Additionally, the same functionality is already available in CSS
with no such drawbacks, as the whole selection process is handled by the
Web browser natively, which is lightweight and does not require custom
code.
Thus, remove the currently used JS script completely, replacing it with
a new CSS class that can be added to an element when required. If the
browser does not support the CSS, the user can still select the element
manually, which makes it safer than the current behaviour that can block
the user from being able to select the element at all.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The restore function of Trash Can ran a rename at the end regardless of whether there was anything to rename. In this case, when the file-to-be-restored did not exist in the destination folder, this resulted in an error. I added a simple check, keeping track of whether the file existed prior to restoring it in the destination folder and depending on this value it will now return nil after the restoration to prevent the renaming function to kick off. Added a test for this specific edge-case as well.
For consistency with syncOwnership explanation right above this one, the
sentence should talk about "sending ownership information", and not just
"sending ownership".
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Duplicate the regular expression for single and double quotes.
Support additional arguments (string substitution) in both variants.
Simplify the translation string group matching by using a lazy
quantifier instead of excluding the quote itself.
This adds support for syncing extended attributes on supported
filesystem on Linux, macOS, FreeBSD and NetBSD. Windows is currently
excluded because the APIs seem onerous and annoying and frankly the uses
cases seem few and far between. On Unixes this also covers ACLs as those
are stored as extended attributes.
Similar to ownership syncing this will optional & opt-in, which two
settings controlling the main behavior: one to "sync" xattrs (read &
write) and another one to "scan" xattrs (only read them so other devices
can "sync" them, but not apply any locally).
Co-authored-by: Tomasz Wilczyński <twilczynski@naver.com>
The cleaning logic in util.go was used by Simple and Trashcan but only
really suited Trashcan since it works based on mtimes which Simple does
not use. The cleaning logic in util.go was moved to trashcan.go.
Staggered and Simple seemed to be able to benefit from the same base so
util.go now has the base for those two with an added parameter which
takes a function so it can still handle versioner-specific logic to
decide which files to clean up. Simple now also correctly cleans files
based on their time-stamp in the title together with a specific maximum
amount to keep. The Archive function in Simple.go was changed to get rid
of duplicated code.
Additionally the trashcan testcase which was used by Trashcan as well as
Simple was moved from versioner_test.go to trashcan_test.go to keep it
clean, there was no need to keep it in a separate test file
Currently, the code contains a "mobile phone" fix to allow wrapping of
long lines in table heading and cells. However, the fix is applied to
all screen sizes equal or below 768 px wide, which causes the layout to
break on tablet-sized screens.
The commit moves the "mobile" fix to the actual mobile media query,
which is applied to screens up to 419 px wide. It is only really needed
there, where it synergises with the existing fix that changes table cell
display to "block". There is no need to wrap the text on larger screens,
as there is more than enough space to display the lines in full on them.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Add a commented entry to the systemd service file templates to point
the user in the right direction when using syncOwnership and starting
via systemd. Which is more upgrade-friendly than setting caps on the
executable directly, as mentioned in the docs.
The current code checks whether the same-named item exists in the tree,
and when it does, it re-uses it when adding new children to it. However,
the code doesn't check whether the existing item is a folder or a file.
It rather assumes that it is always a folder, which is not necessarily
the case.
This commit adds a new check to the code, so that the existing element
is reused only when it is a folder, and ignored otherwise.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Currently, the error message is often quite long and thus it appears
truncated with no possibility for the user to view the full string.
Thus, add a tooltip that displays the message in full on hover. This
follows the convention used in other parts of the GUI in similar cases.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This replaces old style errors.Wrap with modern fmt.Errorf and removes
the (direct) dependency on github.com/pkg/errors. A couple of cases are
adjusted by hand as previously errors.Wrap(nil, ...) would return nil,
which is not what fmt.Errorf does.
Locally paused folders will fail on checkFolderRunningLocked() and
therefore abort the loop. Avoid this by skipping paused folders
directly.
Co-authored-by: Jakob Borg <jakob@kastelo.net>
These strings no longer exist in the GUI, hence there is no need to keep
them in the translation json files either.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
* lib/locations: Fix enum values camelCase.
* lib/locations: Remove unused FailuresFile.
* cmd/syncthing: Turn around role of locations storage.
Previously the locations package was used to provide default paths,
possibly with an overridden home directory. Extra paths supplied on
the command line were handled and passed around in the options object.
To make the changed paths available to any other interested package,
override the location setting from the option if supplied, instead of
vice versa when not supplied. Adapt code using this to read from the
locations package instead of passing through the options object.
* lib/locations: Refactor showPaths to locations package.
Generate a reusable string in locations.PrettyPrintPaths().
Enumerating all possible locations in different packages is error
prone, so add a new public function to generate the listing as a
string in the locations package. Adapt cmd/syncthing --paths to use
that instead of its own console output.
* lib/locations: Include CSRF token in pretty printed paths.
* lib/api: New endpoint /rest/system/paths.
The paths should be available for troubleshooting from a running
instance. Using the --paths CLI option is not easy in some
environments, so expose the locations mapping to a JSON endpoint.
Add utility function ListExpandedPaths() that also filters out any
entries which still contain variable placeholders.
* gui: List runtime paths in separate log viewer tab.
* Wrap paths.
* lib/syncthing: Utilize locations.Get() instead of passing an arg.
* Include base directories, move label to table caption.
* gui: Switch to hard-coded paths instead of iterating over all.
* gui: Break aboutModalView into tabs.
Use tabs to separate authors from included third-party software.
* gui: Move paths from log viewer to about modal.
* lib/locations: Adjust pretty print output order to match GUI.
* gui, authors: Remove additional bot names and fix indent.
The indentation changed because of the tabbed about dialog, fix the
authors script to respect that.
Skip Syncthing*Automation in authors list as well.
* Update AUTHORS list to remove bot names.
* Revert AUTHORS email order change.
* Do not emphasize DB and log file locations.
* Review line wrapping.
* review part 1: strings.Builder, naming
* Rename and extend locations.Set() with error handling.
Remodel the Override() function along the existing SetBaseDir() and
rename it to simply Set(). Make sure to use absolute paths when given
log file or GUI assets override options. Add proper error reporting
if that goes wrong.
* Remove obsolete comment about empty logfile option.
* Don't filter out unexpanded baseDir placeholders, only ${timestamp}.
* Restore behavior regarding special "-" logfile argument.
If the option is given, but with empty value, assume the no log
file (same as "-"). Don't try to convert the special value to an
absolute path though and document this fact in a comment for the Set()
function.
* Use template to check for location key validity.
* Don't filter out timestamp placeholders.
* lib/api: Remove paths from /rest/system/status.
* lib/ur: Properly initialize map in failure data (fixes#8479)
Co-authored-by: Jakob Borg <jakob@kastelo.net>
Currently the `Run()` function of the CLI always uses `os.Args` directly.
This change adds an additional `RunWithArgs()` function that allows passing
arguments as `[]string` instead. This is useful when linking against
Syncthing as a library to be able to also expose the CLI.
Also adds idle time and keepalive parameters because how this is
configured has changed in the new package version. The values are those
that seems like might already be default, if keep-alives were enabled,
which is not obvious from the doc comments.
Also, Go 1.19 gofmt reformatting of comments.
all: Add package runtimeos for runtime.GOOS comparisons
I grew tired of hand written string comparisons. This adds generated
constants for the GOOS values, and predefined Is$OS constants that can
be iffed on. In a couple of places I rewrote trivial switch:es to if:s,
and added Illumos where we checked for Solaris (because they are
effectively the same, and if we're going to target one of them that
would be Illumos...).
Currently, there are two issues with the detailed staggered versioning
information displayed in the folder info. Firstly, the maxAge value of
365d is displayed despite matching the default. Secondly, there is no
consideration for the special case of maxAge equal to 0, meaning that
versions are kept forever.
This commit fixes both of those issues, so that the default maxAge is
not displayed and its value of 0 is displayed as "forever".
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds support for syncing ownership on Unixes and on Windows. The
scanner always picks up ownership information, but it is not applied
unless the new folder option "Sync Ownership" is set.
Ownership data is stored in a new FileInfo field called "platform data". This
is intended to hold further platform-specific data in the future
(specifically, extended attributes), which is why the whole design is a
bit overkill for just ownership.
* lib/model: Override scan config for auto-accepted encrypted folders.
Encrypted folders should not have the fs watcher enabled and rarely
benefit from a scheduled rescan. The GUI adjusts the suggested
settings (watcher disabled, one day rescan interval) when accepting a
receive-encrypted folder. Mirror that behavior to the auto-accept
case where the GUI is not involved.
Versioning also does not work well for encrypted folders, same
treatment.
Currently, the filesystem watcher explanation in the Advanced tab in the
Edit Folder modal window is split into two parts. The first is location
in a tooltip that is visible only when hovering the mouse over the small
checkbox, which makes it not easily accessible, e.g. for new or touch
screen users.
No other settings in the Edit Folder window have their explanation
presentend like that. When needed, it is always displayed in their
respective help blocks, which are always visible on the screen. Thus,
move the filesystem watcher explanation from the tooltip to the help
block, merging it with the other part of the explanation in the process.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* gui: Use discovered IDs from cache when adding a new remote device.
The GUI already does refreshDiscoveryCache() when it comes online and
on every 10 second refresh cycle. Querying the same information
freshly in addDevice() seems unnecessary and adds some more round-trip
delay on a possibly slow network link. Instead use the IDs from the
existing discoveryCache property to populate the suggested ID list.
* Increase maximum suggested device ID count to 100.
For the auto-completion list, which is hidden by default and filtered
by the browser, we can offer more discovered device IDs without
causing much confusion. The list of suggested "nearby" devices is
still limited to the first five.
* Rename $scope.discoveryUnknown.
The old name "discovery" was pretty ambiguous..
This commit updates Fork Awesome from version 1.1.2 from 2018 to version
1.2.0 from 2021. The changes are few and non-breaking, consisting mainly
of new icon additions.
It is worth to note that the new version includes also a Syncthing icon,
which was not present before.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
An off-by-one error could cause tokens to be forgotten. Suppose
tokens := []string{"foo", "bar", "baz", "quux"}
i := 2
token := tokens[i] // token == "baz"
Then, after
copy(tokens[1:], tokens[:i+1])
tokens[0] = token
we have
tokens == []string{"baz", "foo", "bar", "baz"}
The short test actually relied on this bug.
Similar to the "remote has not accepted sharing" message, add a
footnote number 2 to indicate a folder which will not sync with a
certain device because the remote has paused it. Applies to the edit
folder / device modals' sharing tab, as well as the "shared with"
listing and tooltips under device and folder details.
Use the proper encoding function in the relay server when constructing
the URL. In the pool server, parse and re-encode the query values to
sanitize whatever the client sent.
Currently, the "Latest Change" translation string is hard-coded to use
the English-like word order of V ("deleted") + N ("file"). As such, it
is incompatible with languages that require to use a different word
order, e.g. Korean or Japanese. In other words, a proper translation of
the string to those languages is currently impossible.
This commit changes the translation string, so that it includes the file
variable, and thanks to this, it can be easily adopted to languages with
different word order than English.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Showing all folders from disconnected or paused remote devices as
unaccepted would be a lot of false positives. As we cannot know
whether the remote has accepted while it doesn't have an active
connection, let's better report false negatives, as in assuming the
folders are accepted.
* lib/api: Note ItemStarted and ItemFinished for default filtering.
The reasoning why LocalChangeDetected and RemoteChangeDetected events
are not included in the event stream by default (without explicit
filter mask requested) also holds for the ItemStarted and ItemFinished
events. They should be excluded as well when we start to break the
API compatibility for some reason.
* gui: Enumerate unused event types in the eventService.
Define constants for the unused event types as well, for completeness'
sake. They are intentionally not handled in the GUI currently.
* cmd/syncthing: Harmonize uppercase CLI argument placeholders.
Use ALL-UPPERCASE and connecting dashes to distinguish argument
placeholders from literal argument options (e.g. "cpu" or "heap" for
profiling). The dash makes it clear which words form a single
argument and where a new argument starts.
This style is already used for the "syncthing cli debug file" command.
* lib/model: Simplify event data structure.
Using map[string]interface{} is not necessary when all values are
known to be strings.
Having a separate mutex for the three or four instructions needed to
fetch and increment nextID means the overhead exceeds the cost of this
operation. nextID is now handled inside the critical section for
awaiting instead, while the more expensive channel creation has been
moved outside it.
This is mostly a simplification, though it may have minor performance
benefits in some situations. The single-threaded sender benchmark shows
no significant difference:
name old speed new speed delta
RequestsRawTCP-8 55.3MB/s ± 7% 56.6MB/s ± 6% ~ (p=0.190 n=10+10)
RequestsTLSoTCP-8 20.5MB/s ±20% 20.8MB/s ± 8% ~ (p=0.604 n=10+9)
Add a link next to each setting's label to its explanation in the Docs.
This way, it is easy to quickly check what the setting is about without
going to the Docs site separately and searching for it manually.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Right now, the Trash Can versioning info text is displayed for both the
Trash Can and Simple versioning. However, the Simple versioning has its
own info text that is also displayed, which results in two very similar
sentences being shown on the screen.
This commit limits the Trash Can info text to be displayed only when the
Trash Can versioning is selected, and moves the Simple versioning info
text up to the same location as the other one.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Even though technically possible, CJK languages normally don't use
italic text at all, as not only does it make the characters/letters look
unnatural, but also, in the case of complex characters, unreadable too.
For these reasons, it is usually recommended not to use the italic font
style at all [1][2].
This commit changes the default font-style of the i element for Chinese,
Japanese, and Korean langauge to "normal" instead of "italic". In order
to do so, the HTML lang attribute is also changed following each change
of the GUI language.
[1] https://bobtung.medium.com/best-practice-in-chinese-layout-f933aff1728f
[2] https://devblogs.microsoft.com/oldnewthing/20060914-02/?p=29743
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Staggered File Versioning used to have its own cleanInterval that
controlled how often file versions were cleaned. Nowadays, there is a
seperate setting called cleanupIntervalS responsible for the cleanup,
which applies to all File Versioning (except External). Thus, remove the
unneeded code and don't set the param up on new folders anymore.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* gui: Allow to translate and fix incorrect Versions date filter ranges
Translate the previously English-only ranges used in Versions date
filter. In the process, fix the currently incorrect range calculation.
For instance, let us say it is 08:05. Selecting "today" should set the
range to start at 00:00 and end at 08:05. However, what really happens
is that both start and end are set to 08:05, and as a result "today" is
never shown to the user. The case is the same for "yesterday", which in
contrary to "today" is shown, but its range is fixed at 08:05 on the
previous day.
This commit fixes the above by always calculating "today" starting at
00:00 up to the current moment, and calculating "yesterday" from 00:00
on the previous to 00:00 on the current day.
When it comes to "last x days", the commit fixes the calculation so that
it actually covers the whole day, which is done by moving the start date
to 00:00 on the first day.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The ignoreDelete warning code originally comprised of two sentences,
which in total were much longer than the current one. Because of that,
some additional fixes were added to make it present itself better, e.g.
white-space was set to normal to allow for text wrap on narrow screens,
and the help link was moved to a new line to make sure both the anchor
and the question mark always stayed together.
However, the code fixes remained despite the final text being much
shorter than the original. Therefore, remove the unnecessary white-space
formatting, and also move the help link right next to the warning. This
makes it clear that the link is related to this particular warning when
both it and the ignore patterns warning are displayed at the same time.
Ref: https://github.com/syncthing/syncthing/pull/8054#issuecomment-978148117
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* gui: Translate fancytree messages in Versions modal
Currently, the default fancytree info/error messages are used. This
means that a) they are English-only, and b) they are very generic. With
this commit, the messages are added to the translatable strings, and
they are also more specifically related to file versioning.
On a side note, the "moreData" string has been left out on purpose, as
it is not used in the current code. It can be added later if needed.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* gui: Make ID read-only and hide nearby devices when adding pending devices (fixes#8083)
Currently, there is no distinction between adding "new" and "pending"
devices. For this reason, the user is always presented with a list of
"nearby devices" to choose from. This commit adds such distinction to
the code, and in the case of "pending" device, both the device ID is
made read-only and the nearby devices list is hidden.
As a by-product of the function rename and clean-up, this commit also
hides the non-functional "remove" button that is shown when editing
device defaults.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The intention was that if no peers are given, we shouldn't start the
listener. We did that anyway, because:
- splitting an empty string on comma returns a slice with one empty
string in it
- parsing the empty string as a device ID returns the empty device ID
so we end up with a valid replication peer which is the empty device ID.
* lib/protocol: Require at least 3.125% savings from compression
The new lz4 library doesn't need its output buffer to be the maximum
size, unlike the old one (which would allocate if it weren't). It can
take a buffer that is of a smaller size and will report if compressed
data can fit inside the buffer (with a small chance of reporting a false
negative). Use that property to our advantage by requiring compressed
data to be at most n-n/32 = .96875*n bytes long for n input bytes.
* lib/protocol: Remove unused receivers
To make DeepSource happy.
* lib/protocol: Micro-optimize lz4Compress
Only write the length if compression was successful. This is a memory
write, so the compiler can't reorder it.
Only check the return value of lz4.CompressBlock. Length-zero inputs
are always expanded by LZ4 compression (the library documents this),
so the check on len(src) isn't needed.
When GOBIN is set, 'go install' cannot install cross-compilied binaries.
To satisfy cross-compilation, it's necessary to add the '-o' to build
target, otherwise 'go build' will discarding the resulting objects when
compiling multiple packages.
Signed-off-by: bekcpear <i@bitbili.net>
* lib/model: Remove bogus fields from connections API endpoint.
Switch the returned data type for the /rest/system/connections element
"total" to use only the Statistics struct. The other fields of the
ConnectionInfo struct are not populated and misleading.
* Lowercase JSON field names.
* lib/model: Get rid of ConnectionInfo.MarshalJSON().
It was missing the StartedAt field from the embedded Statistics
struct. Just lowercasing the JSON attribute names can be done just as
easily with annotations.
* lib/model: Remove bogus startedAt field from totals.
Instead of using the Statistics type with one field empty, just switch
to a free-form map with the three needed fields.
* cmd/syncthing: Remove unnecessary function arguments.
The openGUI() function does not need a device ID to work, and there is
only one caller anyway which uses EmptyDeviceID.
The loadOrDefaultConfig() function is always called with the same
dummy values.
* cmd/syncthing: Avoid misleading info messages from monitor process.
In order to check whether panic reporting is enabled, the monitor
process utilizes the loadOrDefaultConfig() function. In case there is
no config file yet, info messages may be logged during creation if the
config Wrapper, which is discarded immediately after.
Stop using the DefaultConfig() utility function from lib/syncthing and
directly generate a minimal config instead to avoid these.
Add comments to loadOrDefaultConfig() explaining its limited purpose.
* cmd/syncthing/generate: Always write updated config file.
Previously, an existing config file was left untouched unless either
of the --gui-user or --gui-password options was given. Remove that
condition and simplify the checking code.
* lib/config: Factor out ProbeFreePorts().
* cmd/syncthing: Add option --skip-port-probing.
Applies to both the "generate" and "serve" subcommands, as well as the
deprecated --generate option, just as the --no-default-folder flag.
* cmd/syncthing: Remove unnecessary function arguments.
The openGUI() function does not need a device ID to work, and there is
only one caller anyway which uses EmptyDeviceID.
The loadOrDefaultConfig() function is always called with the same
dummy values.
* cmd/syncthing: Avoid misleading info messages from monitor process.
In order to check whether panic reporting is enabled, the monitor
process utilizes the loadOrDefaultConfig() function. In case there is
no config file yet, info messages may be logged during creation if the
config Wrapper, which is discarded immediately after.
Stop using the DefaultConfig() utility function from lib/syncthing and
directly generate a minimal config instead to avoid these.
Add comments to loadOrDefaultConfig() explaining its limited purpose.
* cmd/syncthing/generate: Always write updated config file.
Previously, an existing config file was left untouched unless either
of the --gui-user or --gui-password options was given. Remove that
condition and simplify the checking code.
Use indexOf instead of startsWith to make the now translatable theme
names appear correctly in IE11. This also prevents console log error
spam in the browser. The same problem was previously reported in #6940.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
When pathSep is a constant, the compiler precomputes pathSep+pathSep and
".."+pathSep instead of emitting function calls to compute "//" and
"../". Benchmark results in lib/osutil:
name old time/op new time/op delta
TraversesSymlink-8 8.86µs ± 3% 8.53µs ± 4% -3.79% (p=0.000 n=18+20)
name old alloc/op new alloc/op delta
TraversesSymlink-8 1.06kB ± 0% 1.06kB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
TraversesSymlink-8 15.0 ± 0% 15.0 ± 0% ~ (all equal)
The locking protocol in nat.Mapping was racy:
* Mapping.addressMap RLock'd, but then returned a map shared between
caller and Mapping, so the lock didn't do anything.
* Operations inside Service.{verifyExistingMappings,acquireNewMappings}
would lock the map for every update, but that means callers to
Mapping.ExternalAddresses can be looping over the map while the
Service methods are concurrently modifying it. When the Go runtime
detects that happening, it panics.
* Mapping.expires was read and updated without locking.
The Service methods now lock the map once and release the lock only when
done.
Also, subscribers no longer get the added and removed addresses, because
none of them were using the information. This was changed for a previous
attempt to retain the fine-grained locking and not reverted because it
simplifies the code.
Accept a subcommand as an alternative to the --generate option. It
accepts a custom config directory through either the --home or
--config options, using the default location if neither is given.
Add the options --gui-user and --gui-password to "generate", but not
the "serve --generate" option form. If either is given, an existing
config will not abort the command, but rather load, modify and save it
with the new credentials. The password can be read from standard
input by passing only a single dash as argument.
Config modification is skipped if the value matches what's already in
the config.
* cmd/syncthing: Utilize lib/locations package in generate().
Instead of manually joining paths with "magic" file names, get them
from the centralized locations helper lib.
* cmd/syncthing: Simplify logging for --generate option.
Visible change: No more timestamp prefixes.
What hash is used to store the password should ideally be an
implementation detail, so that every user of the GUIConfiguration
object automatically agrees on how to handle it. That is currently
distribututed over the confighandler.go and api_auth.go files, plus
tests.
Add the SetHasedPassword() / CompareHashedPassword() API to keep the
hashing method encapsulated. Add a separate test for it and adjust
other users and tests. Remove all deprecated imports of the bcrypt
package.
LoadOrGenerateCertificate() takes two file path arguments, but then
uses the locations package to determine the actual path. Fix that
with a minimally invasive change, by using the arguments instead.
Factor out GenerateCertificate().
The only caller of this function is cmd/syncthing, which passes the
same values, so this is technically a no-op.
* lib/tlsutil: Make storing generated certificate optional. Avoid
temporary cert and key files in tests, keep cert in memory.
Consistently use double dashes and fix typos -conf, -data-dir and
-verify.
Applies also to tests running the syncthing binary for consistency.
* Fix mismatched option name --conf in cli subcommand.
According to the source code comments, the cli option flags should
mirror those from the serve subcommand where applicable. That one is
actually called --config though.
* cli: Fix help text option placeholders.
The urfave/cli package uses the Value field of StringFlag to provide a
default value, not to name the placeholder. That is instead done with
backticks around some part of the Usage field.
* cli: Add missing --data flag in subcommand help text.
The urfave/cli based option parsing uses a fake flags collection to
generate help texts matching the used global options. But the --data
option was omitted from it, although it is definitely required when
using --config as well. Note that it cannot just be ignored, as some
debug stuff actually uses the DB:
syncthing cli --data=/bar --config=/foo debug index dump
By truncating time.Time to an int64 nanosecond count, we lose the
ability to precisely order timestamps before 1678 or after 2262, but we
gain (linux/amd64, Go 1.17.1):
name old time/op new time/op delta
JobQueuePushPopDone10k-8 2.85ms ± 5% 2.29ms ± 2% -19.80% (p=0.000 n=20+18)
JobQueueBump-8 34.0µs ± 1% 29.8µs ± 1% -12.35% (p=0.000 n=19+19)
name old alloc/op new alloc/op delta
JobQueuePushPopDone10k-8 2.56MB ± 0% 1.76MB ± 0% -31.31% (p=0.000 n=18+13)
name old allocs/op new allocs/op delta
JobQueuePushPopDone10k-8 23.0 ± 0% 23.0 ± 0% ~ (all equal)
Results for BenchmarkJobQueueBump are with the fixed version, which no
longer depends on b.N for the amount of work performed. rand.Rand.Intn
is cheap at ~10ns per iteration.
Like Windows and Mac, Android is also an interactive operating system.
On top of that, it usually runs on much slower hardware than the other
two. Because of that, it makes sense to limit the number of hashes used
by default there too.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Add each subdirectory of the guiDir as a translation candidate string.
The key is prefixed with "theme-name-" and the default English
translation corresponds to the directory name turned to title case.
Disable the automatic name mangling in the GUI JS code in favor of
just looking up the translation.
Registry.Get used a full sort to get the minimum of a list, and the sort
was broken because util.AddressUnspecifiedLess assumed it could find out
whether an address is IPv4 or IPv6 from its Network method. However,
net.(TCP|UDP)Addr.Network always returns "tcp"/"udp".
Currently, the default sorting of the file list in the Restore Versions
modal in both case sensitive, and it also does not distinguish between
files and folders. With this change, which uses a slightly modified code
from the fancytree itself (with added folder prioritisation), the sort
becomes case insensitive and also always places folders on top. This is
to make it behave more similar to what file managers do in the commonly
used operating systems (including popular Linux desktop environments).
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This reverts commit cca17f5306.
Using textarea instead of input makes it possible to enter new lines,
which then are added to config.xml as "
". This breaks the whole
script, because these characters are passed to the command line as "\n".
Therefore, the script should rather remain a single-lined input field.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The current detection is flawed, because it looks for a few specific
file systems like "msdos" or "fat" to set the mtime window, while in
reality Android seems to report names like "fuseblk", which can stand
for fat, ext4, or even f2fs.
At the moment, we set the mtime window only for a few known names used
for the fat filesystem. With this change, we take a safer approach of
always setting the time window unless we explicitly detect file systems
like ext2/ext3/ex4, which are known not to experience issues with moving
timestamps on Android.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The list of unshared devices is built from deviceList(), which sorts
the result. But the shared devices are listed in undefined order from
the unsorted currentFolder.devices array. Apply the same sorting
after building the array used in the dialog.
Currently, the dismiss button is displayed as the first of the three
buttons. However, the most common action that the user wants to do when
sharing a new folder is to add it on a different device. Because of
this, the add button should be displayed first as the most prominent of
the three. The ignore button is the opposite of the add button, and also
results in a parmenent action, hence it makes sense to lump the two
together. Thus, the dismiss button should be moved to the last place as
an alternative to the two main actions, when the user is yet unsure what
they want to do with the notification.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
1. Change each modal title text to match the action that is being
executed (i.e. "Revert" to "Revert Local Additions", "Override" to
"Override Changes", "Delete" to "Delete Unexpected Items").
2. Change the icons to match the icons used by each action (i.e. arrow-
circle-down for Revert, arrow-circle-up for Override). Replace the
broken lock icon for Delete with minus-circle.
3. Rearrange the order in the modal HTML code to simplify it a little.
Before this patch, IPv4-compatible addresses (::ffff:aaa.bbb.ccc.ddd)
may be used if a quic6://some.domain:port is specified and both IPv4 and
IPv6 addresses exist for that domain name.
This changes the GC mechanism so that the first pass (which reads all
FileInfos to populate bloom filters with block & version hashes) can
happen concurrently with normal database operations.
The big gcMut still exists, and we grab it temporarily to block all
other modifications while we set up the bloom filters. We then release
the lock and let other things happen, with those other things also
updating the bloom filters as required. Once the first phase is done we
again grab the gcMut, knowing that we are the sole modifier of the
database, and do the cleanup.
I also removed the final compaction step.
* cmd/strelaypoolsrv: Fix minor grammar, use https in links
Add a few minor grammatical/stylistic fixes. Use `https` instead of
`http` in the MaxMind link in the footer.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* wip
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Disable the Versions button when the folder is paused, because it does
not work, i.e. the versioned files are not loaded. The folder needs to
be unpaused to actually be able to view the versioned file list.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* Trigger connection loop on config device addition (fixes#7600)
* Also check for device address equality
* Move EqualStrings from api_test to utils, and use in connections/service.go
* Make sure CommitConfiguration cannot block due on the deviceAddressesChanged channel
* Update lib/connections/service.go
Co-authored-by: Jakob Borg <jakob@kastelo.net>
* cmd/syncthing: Don't fail early on api setup error (fixes 7558)
* switch to factory pattern
* refactor config command to show help on nothing
* wip
* wip
* already abort in before
This makes us use TLS 1.3+ on sync connections by default. A new option
`insecureAllowOldTLSVersions` exists to allow communication with TLS
1.2-only clients (roughly Syncthing 1.2.2 and older). Even with that
option set you get a slightly simplified setup, with the cipher suite
order fixed instead of auto detected.
* cmd/syncthing: Don't fail early on api setup error (fixes 7558)
* switch to factory pattern
* refactor config command to show help on nothing
* wip
* wip
* already abort in before
No longer hide the web UI controls for the new untrusted/encrypted
device feature. Testing hasn't been very widespread, but there has been
some and quite a few bugs have been caught and fixed. I believe its time
to not hide it anymore, and cautiously recommend usage. E.g. mention
that the feature hasn't been widely used yet and anyone using it is an
early adopter, but drop the bit about not using it with production data.
We can maybe stress the need for backups in general and especially
using this.
Remove the animation due to its excessive CPU usage, especially when a
large number of files is being downloaded and listed at the same time.
Also, remove the stripes, as they serve no purpose in the now-static
progress bar.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
There was a logic mistake, so the limit in question wasn't used. On my
macOS this doesn't seem to matter, the hard limit returned is 2^63-1 and
setting the soft limit to that works. However I'm assuming that's not
the case for older macOSes since it was so nicely documented, so we
should still have this working. (10240 FDs should be enough for
anybody.)
This is a mostly pointless change to make security scanners and static
analysis tools happy, as they all hate seeing md5. None of our md5 uses
were security relevant, but still. Only visible effect of this change is
that our temp file names for very long file names become slightly longer
than they were previously...
The current text gives an impression that we are currently using a
Receive Encrypted folder, even if we are not. Thus, make the current
text displayed only when the folder is in fact Receive Encrypted, and
add a new string to be displayed when using different folder types.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds a couple of dummy asset files protected by the "noassets"
build tag. The purpose is that it should be possible for, for example,
CI tools and static analysis things to compile and analyze the source
tree without our custom asset generation step. Also makes `go test -tags
noassets ./...` work without building assets first.
Disabled options are currently barely distinguishable from enabled
ones. This changes their background to grey, following the Bootstrap
defaults already used for disabled <select>.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This truncates times meant for API consumption to second precision,
where fractions won't typically matter or add any value. Exception to
this is timestamps on logs and events, and of course I'm not touching
things like file metadata.
I'm not 100% certain this is an exhaustive change, but it's the things I
found by grepping and following the breadcrumbs from lib/api...
I also considered general-but-ugly solutions, like having the API
serializer itself do reflection magic or even regexps on returned
objects, but decided against it because aurgh...
This adds a button in the top right that changes the config back to the
default theme.
Code wise, it takes the header that was previously a part of the
dashboard component and moves it to the app component, and then adds the
button there. Possibly the header should be a component of it's own, but
that's more of refactor that can happen separately I think.
The config change uses the new config API to just patch the relevant
setting.
I'm not doing an automatic reload because 1) I don't want to figure out
how to do it correctly and 2) this doesn't work reliably anyway, as
for example the current gen GUI does a reload and ends up with
connection refused as the API service is still reloading...
* Provide a sysctl config to raise max UDP buffer size
* Add sysctl config to deb
* Check if `deb-systemd-invoke` is available
Co-authored-by: otbutz <tbutz@optitool.de>
This loosens the ‘is this localhost?’ check to include *.localhost host
names.
This allows for clearer (hence better) names to be used in browsers,
e.g. when accessing a remote syncthing instance ‘foo’ using a ssh port
forward, one can use foo.localhost to remind oneself which one is which.
💡 Without these changes, Syncthing shows a ‘Host check error’ when
pointing a browser at http://foo.localhost/, and with these changes, the
interface loads as usual.
The .localhost top level domain is a reserved top-level domain (RFC 2606):
> The ".localhost" TLD has traditionally been statically defined in
> host DNS implementations as having an A record pointing to the
> loop back IP address and is reserved for such use. Any other use
> would conflict with widely deployed code which assumes this use.
> – https://tools.ietf.org/html/rfc2606
As Wikipedia puts it:
> This allows the use of these names for either documentation purposes
or in local testing scenarios. – https://en.wikipedia.org/wiki/.localhost
On Linux systems, systemd-resolved resolves *.localhost, on purpose:
https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html
See also #4815, #4816.
* release:
all: Fix Microsoft documentation links in code comments (#7387)
gui: Handle info labels that are longer than available space (fixes#944) (#7386)
gui: Show "Last seen" at the top when device is disconnected (ref #7166) (#7373)
gui: Remove cleanupIntervalS leftovers from External Versioning (ref #7360) (#7378)
gui: Allow setting custom path for all versioning except external (#7377)
lib/fs: Consider options in case-fs caching (fixes#7371) (#7381)
lib/scanner: Pass on errors while hashing (#7380)
build: Update pfilter (#7376)
gui: Hide the Rescan All button when no folders exist (#7367)
lib/connections: Allow QUIC with Go 1.16 (#7372)
gui: Hide non-functional Versions button when using External Versioning (#7365)
gui, man, authors: Update docs, translations, and contributors
gui: Fix setting external versioning command (fixes#7361) (#7362)
lib/versioner: Improve error messages (fixes#7354) (#7357)
gui: Disable Cleanup Interval with External File Versioning as unsupported (#7360)
Apply to table headers the same code as already used for table data.
This way, the headers will be either pushed to the next line, or cut
with an ellipsis if the single word is too long.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Move the "Last seen" field to the very top in the device information.
This way, if a device has disconnected unexpectly, we can quickly check
the time when it was last available. Right now, due to the very long
address field, it is usually necessary to scroll down in order to view
the "Last seen" field.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
If there are no folders present, show only the "Add Folder" button, and
hide the "Rescan All" button. Only show the latter when at least one
folder exists.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Apply to table headers the same code as already used for table data.
This way, the headers will be either pushed to the next line, or cut
with an ellipsis if the single word is too long.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Move the "Last seen" field to the very top in the device information.
This way, if a device has disconnected unexpectly, we can quickly check
the time when it was last available. Right now, due to the very long
address field, it is usually necessary to scroll down in order to view
the "Last seen" field.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
If there are no folders present, show only the "Add Folder" button, and
hide the "Rescan All" button. Only show the latter when at least one
folder exists.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The button does nothing when the External Versioning is being used, so
it should not be displayed at all to avoid confusing the users.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
As for right now, the External File Versioning does not support Cleanup
Interval. Therefore, the option should no be available at all when using
it.
Ref: https://forum.syncthing.net/t/16346
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The button does nothing when the External Versioning is being used, so
it should not be displayed at all to avoid confusing the users.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
As for right now, the External File Versioning does not support Cleanup
Interval. Therefore, the option should no be available at all when using
it.
Ref: https://forum.syncthing.net/t/16346
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
An untrusted device will receive padded info for small blocks, and hence
sometimes request a larger block than actually exists on disk.
Previously we let this pass because we didn't have a hash to compare to
in that case and we ignored the EOF error based on that.
Now the untrusted device does pass an encrypted hash that we decrypt and
verify. This means we can't check for len(hash)==0 any more, but on the
other hand we do have a valid hash we can apply to the data we actually
read. If it matches then we don't need to worry about the read
supposedly being a bit short.
Benchmark results on Linux/amd64, using updated benchmark for old and
new:
name old time/op new time/op delta
HashFile-8 88.6ms ± 1% 88.3ms ± 1% -0.33% (p=0.046 n=19+19)
name old speed new speed delta
HashFile-8 201MB/s ± 1% 202MB/s ± 1% +0.33% (p=0.044 n=19+19)
name old alloc/op new alloc/op delta
HashFile-8 59.4kB ± 0% 46.1kB ± 0% -22.47% (p=0.000 n=14+20)
name old allocs/op new allocs/op delta
HashFile-8 29.0 ± 0% 27.0 ± 0% -6.90% (p=0.000 n=20+20)
Co-authored-by: greatroar <@>
* lib/db: Add ExpirePendingFolders().
Use-case is to drop any no-longer-pending folders for a specific
device when parsing its ClusterConfig message where previously offered
folders are not mentioned any more.
The timestamp in ObservedFolder is stored with only second precision,
so round to seconds here as well. This allows calling the function
within the same second of adding or updating entries.
* lib/model: Weed out pending folders when receiving ClusterConfig.
Filter the entries by timestamp, which must be newer than or equal to
the reception time of the ClusterConfig. For just mentioned ones,
this assumption will hold as AddOrUpdatePendingFolder() updates the
timestamp.
* lib/model, gui: Notify when one or more pending folders expired.
Introduce new event type FolderOfferCancelled and use it to trigger a
complete refreshCluster() cycle. Listing individual entries would be
much more code and probably just as much work to answer the API
request.
* lib/model: Add comment and rename ExpirePendingFolders().
* lib/events: Rename FolderOfferCancelled to ClusterPendingChanged.
* lib/model: Reuse ClusterPendingChanged event for cleanPending()
Changing the config does not necessarily mean that the
/resut/cluster/pending endpoints need to be refreshed, but only if
something was actually removed. Detect this and indicate it through
the ClusterPendingChanged event, which is already hooked up to requery
respective endpoints within the GUI.
No more need for a separate refreshCluster() in reaction to
ConfigSaved event or calling refreshConfig().
* lib/model: Gofmt.
* lib/db: Warn instead of info log for failed removal.
* gui: Fix pending notifications not loading on GUI start.
* lib/db: Use short device ID in log message.
* lib/db: Return list of expired folder IDs after deleting them.
* lib/model: Refactor Pending...Changed events.
* lib/model: Adjust format of removed pending folders enumeration.
Use an array of objects with device / folder ID properties, matching
the other places where it's used.
* lib/db: Drop invalid entries in RemovePendingFoldersBeforeTime().
* lib/model: Gofmt.
My local gofmt did not complain here, strangely...
* gui: Handle PendingDevicesChanged event.
Even though it currently only holds one device at a time, wrap the
contents in an array under the "added" property name.
* lib/model: Fix null values in PendingFoldersChanged removed member.
* gui: Handle PendingFoldersChanged event.
* lib/model: Simplify construction of expiredPendingList.
* lib/model: Reduce code duplication in cleanPending().
Use goto and a label for the common parts of calling the DB removal
function and building the event data part.
* lib/events, gui: Mark ...Rejected events deprecated.
Extend comments explaining the conditions when the replacement event
types are emitted.
* lib/model: Wrap removed devices in array of objects as well.
* lib/db: Use iter.Value() instead of needless db.Get(iter.Key())
* lib/db: Add comment explaining RemovePendingFoldersBeforeTime().
* lib/model: Rename fields folderID and deviceID in event data.
* lib/db: Only list actually expired IDs as removed.
Skip entries where Delete() failed as well as invalid entries that got
removed automatically.
* lib/model: Gofmt
This splits the ignore getting to two methods, one that loads from disk
(the old one) and one that just returns whatever is already loaded (the
new one). The folder summary service which is just interested in stats
now uses the latter method. This means that it, and API calls that call
it, does not get blocked by folder I/O.
This adds two new configuration options:
// The number of connections at which we stop trying to connect to more
// devices, zero meaning no limit. Does not affect incoming connections.
ConnectionLimitEnough int
// The maximum number of connections which we will allow in total, zero
// meaning no limit. Affects incoming connections and prevents
// attempting outgoing connections.
ConnectionLimitMax int
These can be used to limit the number of concurrent connections in
various ways.
This adds a statistic to track the last connection duration per device.
It isn't used for much in this PR, but it's available for #7223 to use
in deciding how to order device connection attempts (deprioritizing
devices that just dropped our connection the last time).
The test would fail if the umask on UNIX is greater than 0022, because
the OS transparently subtracts it from the mode passed to Mkdir(), as
the Go documentation confirms.
Our goal here is not to test os.Mkdir(), so just make sure the desired
mode is actually set by forcing it afterwards.
This breaks out some methods from the connection loop to make it simpler
to manage and understand.
Some slight simplifications to remove the `seen` variable (we can filter
`nextDial` based on times are in the future or not, so we don't need to
track `seen`) and adding a minimum loop interval (5s) in case some
dialer goes haywire and requests a 0s redial interval or such.
Otherwise no significant behavioral changes.
This does two things:
- Exclude QUIC from go1.16 builds, automatically, for now, since it
doesn't work and just panics.
- Provide some fake listeners and dialers when QUIC is disabled.
These fake listeners and dialers indicate that they are disabled and
unsupported, which silences "Dialing $address: unknown address scheme:
quic" type of stuff which is not super helpful to the user.
This fixes a regression by restoring two functions removed in commit
a20d85d451, but still used from the HTML
template. Adjust the restored versions to access the new flag storage
structure.
The replacement functions selectAllSharedFolders() and
selectAllUnrelatedFolders() are not functional, so this is just a
quick fix to restore a working state before making the new functions
actually work sensibly.
Just like the respective functions for sharing device selections, give
it a boolean argument instead of writing two almost identical
functions.
In line with the other selectAll...() functions, avoid calling
angular.forEach() and iterate over the folders object directly.
This fixes a regression by restoring two functions removed in commit
a20d85d451, but still used from the HTML
template. Adjust the restored versions to access the new flag storage
structure.
The replacement functions selectAllSharedFolders() and
selectAllUnrelatedFolders() are not functional, so this is just a
quick fix to restore a working state before making the new functions
actually work sensibly.
Just like the respective functions for sharing device selections, give
it a boolean argument instead of writing two almost identical
functions.
In line with the other selectAll...() functions, avoid calling
angular.forEach() and iterate over the folders object directly.
When using a Web browser with JavaScript either disabled or unavailable,
show a warning to let the user know that the Web GUI requires JS in
order to operate.
To achieve this, add a <div> that wraps both the navbar and the main
content, and then move the CSS class ng-cloak from the <html> element to
that <div>. This way, only the JavaScript-dependent part is hidden when
JS is unavailable, and not the whole website, as it is the case right
now. Then, add a <noscript> element right at the start of the <body>
element, so that the warning is also shown right away in text-based Web
browsers. The <noscript> element includes a stripped down version of the
navbar showing only the Syncthing logo, and then a container with the
warning itself. Lastly, leave the footer untouched and always visible,
because it does not rely on JavaScript at all.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This removes the switch for using a Badger database, because it has bugs
that it seems there is no interest in fixing, and no actual bug tracker
to track them in.
It retains the actual implementation for the sole purpose of being able
to do the conversion back to LevelDB if anyone is actually running with
USE_BADGER. At some point in a couple of versions we can remove the
implementation as well.
Add specific errors for the failures, resulting in this rather than just
the generic "invalid filename":
[MRIW7] 08:50:50 INFO: Puller (folder default, item "NUL"): syncing: filename is invalid: name is reserved
[MRIW7] 08:50:50 INFO: Puller (folder default, item "fail."): syncing: filename is invalid: name ends with space or period
[MRIW7] 08:50:50 INFO: Puller (folder default, item "sup:yo"): syncing: filename is invalid: name contains reserved character
[MRIW7] 08:50:50 INFO: default: Failed to sync 3 items
Use the IoctlFileClone and IoctlFileCloneRange ioctl wrappers and the
FileCloneRange type provided by golang.org/x/sys/unix instead of
locally implementing them. This also allows to re-enable the code for
ppc/ppc64/ppc64le again (see commit 758a1a6a37 ("lib/fs: Disable ioctl
on ppc (fixes#6898) (#6901)")) since golang.org/x/sys/unix internally
uses the correct FICLONE and FICLONERANGE values depending on $GOARCH.
Most notably, it now detects all-lowercase files and returns these
as-is. The tests have been expanded with two cases and are now used
as a benchmark (admittedly a rather trivial one).
name old time/op new time/op delta
UnicodeLowercaseMaybeChange-8 4.59µs ± 2% 4.57µs ± 1% ~ (p=0.197 n=10+10)
UnicodeLowercaseNoChange-8 3.26µs ± 1% 3.09µs ± 1% -5.27% (p=0.000 n=9+10)
This changes the cache to cache less things, yet retain the required
efficiency for our walk usecase. This uses less memory.
Specifically, instead of keeping result and child caches for each path
level, only keep a single cached child. In practice our operations are
depth-first, or almost depth-first, and then we retain the same hit
ratio for a smaller cache size.
I improved the benchmark so that it counts the Lstat and DirNames
operations performed, and they do not change significantly. The amount
of allocated memory is reduced by 20% and the walk itself is actually
slightly faster.
This also removes the clear based on number of cached names (as that is
not a thing any more) and the timer based clear (which was unused). This
means we'll retain the last cache state forever until it's cleared by a
write operation, but we did that before too and that state is now a lot
smaller...
The overhead compared to not using a casefs, for our typical "double
walk" (walk the tree then stat everything again) is 2x the dirnames we
would otherwise call, and no overhead on the stats (unchanged from old
implementation)
```
name old time/op new time/op delta
WalkCaseFakeFS100k/rawfs-8 306ms ± 1% 305ms ± 2% ~ (p=0.182 n=9+10)
WalkCaseFakeFS100k/casefs-8 579ms ± 5% 557ms ± 1% -3.77% (p=0.000 n=10+10)
name old B/entry new B/entry delta
WalkCaseFakeFS100k/rawfs-8 590 ± 0% 590 ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 1.09k ± 0% 0.87k ± 0% -19.98% (p=0.000 n=10+10)
name old DirNames/entry new DirNames/entry delta
WalkCaseFakeFS100k/rawfs-8 0.51 ± 0% 0.51 ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 1.02 ± 0% 1.02 ± 0% ~ (all equal)
name old DirNames/op new DirNames/op delta
WalkCaseFakeFS100k/rawfs-8 51.2k ± 0% 51.2k ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 102k ± 0% 102k ± 0% ~ (all equal)
name old Lstat/entry new Lstat/entry delta
WalkCaseFakeFS100k/rawfs-8 3.02 ± 0% 3.02 ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 3.02 ± 0% 3.02 ± 0% ~ (all equal)
name old Lstat/op new Lstat/op delta
WalkCaseFakeFS100k/rawfs-8 302k ± 0% 302k ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 302k ± 0% 302k ± 0% ~ (all equal)
name old allocs/entry new allocs/entry delta
WalkCaseFakeFS100k/rawfs-8 15.7 ± 0% 15.7 ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 27.5 ± 0% 26.1 ± 0% -5.09% (p=0.000 n=10+10)
name old ns/entry new ns/entry delta
WalkCaseFakeFS100k/rawfs-8 2.02k ± 1% 2.02k ± 2% ~ (p=0.163 n=9+10)
WalkCaseFakeFS100k/casefs-8 3.83k ± 5% 3.68k ± 1% -3.77% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
WalkCaseFakeFS100k/rawfs-8 89.2MB ± 0% 89.2MB ± 0% ~ (p=0.364 n=9+10)
WalkCaseFakeFS100k/casefs-8 164MB ± 0% 131MB ± 0% -19.97% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
WalkCaseFakeFS100k/rawfs-8 2.38M ± 0% 2.38M ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 4.16M ± 0% 3.95M ± 0% -5.05% (p=0.000 n=10+10)
```
Since iterators must be released before committing or discarding a
transaction we have the pattern of both deferring a release plus doing
it manually. But we can't release twice because we track this with a
WaitGroup that will panic when there are more Done()s than Add()s. This
just adds a boolean to let an iterator keep track.
Use indexOf() == 0 instead of startsWith() to maintain compatibility
and prevent JavaScript error console spam in the Web GUI when used in
Internet Explorer 11 under Windows 7.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
We created a new fileset before stopping the folder during restart. When
we create that fileset it loads the current metadata and sequence
numbers from the database. But the folder may have time to update those
before stopping, leaving the new fileset with bad data.
This would cause wrong accounting (forgotten files) and potentially
sequence reuse causing files not sent to other devices.
This change reuses the fileset on restart, skipping the issue entirely.
It also moves the creation of the fileset back under the lock so there
should be no chance of concurrency issues here.
We created a new fileset before stopping the folder during restart. When
we create that fileset it loads the current metadata and sequence
numbers from the database. But the folder may have time to update those
before stopping, leaving the new fileset with bad data.
This would cause wrong accounting (forgotten files) and potentially
sequence reuse causing files not sent to other devices.
This change reuses the fileset on restart, skipping the issue entirely.
It also moves the creation of the fileset back under the lock so there
should be no chance of concurrency issues here.
The FileSet.Drop operation in there needs to potentially update a whole lot of global lists, which can take a while (longer than the deadlock interval apparently)
When cap(permanentRelays) >= len(permanentRelays) + len(knownRelays),
append(permanentRelays, knownRelays...)
returns a slice of the array underlying permanentRelays. The subsequent
rand.Shuffle then mixes the permanent and known relays. Sequential
requests may cause strelaypoolsrv to forget its permanent relays. Worse,
concurrent requests may cause shuffling of the same slice on multiple
processors concurrently.
Co-authored-by: greatroar <@>
* Add clean up for Simple File Versioning pt.1
created test
* Add clean up for Simple File Versioning pt.2
Passing the test
* stuck on how javascript communicates with backend
* Add trash clean up for Simple File Versioning
Add trash clean up functionality of to allow the user to delete backups
after specified amount of days.
* Fixed html and js style
* Refactored cleanup test cases
Refactored cleanup test cases to one file and deleted duplicated code.
* Added copyright to test file
* Refactor folder cleanout to utility function
* change utility function to package private
* refactored utility function; fixed build errors
* Updated copyright year.
* refactor test and logging
* refactor html and js
* revert style change in html
* reverted changes in html and some js
* checkout origin head version edit...html
* checkout upstream master and correct file
Quoting the manual:
-trimpath
remove all file system paths from the resulting executable.
Instead of absolute file system paths, the recorded file names
will begin with either "go" (for the standard library),
or a module path@version (when using modules),
or a plain import path (when using GOPATH).
That is, when we panic, instead of:
goroutine 1 [running]:
main.main()
/Users/jb/dev/syncthing/syncthing/cmd/syncthing/main.go:272 +0x116
we get:
goroutine 1 [running]:
main.main()
github.com/syncthing/syncthing@/cmd/syncthing/main.go:272 +0x116
(Module path and file path within module.)
Our authentication is based on device ID (certificate fingerprint) but
we also check the certificate name for ... historical extra security
reasons. (I don't think this adds anything but it is what it is.) Since
that check breaks in Go 1.15 this change does two things:
- Adds a manual check for the peer certificate CommonName, and if they
are equal we are happy and don't call the more advanced
VerifyHostname() function. This allows our old style certificates to
still pass the check.
- Adds the cert name "syncthing" as a DNS SAN when generating the
certificate. This is the correct way nowadays and makes VerifyHostname()
happy in Go 1.15 as well, even without the above patch.
Apparently our Tags field depended on having specific files react to
tags and add themselves there. This, instead, works for all tags.
Also, pass tags to the test command line.
The QUIC package is notorious for being incompatible with either too
old or too new Go releases. Currently it doesn't build with Go 1.15 RC
and I want to test the rest with Go 1.15. With this I can do `go run
build.go --tags noquic` to do that.
With this change we emulate a case sensitive filesystem on top of
insensitive filesystems. This means we correctly pick up case-only renames
and throw a case conflict error when there would be multiple files differing
only in case.
This safety check has a small performance hit (about 20% more filesystem
operations when scanning for changes). The new advanced folder option
`caseSensitiveFS` can be used to disable the safety checks, retaining the
previous behavior on systems known to be fully case sensitive.
Co-authored-by: Jakob Borg <jakob@kastelo.net>
Prompted by https://forum.syncthing.net/t/infinite-filesystem-recursion-detected/15285. In my opinion the filesystem shouldn't throw warnings but pass on errors for the caller to decide what's to be happening with it. Right now in this PR an infinite recursion is a normal scan error, i.e. folder is in failed state and displays failed items, but no warning. I think that's appropriate but if deemed appropriate an additional warning can be thrown in the scanner.
This fixes the change in #6674 where the weak hash became a deciding
factor. Now we again just use it to accept a block, but don't take a
negative as meaning the block is bad.
This fixes the change in #6674 where the weak hash became a deciding
factor. Now we again just use it to accept a block, but don't take a
negative as meaning the block is bad.
If the GC finds a key k that it wants to keep, it records that in a
Bloom filter. If a key k' can be removed but its hash collides with k,
it will be kept. Since the old Bloom filter code was completely
deterministic, the next run would encounter the same collision, assuming
k must still be kept.
A randomized hash function that uses all the SHA-256 bits solves this
problem: the second run has a non-zero probability of removing k', as
long as the Bloom filter is not completely full.
* Fix ui, hide report date
* Undo Goland madness
* UR now web scale
* Fix migration
* Fix marshaling, force tick on start
* Fix tests
* Darwin build
* Split "all" build target, add package name as a tag
* Remove pq and sql dep from syncthing, split build targets
* Empty line
* Revert "Empty line"
This reverts commit f74af2b067.
* Revert "Remove pq and sql dep from syncthing, split build targets"
This reverts commit 8fc295ad00.
* Revert "Split "all" build target, add package name as a tag"
This reverts commit f4dc889951.
* Normalise contract types
* Fix build add more logging
This is shorter, skips two allocations, makes the function inlineable
and is safer, since the compiler now check whether
DeviceIDLength == sha256.Size.
Currently a random dev version has a version string like this:
v1.7.0-rc.1+23-gef3441bd6
That is, the tag name followed by a plus sign and the git describe
metadata (number of commits and hash) plus -dirty or -branchname in some
cases.
We introduced the plus sign in #473, where a dev version would
previously be called v0.9.0-42-gwhatever which is considered older than
v0.9.0 and hence caused a downgrade. The problem with the plus is that
per semver everything after the plus is ignored as build metadata, which
means we won't upgrade from v1.7.0-rc.1 to v1.7.0-rc.1+22-g946170f3f.
With this change the we instead either just add a dev suffix (if we're
already on a prerelease version) or we wind the patch version and add a
dev suffix.
v1.7.0-rc.1+23-gef3441bd6 => v1.7.0-rc.1.dev.23.gef3441bd6
v1.6.1+80-gef3441bd6 => v1.6.2-dev.80.gef3441bd6
This should preserve the ordering and keep versions semver-ish.
This changes the error handling in loading ignores slightly:
- There is a new ParseError type that is returned as the error
(somewhere in the chain) when the problem was not an I/O error loading
the file, but some issue with the contents.
- If the file was read successfully but not parsed successfully we still
return the lines read (in addition to nil patterns and a ParseError).
- In the API, if the error IsParseError then we return a successful
HTTP response with the lines and the actual error included in the JSON
object.
- In the GUI, as long as the HTTP call to load the ignores was
successful we can edit the ignores. If there was an error we show this
as a validation error on the dialog.
Also some cleanup on the Javascript side as it for some reason used
jQuery instead of Angular for this editor...
crypto/rand output is cryptographically secure by the Go library
documentation's promise. That, rather than strength (= passes randomness
tests) is the property that Syncthing needs).
This makes Go 1.15 test/vet happy, avoiding "conversion from untyped int
to string yields a string of one rune" warning where we do
string(KeyTypeWhatever) in namespaced.go.
It also clarifies and enforces the currently allowed range of these
numbers so I think it's fine.
This matches the convention of the stdlib and avoids ambiguity: when
customErr{} and &customErr{} both implement error, client code needs to
check for both.
Memory use should remain the same, since storing a non-pointer type in
an interface value still copies the value to the heap.
This extracts the extra tags from any `[foo]` stuff at the end of the
version and sends them to Sentry for indexing.
If I need to modify that regexp again I'll probably write a from scratch
tokenizer and parser for our version string instead...
This adds some env vars to the long version string as if they were build
tags. The purpose is to better understand what code was running or not
in the version output, usage reporting and crash reports. In order to
prevent possible privacy issues the actual value of the variable is not
reported, just the fact that it was set to something non-empty.
Example:
% ./bin/syncthing --version
syncthing v1.6.1+47-g1eb104f3-buildtags "Fermium Flea" (go1.14.3 darwin-amd64) jb@kvin.kastelo.net 2020-06-03 07:25:46 UTC [stnoupgrade, use_badger]
Group the global list of files by version, instead of having one flat list for all devices. This removes lots of duplicate protocol.Vectors.
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This reduces the size of our write batches before we flush them. This
has two effects: reducing the amount of data lost if we crash when
updating the database, and reducing the amount of memory used when we do
large updates without checkpoint (e.g., deleting a folder).
I ran our SyncManyFiles benchmark as it is the one doing most
transactions, however there was no relevant change in any metric (it's
limited by our fsync I expect). This is good as any visible change would
just be a decrease in performance.
I don't have a benchmark on deleting a large folder, taking that part on
trust for now...
* cmd/stindex: Unify access to key from cached variable.
Avoid calling the Key() method from the iterator each time the value
is needed. Just reuse the cache variable already assigned before the
switch block.
* cmd/stindex: Display the prefix byte value for unknown key types.
Make it easier to diagnose corrupt / unknown key type entries by
showing their decimal value, correlating with the definitions in
keyer.go.
* cmd/stindex: Add missing KeyType values in stindex dump code.
Recently added DB key prefixes KeyTypeBlockListMap and KeyTypeVersion
were unknown to the stindex dumping tool. Add basic parsing to dump
their key structure.
If we fail to take the rename shortcut we may crash on a later loop,
because we do trickiness with the indexes but the original buckets[key]
in "range buckets[key]" isn't re-evaluated so i exceeds the max index.
This adds indirection of large version vectors in the same manner as we
already to block lists. The effect is the same: less duplicated data in
some situations.
To mitigate the impact for when this indirection
wouldn't be needed I've added an indirection cutoff for both blocks and
the new version vector stuff: we don't do the indirection at all for
small block lists or small version vectors, instead storing it directly
like we used to do. This is faster for small files and small setups.
Storing assets as []byte requires every compiled-in asset to be copied
into writable memory at program startup. That currently takes up 1.6MB
per syncthing process. Strings stay in the RODATA section and should be
shared between processes running the same binary.
This makes version vector values clock based instead of just incremented
from zero. The effect is that a vector that is created from scratch
(after database reset) will have a higher value for the local device
than what it could have been previously, causing a conflict. That is, if
we are A and we had
{A: 42, B: 12}
in the old scheme, a reset and rescan would give us
{A: 1}
which is a strict ancestor of the older file (this might be wrong). With
the new scheme we would instead have
{A: someClockTime, b: otherClockTime}
and the new version after reset would become
{A: someClockTime+delta}
which is in conflict with the previous entry (better).
In case the clocks are wrong (current time is less than the value in the
vector) we fall back to just simple increment like today.
This scheme is ineffective if we suffer a database reset while at the
same time setting the clock back far into the past. It's however no
worse than what we already do.
This loses the ability to emit the "added" event, as we can't look for
the magic 1 entry any more. That event was however already broken
(#5541).
Another place where we infer meaning from the vector itself is in
receive only folders, but there the only criteria is that the vector is
one item long and includes just ourselves, which remains the case with
this change.
* wip
This changes the build script to build all the things in one go
invocation, instead of one invocation per cmd. This is a lot faster
because it means more things get compiled concurrently. It's especially
a lot faster when things *don't* need to be rebuilt, possibly because it
only needs to build the dependency map and such once instead of once per
binary.
In order for this to work we need to be able to pass the same ldflags to
all the binaries. This means we can't set the program name with an
ldflag.
When it needs to rebuild everything (go clean -cache):
( ./old-build -gocmd go1.14.2 build all 2> /dev/null; ) 65.82s user 11.28s system 574% cpu 13.409 total
( ./new-build -gocmd go1.14.2 build all 2> /dev/null; ) 63.26s user 7.12s system 1220% cpu 5.766 total
On a subsequent run (nothing to build, just link the binaries):
( ./old-build -gocmd go1.14.2 build all 2> /dev/null; ) 26.58s user 7.53s system 582% cpu 5.853 total
( ./new-build -gocmd go1.14.2 build all 2> /dev/null; ) 18.66s user 2.45s system 1090% cpu 1.935 total
This makes the GC runner a service that will stop fairly quickly when
told to.
As a bonus, STTRACE=app will print the service tree on the way out,
including any errors they've flagged.
* release:
lib/db: Don't get blocklists on drop and missing continue (ref #6457) (#6502)
Revert "cmd/syncthing: Do auto-upgrade before startup (fixes#6384) (#6385)"
lib/ur: Correct freaky error handling (fixes#6499) (#6500)
So, in a funny plot twist, it turns out that WriteFile in Go 1.13
doesn't actually set the read only bit on Windows when called with
permissions 0444 so my test was broken. With an improved test it turns
out that Rename does not, in fact, overwrite a read-only file on
Windows. This adds a fix for that.
(Rename might get improved in Go 1.15...)
Successful LRU cache lookups modify the cache's recency list, so
RWMutex.RLock isn't enough protection.
Secondarily, multiple concurrent lookups with the same key should not
create separate rate limiters, so release the lock only when presence
of the key in the cache has been ascertained.
Co-authored-by: greatroar <@>
This adds the functionality to run a user search with a filter for LDAP
authentication. The search is done after successful bind, as the binding
user. The typical use case is to limit authentication to users who are
member of a group or under a certain OU. For example, to only match
users in the "Syncthing" group in otherwise default Active Directory
set up for example.com:
<searchBaseDN>CN=Users,DC=example,DC=com</searchBaseDN>
<searchFilter>(&(sAMAccountName=%s)(memberOf=CN=Syncthing,CN=Users,DC=example,DC=com))</searchFilter>
The search filter is an "and" of two criteria (with the ampersand being
XML quoted),
- "(sAMAccountName=%s)" matches the user logging in
- "(memberOf=CN=Syncthing,CN=Users,DC=example,DC=com)" matches members
of the group in question.
Authentication will only proceed if the search filter matches precisely
one user.
The previous implementation was very generic; its tests didn't cover the
actual alphabet for device IDs.
Benchmark results on amd64:
name old time/op new time/op delta
Luhnify-8 1.00µs ± 1% 0.28µs ± 4% -72.38% (p=0.000 n=9+10)
Unluhnify-8 992ns ± 2% 274ns ± 1% -72.39% (p=0.000 n=10+9)
We set the STRESTART environment when starting the inner process after
the first time, but this didn't persist when restarting the monitor
process. Now it does.
As of the latest database checker we are again putting files without
blocks. I'm not 100% convinced that's a great idea, but we also do it
for ignored files apparently so it looks like we probably should support
it. This adds an escape hatch that must be manually enabled...
As of the latest database checker we are again putting files without
blocks. I'm not 100% convinced that's a great idea, but we also do it
for ignored files apparently so it looks like we probably should support
it. This adds an escape hatch that must be manually enabled...
* release:
lib/db, lib/syncthing: Repair db once on upgrade (ref #6425, #6427) (#6429)
lib/db: Fix removeFromGlobal and no filenames in error (fixes#6427) (#6428)
lib/db: Remove emptied global list in checkGlobals (fixes#6425) (#6426)
- In the few places where we wrap errors, use the new Go 1.13 "%w"
construction instead of %s or %v.
- Where we create errors with constant strings, consistently use
errors.New and not fmt.Errorf.
- Remove capitalization from errors in the few places where we had that.
If we decide to recalculate the metadata we shouldn't start from
whatever we loaded from the database, as that data is wrong. We should
start from a clean slate.
If we decide to recalculate the metadata we shouldn't start from
whatever we loaded from the database, as that data is wrong. We should
start from a clean slate.
I was working on indirecting version vectors, and that resulted in some
refactoring and improving the existing block indirection stuff. We may
or may not end up doing the version vector indirection, but I think
these changes are reasonable anyhow and will simplify the diff
significantly if we do go there. The main points are:
- A bunch of renaming to make the indirection and GC not about "blocks"
but about "indirection".
- Adding a cutoff so that we don't actually indirect for small block
lists. This gets us better performance when handling small files as it
cuts out the indirection for quite small loss in space efficiency.
- Being paranoid and always recalculating the hash on put. This costs
some CPU, but the consequences if a buggy or malicious implementation
silently substituted the block list by lying about the hash would be bad.
I was working on indirecting version vectors, and that resulted in some
refactoring and improving the existing block indirection stuff. We may
or may not end up doing the version vector indirection, but I think
these changes are reasonable anyhow and will simplify the diff
significantly if we do go there. The main points are:
- A bunch of renaming to make the indirection and GC not about "blocks"
but about "indirection".
- Adding a cutoff so that we don't actually indirect for small block
lists. This gets us better performance when handling small files as it
cuts out the indirection for quite small loss in space efficiency.
- Being paranoid and always recalculating the hash on put. This costs
some CPU, but the consequences if a buggy or malicious implementation
silently substituted the block list by lying about the hash would be bad.
One of the causes of "panic: database is closed" is that we try to send
summaries after it's been closed. Calculating summaries can take a long
time and if we have a lot of folders it's not unreasonable to think
that we might be stopped in this loop, so prepare to bail here.
* push
During NAT discovery we block for 10s (NatTimeoutS) before returning.
This is mostly noticeable when Ctrl-C:ing a Syncthing directly after
startup as we wait for those ten seconds before shutting down. This
makes it check the context a little bit more frequently.
This adds metadata updates to the same write batch as the underlying
file change. The odds of a metadata update going missing is greatly
reduced.
Bonus change: actually commit the transaction in recalcMeta.
lib/db: Recover sequence number and metadata on startup (fixes#6335)
If we crashed after writing new file entries but before updating
metadata in the database the sequence number and metadata will be wrong.
This fixes that.
We could potentially get a snapshot and then fail to get a releaser,
leaking the snapshot. This takes the releaser first and makes sure to
release it on snapshot error.
The readWriteTransaction offered both commit() (the one to use) and
Commit() (via embedding) where the latter didn't close the read
transaction. This removes the lower cased variant in order to prevent
the mistake.
The only place where the mistake was made was the new gc runner, where
it would leave a read snapshot open forever.
During some other work I discovered these tests weren't great, so I've
rewritten them to be a little better. The real changes here are:
- Don't play games with not starting the folder and such, and don't
construct a fake folder instance -- just use the one the model has. The
folder starts and scans but the folder contents are empty at this point
so that's fine.
- Use a fakefs instead of a temp dir.
- To support the above, implement a fakefs option `?content=true` to
make the fakefs actually retain written content. Use sparingly,
obviously, but it means the fakefs can usually be used instead of an
on disk real directory.
This adds a new config with the simple and concise name
maxConcurrentIncomingRequestKiB. This limits how many bytes we have "in
the air" in the form of response data being read and processed.
After some testing I think that not having this limiter is seldom a
great idea and thus I propose a default value of 256 MiB for this new
setting.
I also refactored the folder IO limiter to be a model/folder attribute
instead of a package global.
Adds a new folder state "Waiting to Sync" in the same vein as the
existing "Waiting to Scan". This vastly improves performances in the
rare cases when there are lots and lots of folders operating.
Also retain the interval over restarts by storing last GC time in the
database. This to make sure that GC eventually happens even if the
interval is configured to a long time (say, a month).
* lib/db: Deduplicate block lists in database (fixes#5898)
This moves the block list in the database out from being just a field on
the FileInfo to being an object of its own. When putting a FileInfo we
marshal the block list separately and store it keyed by the sha256 of
the marshalled block list. When getting, if we are not doing a
"truncated" get, we do an extra read and unmarshal for the block list.
Old block lists are cleared out by a periodic GC sweep. The alternative
would be to use refcounting, but:
- There is a larger risk of getting that wrong and either dropping a
block list in error or keeping them around forever.
- It's tricky with our current database, as we don't have dirty reads.
This means that if we update two FileInfos with identical block lists in
the same transaction we can't just do read/modify/write for the ref
counters as we wouldn't see our own first update. See above about
tracking this and risks about getting it wrong.
GC uses a bloom filter for keys to avoid heavy RAM usage. GC can't run
concurrently with FileInfo updates so there is a new lock around those
operation at the lowlevel.
The end result is a much more compact database, especially for setups
with many peers where files get duplicated many times.
This is per-key-class stats for a large database I'm currently working
with, under the current schema:
```
0x00: 9138161 items, 870876 KB keys + 7397482 KB data, 95 B + 809 B avg, 1637651 B max
0x01: 185656 items, 10388 KB keys + 1790909 KB data, 55 B + 9646 B avg, 924525 B max
0x02: 916890 items, 84795 KB keys + 3667 KB data, 92 B + 4 B avg, 192 B max
0x03: 384 items, 27 KB keys + 5 KB data, 72 B + 15 B avg, 87 B max
0x04: 1109 items, 17 KB keys + 17 KB data, 15 B + 15 B avg, 69 B max
0x06: 383 items, 3 KB keys + 0 KB data, 9 B + 2 B avg, 18 B max
0x07: 510 items, 4 KB keys + 12 KB data, 9 B + 24 B avg, 41 B max
0x08: 1349 items, 12 KB keys + 10 KB data, 9 B + 8 B avg, 17 B max
0x09: 194 items, 0 KB keys + 123 KB data, 5 B + 634 B avg, 11484 B max
0x0a: 3 items, 0 KB keys + 0 KB data, 14 B + 7 B avg, 30 B max
0x0b: 181836 items, 2363 KB keys + 10694 KB data, 13 B + 58 B avg, 173 B max
Total 10426475 items, 968490 KB keys + 9202925 KB data.
```
Note 7.4 GB of data in class 00, total size 9.2 GB. After running the
migration we get this instead:
```
0x00: 9138161 items, 870876 KB keys + 2611392 KB data, 95 B + 285 B avg, 4788 B max
0x01: 185656 items, 10388 KB keys + 1790909 KB data, 55 B + 9646 B avg, 924525 B max
0x02: 916890 items, 84795 KB keys + 3667 KB data, 92 B + 4 B avg, 192 B max
0x03: 384 items, 27 KB keys + 5 KB data, 72 B + 15 B avg, 87 B max
0x04: 1109 items, 17 KB keys + 17 KB data, 15 B + 15 B avg, 69 B max
0x06: 383 items, 3 KB keys + 0 KB data, 9 B + 2 B avg, 18 B max
0x07: 510 items, 4 KB keys + 12 KB data, 9 B + 24 B avg, 41 B max
0x09: 194 items, 0 KB keys + 123 KB data, 5 B + 634 B avg, 11484 B max
0x0a: 3 items, 0 KB keys + 0 KB data, 14 B + 17 B avg, 51 B max
0x0b: 181836 items, 2363 KB keys + 10694 KB data, 13 B + 58 B avg, 173 B max
0x0d: 44282 items, 1461 KB keys + 61081 KB data, 33 B + 1379 B avg, 1637399 B max
Total 10469408 items, 969939 KB keys + 4477905 KB data.
```
Class 00 is now down to 2.6 GB, with just 61 MB added in class 0d.
There will be some additional reads in some cases which theoretically
hurts performance, but this will be more than compensated for by smaller
writes and better compaction.
On my own home setup which just has three devices and a handful of
folders the difference is smaller in absolute numbers of course, but
still less than half the old size:
```
0x00: 297122 items, 20894 KB keys + 306860 KB data, 70 B + 1032 B avg, 103237 B max
0x01: 115299 items, 7738 KB keys + 17542 KB data, 67 B + 152 B avg, 419 B max
0x02: 1430537 items, 121223 KB keys + 5722 KB data, 84 B + 4 B avg, 253 B max
...
Total 1947412 items, 151268 KB keys + 337485 KB data.
```
to:
```
0x00: 297122 items, 20894 KB keys + 37038 KB data, 70 B + 124 B avg, 520 B max
0x01: 115299 items, 7738 KB keys + 17542 KB data, 67 B + 152 B avg, 419 B max
0x02: 1430537 items, 121223 KB keys + 5722 KB data, 84 B + 4 B avg, 253 B max
...
0x0d: 18041 items, 595 KB keys + 71964 KB data, 33 B + 3988 B avg, 101109 B max
Total 1965447 items, 151863 KB keys + 139628 KB data.
```
* wip
* wip
* wip
* wip
Makes the logic a bit clearer and safer. This also sneakily redefines
the 0 interval to also mean disabled, whereas it previously meant ...
sometimes default to 1s, sometimes just spin.
* lib/db, lib/protocol: Compact FileInfo and BlockInfo alignment
This fixes the following two lint warnings
FileInfo: struct of size 160 bytes could be of size 136 bytes
BlockInfo: struct of size 48 bytes could be of size 40 bytes
by reordering fields in alignment order (64 bit fields, then 32 bit
fields, then 16 bit fields (if any), then small ones). The end result is
a slightly less aesthetically pleasing struct field order, but since
these are the objects we often juggle in bulk and keep large queues of I
think it's worth it.
It's a micro optimization, but a cheap one.
This adds error returns to model methods called by the protocol layer.
Returning an error will cause the connection to be torn down as the
message couldn't be handled. Using this to signal that a folder isn't
currently available will then cause a reconnection a few moments later,
when it'll hopefully work better.
Tested manually by running with STRECHECKDBEVERY=0 on a nontrivially
sized setup. This panics reliably before this patch, but just causes a
disconnect/reconnect now.
As foretold by the prophecy, "once the database refactor is merged, then
shall appear a request to propagate errors from the store known
throughout the land as the NamedspacedKV, and it shall be good".
This PR does two things, because one lead to the other:
- Move the leveldb specific stuff into a small "backend" package that
defines a backend interface and the leveldb implementation. This allows,
potentially, in the future, switching the db implementation so another
KV store should we wish to do so.
- Add proper error handling all along the way. The db and backend
packages are now errcheck clean. However, I drew the line at modifying
the FileSet API in order to keep this manageable and not continue
refactoring all of the rest of Syncthing. As such, the FileSet methods
still panic on database errors, except for the "database is closed"
error which is instead handled by silently returning as quickly as
possible, with the assumption that we're anyway "on the way out".
Since we've taken upon ourselves to create a log file by default on
Windows, this adds proper management of that log file. There are two new
options:
-log-max-old-files="3" Number of old files to keep (zero to keep only current).
-log-max-size="10485760" Maximum size of any file (zero to disable log rotation).
The default values result in four files (syncthing.log, synchting.0.log,
..., syncthing.3.log) each up to 10 MiB in size. To not use log rotation
at all, the user can say --log-max-size=0.
* lib/versioner: Reduce surface area
This is a refactor while I was anyway rooting around in the versioner.
Instead of exporting every possible implementation and the factory and
letting the caller do whatever, this now encapsulates all that and
exposes a New() that takes a config.VersioningConfiguration.
Given that and that we don't know (from the outside) how a versioner
works or what state it keeps, we now just construct it once per folder
and keep it around. Previously it was recreated for each restore
request.
* unparam
* wip
- The 'help-box' with nearby devices will appear only when there are
any nearby devices
- IDs of nearby devices will look like links (i.e. underlined when
hovered over)
We incorrectly gave a too small buffer to lz4.Compress, causing it to
allocate in some cases (when the data actually becomes larger when
compressed). This then panicked when passed to the buffer pool.
This ensures a buffer that is large enough, and adds tripwires closer to
the source in case this ever pops up again. There is a test that
exercises the issue.
* gui, lib/api: Adds support for prefers-color-scheme on default theme (fixes#6115)
- Renames current default theme into a new "light" theme
- Modifies assets serving to allow getting assets from different themes
* lib/api: Serve assets from arbitrary theme when path starts with "theme-assets"
* lib/api: Moves constant out of function
* Loads light theme in browsers without support for prefers-color-scheme
* gui: Disables dark theme when printing
* Prevents repeated injection and adds support for older browsers
The CSS is always loaded if there is no support for `matchMedia`.
This is apparently an old benchmarking tool. I'd forgotten about it.
Since 67b8ef1f3e the build script tries to
build all binaries explicitly by default, and this fails on Windows as
this tool doesn't build on Windows.
Kill it with fire.
* Fix bufferpool puts (ref #4976)
There was a logic error in Put() which made us put all large blocks into
segment zero, where we subsequently did not look for them.
I also added a lowest threshold, as we otherwise allocate a 128KiB
buffer when we need 24 bytes for a header and such.
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* smaller stress
* cap/len
* wip
* wip
This makes sure addresses are sorted when coming in from the API. The
database merge operation still checks for correct ordering (which is
quick) and sorts if it isn't correct (legacy database record or
replication peer), but then does a copy first.
Tested with -race in production...
This adds a certificate lifetime parameter to our certificate generation
and hard codes it to twenty years in some uninteresting places. In the
main binary there are a couple of constants but it results in twenty
years for the device certificate and 820 days for the HTTPS one. 820 is
less than the 825 maximum Apple allows nowadays.
This also means we must be prepared for certificates to expire, so I add
some handling for that and generate a new certificate when needed. For
self signed certificates we regenerate a month ahead of time. For other
certificates we leave well enough alone.
Without the desktop plug, launching the syncthing snap will not launch a browser window to the admin UI. Adding this one line will fix that. Tested here on my Ubuntu system with a build from tip of master.
The relay and discosrv didn't use the new lib/build package, now they
do. Conversely the lib/build package wasn't aware there might be other
users and hard coded the program name - now it's set by the build
script
This adds a field `guiAddressUsed` to the system status response, that
holds the current listening address actually in use. This may be
different from the one stored in the config because it may have been
overridden by environment or command line flag.
The GUI now checks this field to see if we are listening on localhost.
If we are not, the authentication required warning is displayed,
regardless of the *configured* listening address.
This adds a field `guiAddressUsed` to the system status response, that
holds the current listening address actually in use. This may be
different from the one stored in the config because it may have been
overridden by environment or command line flag.
The GUI now checks this field to see if we are listening on localhost.
If we are not, the authentication required warning is displayed,
regardless of the *configured* listening address.
Allow extending LDFLAGS by setting EXTRA_LDFLAGS to be able to pass
-extldflags=-zrelro -ldflags=-extldflags=-znow for Arch Linux packaging
to get full relro.
This is an experiment in testing, based on the advise to always call
t.Parallel() at the start of every test. Doing so makes tests run in
parallel, which is usually faster, but also exposes package level state
and potential race conditions better.
To support this I had to redesign the CSRF manager to not be package
global, which was indeed an improvement. And tests run five times faster
now.
This splits large writes into smaller ones when using a rate limit,
making them into a legitimate trickle rather than large bursts with a
long time in between.
This is the result of:
- Changing build.go to take the protobuf version from the modules
instead of hardcoded
- `go get github.com/gogo/protobuf@v1.3.0` to upgrade
- `go run build.go proto` to regenerate our code
Assume a folder error was set due to bad ignores on the latest scan.
Previously, doing a manual rescan would result in:
1. Clearing the folder error, which schedules (immediately) an fs
watcher restart
2. Attempting to load the ignores, which fails, so we set a folder
error and bail.
3. Now the fs watcher restarts, as scheduled, so we trigger a scan.
Goto 1.
This change fixes this by not clearing the error until the error is
actually cleared, that is, if both the health check and ignore loading
succeeds.
This introduces a better set of defaults for large databases. I've
experimentally determined that it results in much better throughput in a
couple of scenarios with large databases, but I can't give any
guarantees the values are always optimal. They're probably no worse than
the defaults though.
This is a tiny tool to grab the GitHub releases info and generate a
more concise version of it. The conciseness comes from two aspects:
- We select only the latest stable and pre. There is no need to offer
upgrades to versions that are older than the latest. (There might be, in
the future, when we hit 2.0. We can revisit this at that time.)
- We use our structs to deserialize and reserialize the data. This means
we remove all attributes that we don't understand and hence don't
require.
All in all the new response is about 10% the size of the previous one and
avoids the issue where we only serve a bunch of release candidates and
no stable.
NATSymmetricUDPFirewall actually is not NAT at all, but means the machine has a global IP address and an UDP firewall in front (RFC calls it Symmetric UDP Firewall). This is punchable fine, both theoretically and also practically in testing.
This changes the on disk format for new raw reports to be gzip
compressed. Also adds the ability to serve these reports in plain text,
to insulate web browsers from the change (previously we just served the
raw reports from disk using Caddy).
* lib/model: Don't panic on failed chmod-back on directory (fixes#5836)
This makes the "in writable dir"-wrapper log chmod-back errors instead
of panicking. To do that we need a logger so the function moved into the
model package which is also the only place it's used. The tests came
along.
(The test also exercised osutil.RenameOrCopy like some sort of
piggybacking. I removed that.)
This adds a set of magical environment variables that can be used to
tweak the database parameters. It's totally undocumented and not
intended to be a long term or supported thing.
It's ugly, but there is a backstory. I have a couple of large
installations where the database options are inefficient or otherwise
suboptimal (24/7 compaction going on and stuff like that). I don't
*know* the correct database parameters, nor yet the formula or method to
derive them by, so this requires experimentation. Experimentation needs
to happen partly in production, and rolling out new builds for every
tweak isn't practical. This provides override points for all reasonable
values, while not changing anything by default.
Ideally, at the end of such experimentation, we'll know which values are
relevant to change and in what manner, and can provide a more user
friendly knob to do so - or do it automatically based on the database
size.
* add skeleton for lib/syncthing
* copy syncthingMain to lib/syncthing (verbatim)
* Remove code to deduplicate copies of syncthingMain
* fix simple build errors
* move stuff from main to syncthing with minimal mod
* merge runtime options
* actually use syncthing.App
* pass io.writer to lib/syncthing for auditing
* get rid of env stuff in lib/syncthing
* add .Error() and comments
* review: Remove fs interactions from lib
* and go 1.13 happened
* utility functions
Per the sync/atomic bug note:
> On ARM, x86-32, and 32-bit MIPS, it is the caller's
> responsibility to arrange for 64-bit alignment of 64-bit words
> accessed atomically. The first word in a variable or in an
> allocated struct, array, or slice can be relied upon to be
> 64-bit aligned.
All atomic accesses of 64-bit variables in syncthing code base are
currently ok (i.e they are all 64-bit aligned).
Generally, the bug is triggered because of incorrect alignement
of struct fields. Free variables (declared in a function) are
guaranteed to be 64-bit aligned by the Go compiler.
To ensure the code remains correct upon further addition/removal
of fields, which would change the currently correct alignment, I
added the following comment where required:
// atomic, must remain 64-bit aligned
See https://golang.org/pkg/sync/atomic/#pkg-note-BUG.
* lib/versioner: Add placeholder to provide the absolute file path to external commands
This commit adds support for a new placeholder, %FILE_PATH_FULL%, to the
command of the external versioner. The placeholder will be replaced by
the absolute path of the file that should be deleted.
* Revert "lib/versioner: Add placeholder to provide the absolute file path to external commands"
This reverts commit fb48962b94.
* lib/versioner: Replace all placeholders in external command (fixes#5849)
Before this commit, only these placeholders were replaced that span a
whole word, for example "%FOLDER_PATH%". Words that consisted of more
than one placeholder or additional characters, for example
"%FOLDER_PATH%/%FILE_PATH%", were left untouched.
* fixup! lib/versioner: Replace all placeholders in external command (fixes#5849)
Use a global raven.Client because they allocate an http.Client for each,
with a separate CA bundle and infinite connection idle time. Infinite
connection idle time means that if the client is never used again it
will always keep the connection around, not verifying whether it's
closed server side or not. This leaks about a megabyte of memory for
each client every created.
client.Close() doesn't help with this because the http.Client is still
around, retained by its own goroutines.
The thing with the map is just to retain the API on sendReport, even
though there will in practice only ever be one DSN per process
instance...
* lib/ur: Implement crash (panic) reporting (fixes#959)
This implements a simple crash reporting method. It piggybacks on the
panic log files created by the monitor process, picking these up and
uploading them from the usage reporting routine.
A new config value points to the crash receiver base URL, which defaults
to "https://crash.syncthing.net/newcrash" (following the pattern of
"https://data.syncthing.net/newdata" for usage reports, but allowing us
to separate the service as required).
* lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes#5697)
* forgot unsupported watch
* and more non(-standard)-unixy fixes
* and windows test
* review
* lib/api, lib/connections, gui: Show connection error for disconnected devices (fixes#3345)
This adds functionality in the connetions service to track the last
error per address. That is in turn exposed in the /rest/system/status
API method, as that is also where we already show the listener status
from the connection service.
The GUI uses this info where it lists addresses, showing errors (if any)
in red underneath each address.
I also slightly refactored the existing status method on the connection
service to have a better name and return typed information.
* ok
* review
* formatting
* review
The check in ClusterConfig() when iterating through announced devices
in a folder explicitly skips entries without a non-zero IndexID.
Therefore, the check for IndexID == 0 just below will never be true
and the intended cleanup of local index data will not happen.
Plainly remove that check to make the intended case distinction work.
* lib/protocol: Wait for reader/writer loops on close (fixes#4170)
* waitgroup
* lib/model: Don't hold lock while closing connection
* fix comments
* review (lock once, func argument) and naming
* lib/model: Send cluster config before releasing pmut
* reshuffle
* add model.connReady to track cluster-config status
* Corrected comments/strings
* do it in protocol
Using AngularJS markup (that is, AngularJS expressions enclosed in double curly braces) in HTML attributes that reference URLs is not recommended: the browser may attempt to fetch the URL before the AngularJS compiler evaluates the markup, resulting in a request for an invalid URL.
This constructs the map of hashes of zero blocks from constants instead
of calculating it at startup time. A new test verifies that the map is
correct.
* cmd/syncthing, lib/gui: Separate gui into own package (ref #4085)
* fix tests
* Don't use main as interface name (make old go happy)
* gui->api
* don't leak state via locations and use in-tree config
* let api (un-)subscribe to config
* interface naming and exporting
* lib/ur
* fix tests and lib/foldersummary
* shorter URVersion and ur debug fix
* review
* model.JsonCompletion(FolderCompletion) -> FolderCompletion.Map()
* rename debug facility https -> api
* folder summaries in model
* disassociate unrelated constants
* fix merge fail
* missing id assignement
The change in 225c0dda80 (#5511) results in
conflicts being immediately scanned and synced, while the test expects just one
conflict on either side. Change it to expect the same conflict on both sides.
* lib/tlsutil: Enable TLS 1.3 when available, on test builds (fixes#5065)
This enables TLS 1.3 negotiation on Go 1.12 by setting the GODEBUG
variable. For now, this just gets enabled on test versions (those with a
dash in the version number).
Users wishing to enable this on production builds can set GODEBUG
manually.
The string representation of connections now includes the TLS version
and cipher suite. This becomes part of the log output on connections.
That is, when talking to an old client:
Established secure connection .../TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
and now potentially:
Established secure connection .../TLS1.3-TLS_AES_128_GCM_SHA256
(The cipher suite was there previously in the log output, but not the
TLS version.)
I also added this info as a new Crypto() method on the connection, and
propagate this out to the API and GUI, where it can be seen in the
connection address hover (although with bad word wrapping sometimes).
* wip
* wip
Flush the batch when exceeding a certain size, instead of when reaching a number
of batched operations.
Move batch to lowlevel to be able to use it in NamespacedKV.
Increase the leveldb memory buffer from 4 to 16 MiB.
* cleanup Fatal in lib/config/config.go
* cleanup Fatal in lib/config/folderconfiguration.go
* cleanup Fatal in lib/model/model.go
* cleanup Fatal in cmd/syncthing/monitor.go
* cleanup Fatal in cmd/syncthing/main.go
* cleanup Fatal in lib/api
* remove Fatal methods from logger
* lowercase in errors.Wrap
* one less channel
I'm working through linter complaints, these are some fixes. Broad
categories:
1) Ignore errors where we can ignore errors: add "_ = ..." construct.
you can argue that this is annoying noise, but apart from silencing the
linter it *does* serve the purpose of highlighting that an error is
being ignored. I think this is OK, because the linter highlighted some
error cases I wasn't aware of (starting CPU profiles, for example).
2) Untyped constants where we though we had set the type.
3) A real bug where we ineffectually assigned to a shadowed err.
4) Some dead code removed.
There'll be more of these, because not all packages are fixed, but the
diff was already large enough.
This adds a folder option "CopyOwnershipFromParent" which, when set,
makes Syncthing attempt to retain the owner/group information when
syncing files. Specifically, at the finisher stage we look at the parent
dir to get owner/group and then attempt a Lchown call on the temp file.
For this to succeed Syncthing must be running with the appropriate
permissions. On Linux this is CAP_FOWNER, which can be granted by the
service manager on startup or set on the binary in the filesystem. Other
operating systems do other things, but often it's not required to run as
full "root". On Windows this patch does nothing - ownership works
differently there and is generally less of a deal, as permissions are
inherited as ACLs anyway.
There are unit tests on the Lchown functionality, which requires the
above permissions to run. There is also a unit test on the folder which
uses the fake filesystem and hence does not need special permissions.
This removes our vendored dependencies. They provide no value for our
own build or development processes. For our source releases, the build
job can accomplish the same thing by a "go mod vendor" to recreate the
vendor dir (from the cryptographically verified dependencies).
To do so the BlockMap struct has been removed. It behaves like any other prefixed
part of the database, but was not integrated in the recent keyer refactor. Now
the database is only flushed when files are in a consistent state.
There was a problem in iterating the sequence index that could result
in missing updates. The issue is that while the index was (correctly)
iterated in a snapshot, the actual file infos were read dirty outside of
the snapshot. This fixes this by doing the reads inside the snapshot,
and also updates a couple of other places that did the same thing more
or less harmfully (I didn't investigate).
To avoid similar issues in the future I did some renaming of the
getFile* methods - the ones in a transaction are just getFile, while the
ones directly on the database are variants of getFileDirty to highlight
what's going on.
This adds booleans to the /system/version response to advice the GUI
whether the running version is a candidate release or not. (We could
parse it from the version string, but why duplicate the logic.)
Additionally the settings dialog locks down the upgrade and usage
reporting options on candidate releases. This matches the current
behavior, it just makes it obvious what actually *can* be chosen.
* lib/fs, lib/model: Improve filesystem operations during tests (fixes#5422)
Introduces MustFilesystem that panics on errors and should be used for operations
during testing which must never fail.
Create temporary directories outside of testdata.
* don't do a filesystem, just a wrapper around os for testing
* fix copyright
This avoids waiting until next ping and timeout until the connection is actually
closed both by notifying the peer of the disconnect and by immediately closing
the local end of the connection after that. As a nice side effect, info level
logging about dropped connections now have the actual reason in it, not a generic
timeout error which looks like a real problem with the connection.
* go mod init; rm -rf vendor
* tweak proto files and generation
* go mod vendor
* clean up build.go
* protobuf literals in tests
* downgrade gogo/protobuf
Here the event Logger is rewritten as a service with a main loop instead
of mutexes. This loop has a select with essentially two legs: incoming
events, and subscription changes. When both are possible select will
chose one randomly, thus ensuring that in practice unsubscribes will
happen timely and not block the system.
Updates the package and fixes a test that depended on the old behavior
of Write() being equivalent to Reset()+Write() which is no longer the
case. The scanner already did resets after each block write, so this is
fine.
This changes the TLS and certificate handling in a few ways:
- We always use TLS 1.2, both for sync connections (as previously) and
the GUI/REST/discovery stuff. This is a tightening of the requirements
on the GUI. AS far as I can tell from caniusethis.com every browser from
2013 and forward supports TLS 1.2, so I think we should be fine.
- We always greate ECDSA certificates. Previously we'd create
ECDSA-with-RSA certificates for sync connections and pure RSA
certificates for the web stuff. The new default is more modern and the
same everywhere. These certificates are OK in TLS 1.2.
- We use the Go CPU detection stuff to choose the cipher suites to use,
indirectly. The TLS package uses CPU capabilities probing to select
either AES-GCM (fast if we have AES-NI) or ChaCha20 (faster if we
don't). These CPU detection things aren't exported though, so the tlsutil
package now does a quick TLS handshake with itself as part of init().
If the chosen cipher suite was AES-GCM we prioritize that, otherwise we
prefer ChaCha20. Some might call this ugly. I think it's awesome.
Fixes error "code in directory GOPATH/src/github.com/golang/lint/golint expects import golang.org/x/lint/golint" during the
go run build.go setup command.
See https://github.com/golang/lint/issues/415
In a recent change (#5201) this return disappeared. The effect is that
we first shortcut the file and then also treat it normally. This results
in to database updates after each other, which are bound to end up in
the same batch. This means we remove one sequence entry and add two.
Not marking the issues as fixed, because I need to do more testing and
there are other discrepancies...
This adds a thin type that holds the state associated with the
leveldb.DB, leaving the huge Instance type more or less stateless. Also
moves some keying stuff into the DB package so that other packages need
not know the keying specifics.
(This does not, yet, fix the cmd/stindex program, in order to keep the
diff size down. Hence the keying constants are still exported.)
* lib/model, cmd/syncthing: Wait for folder restarts to complete (fixes#5233)
This is the somewhat ugly - but on the other hand clear - fix for what
is really a somewhat thorny issue. To avoid zombie folder runners a new
mutex is introduced that protects the RestartFolder operation. I hate
adding more mutexes but the alternatives I can think of are worse.
The other part of it is that the POST /rest/system/config operation now
waits for the config commit to complete. The point of this is that until
the commit has completed we should not accept another config commit. If
we did, we could end up with two separate RestartFolders queued in the
background. While they are both correct, and will run without
interfering with each other, we can't guarantee the order in which they
will run. Thus it could happen that the newer config got committed
first, and the older config commited after that, leaving us with the
wrong config running.
* test
* wip
* hax
* hax
* unflake test
* per folder mutexes
* paranoia
* race
* lib/fs: Add fakefs
This adds a new fake filesystem type. It's described rather extensively
in fakefs.go, but the main point is that it's for testing: when you want
to spin up a Syncthing and have a terabyte or two of random files that
can be synced somewhere, or an inifitely large filesystem to sync files
into.
It has pseudorandom properties such that data read from one fakefs can
be written into another fakefs and read back and it will look
consistent, without any of the data actually being stored.
To use:
<folder id="default" path="whatever" ...>
<filesystemType>fake</filesystemType>
This will create an empty fake filesystem. You can also specify that it
should be prefilled with files:
<folder id="default" path="whatever?size=2000000" ...>
<filesystemType>fake</filesystemType>
This will create a filesystem filled with 2TB of random data that can be
scanned and synced. There are more options, see fakefs.go.
Prefilled data is based on a deterministic seed, so you can index the
data and restart Syncthing and the index is still correct for all the
stored data.
The previous "Bad Request" was really confusing as it implies it's
somethign wrong with the request, which there isn't - the problem is
that server configuration forbids the request.
This removes the user and group juggling, which would fail when given
for example a PGID that already existed as the "syncthing" group could
then not be created with that PGID. It's not reasonable to expect the
user to know which group/user names/IDs are already present in the
Docker image.
Instead we now just launch under the specified IDs, while manually
setting the HOME env var to give us a home directory - the only thing we
needed the user entry for anyway.
Also updates to Go 1.11 and building without upgrades instead of
disabling by env var.
* release:
lib/model: Fixes on receive-only test setup and pulling (#5136)
lib/fs: Don't add path separators at end of path (fixes#5144) (#5146)
lib/fs: Evaluate root when watching not on fs creation (fixes#5043) (#5105)
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.
This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).
The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
The problem here is that we would update the sequence index before
updating the FileInfos, which would result in a high sequence number
pointing to a low-sequence FileInfo. The index sender would pick up the
high sequence number, send the old file, and think everything was good.
On the receiving side the old file is a no-op and ignored. The file
remains out of sync until another update for it happens.
This fixes that by correcting the order of operations in the database
update: first we remove old sequence index entries, then we update the
FileInfos (which now don't have anything pointing to them) and then we
add the sequence indexes (which the index sender can see).
The other option is to add "proper" transactions where required at the
database layer. I actually have a branch for that, but it's literally
thousands of lines of diff and I'm putting that off for another day as
this solves the problem...
This removes the out of disk space check from CheckHealth. The disk space is now
only checked if there are files to pull, in which case pulling those files is
stopped, but everything else (dirs, links, deletes) keeps running -> can recover
disk space through pulling.
Removes the manual handling of the AUTHORS file, giving every committer automatic credit for their contribution.
Manual tweaks to the file are still accepted and retained by the scripts.
A dedicated user is necessary to create relative references via
~/<folder> or $HOME/<folder>. Having the syncthing process just running
under a unprivileged UID/GID, will remove the home folder relation and
therefore will result in nonexistent shares after update.
Signed-off-by: Benedikt Heine <bebe@bebehei.de>
Adds a receive only folder type that does not send changes, and where the user can optionally revert local changes. Also changes some of the icons to make the three folder types distinguishable.
This is an improvement of PR #4493 and related to (and maybe fixing) #4961
and #4475. Maybe fixing, because there is no clear reproducer for that
problem.
The previous PR added a mechanism to resurrect missing parent directories,
if there is a valid child file to be pulled. The same mechanism does not
exist for dirs and symlinks, even though a missing parent can happen for
those items as well. Therefore this PR extends the resurrection to all types
of pulled items.
In addition I moved the IsDeleted branch while iterating over
processDirectly to the existing IsDeleted branch in the WithNeed iteration.
This saves one pointless assignment and IsDeleted query. Also
Allows for configuring the UID and GID Syncthing runs as in the container. Uses su-exec from the Alpine repos to accomplish this. Addition of su-exec results in <2MB increase in image size.
This makes the environment variables easier to read/change.
Also expand the list so it's not inline, more readable that way.
The new architecture syntax for snapcraft allows specifying both the building architecture and the running architecture of the snap, so if we specify the build-on architecture as the host architecture and the run-on architecture as the target architecture, then snapcraft shouldn't need to install any cross-compilers, etc.
We have the invalid bit to indicate that a file isn't good. That's enough for remote devices. For ourselves, it would be good to know sometimes why the file isn't good - because it's an unsupported type, because it matches an ignore pattern, or because we detected the data is bad and we need to rescan it.
Or, and this is the main future reason for the PR, because it's a change detected on a receive only device. We will want something like the invalid flag for those changes, but marking them as invalid today means the scanner will rehash them. Hence something more fine grained is required.
This introduces a LocalFlags fields to the FileInfo where we can stash things that we care about locally. For example,
FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
FlagLocalIgnored = 1 << 1 // Matches local ignore patterns
FlagLocalMustRescan = 1 << 2 // Doesn't match content on disk, must be rechecked fully
The LocalFlags fields isn't sent over the wire; instead the Invalid attribute is calculated based on the flags at index sending time. It's on the FileInfo anyway because that's what we serialize to database etc.
The actual Invalid flag should after this just be considered when building the global state and figuring out availability for remote devices. It is not used for local file index entries.
It's pretty much the GitHub standard, says the obvious, and does it
better than our old one which was a little bit rambling.
Intentionally just doing it instead of discussing it, in the hope of
avoiding trolling. So sue me.
To optimize WithNeed, which is called for the local device whenever an index
update is received. No tracking for remote devices to conserve db space, as
WithNeed is only queried for completion.
I'm trying to slowly clean this up a bit, and moving functionality out
into the folder types and having those methods not reach into model is
part of it. That can mean takign some odd arguments in the meantime,
some of those should probably become interfaces or properties on folder
in the long term.
The functionality was anyway mostly implemented there and not isolated
in the folderScanner type. The attempt to refactor it out in the other
direction wouldn't work given that the event loop and stuff is on
`folder`.
This adds a couple of utilities for transporting databases in JSON and a
test to load a database and verify a couple of invalid bits. The test
itself is quite pointless at the moment, but it lays the groundwork for
testing the migration of this data in the next step (after the invalid
bit should be changed to local flags for local files).
When that happens we need to have a database in the old format already
there in order to be able to test the migration.
The directive lives in its own isolated scope (where we put the visible() function). The stuff transcluded into the notification directive lives in the root scope and doesn't have access to the directive scope. Hence we cannot call dismiss() from inside the directive.
Similarly, config does not exist by itself in the directive scope, we need $parent to access the root scope.
Reference: https://docs.angularjs.org/guide/directive#isolating-the-scope-of-a-directive
How this worked before is a mystery. My guess is Angular bug with directive scope that was fixed in 1.3. One possibility is that transclude plus scope: true (which doesn't make sense as that is supposed to be an object) resulted in the root scope being used in the directive itself. This would then "work" as long as there is only one notification, as visible() and dismiss() would then be registered on the root scope, thus accessible from within the notification but also overridden by any notification rendered.
The directive lives in its own isolated scope (where we put the visible() function). The stuff transcluded into the notification directive lives in the root scope and doesn't have access to the directive scope. Hence we cannot call dismiss() from inside the directive.
Similarly, config does not exist by itself in the directive scope, we need $parent to access the root scope.
Reference: https://docs.angularjs.org/guide/directive#isolating-the-scope-of-a-directive
How this worked before is a mystery. My guess is Angular bug with directive scope that was fixed in 1.3. One possibility is that transclude plus scope: true (which doesn't make sense as that is supposed to be an object) resulted in the root scope being used in the directive itself. This would then "work" as long as there is only one notification, as visible() and dismiss() would then be registered on the root scope, thus accessible from within the notification but also overridden by any notification rendered.
To newer names better reflecting their types and yet sorting together
with folder.go. Doing it now without asking because there are no open
PRs that will get killed by it, and to avoid bikeshedding the names.
The actual pull method (which is really the only thing that differs
between them) is now an interface member which gets overridden by the
subclass.
"Subclass?!" Well, this is dynamic dispatch with overriding, I guess.
Added EXPOSE to Dockerfile. this way these ports will show up in docker GUIs like cockpit.
Added VOLUME parameter, this renders creating the folder (/var/syncthing) obsolete.
Instead of walking and unmarshalling the entire db and sorting the resulting
file infos by sequence, add store device keys by sequence number in the
database. Thus only the required file infos need be unmarshalled and are already
sorted by index.
Bumping the limit to 2 * the max block size (16 MiB) is a slight
increase compared to previously. Nonetheless I think it's good to allow
us to queue one request and have one on the way in, or conversely have
one large block on the way in and be able to ask for smaller blocks from
others at the same time.
We did this to minimize source size and make it compile faster. Nowadays
byte slice literals compile as fast as anything, and the source isn't
checked in so it doesn't matter how big it is.
Not using base64 avoids a decode step at startup and makes the binary
about 400k smaller.
Two small behavior changes: don't "charge" the data to the global rate
limit until it's been accepted by the device specific limiter, and fix
the send/recv direction in the log print on per device rate limits.
clearAddresses write locks the struct and then calls notify. notify in turn tries to obtain a read lock on the same mutex. The result was a deadlock. This change unlocks the struct before calling notify.
Copied over from settings. Moves ignore patterns from its own modal into a tab
of the folder edit modal. Advanced settings are in a tab instead of a expandable
section and versioning gets it's own tab.
Also added modal hidden event handler to remove the anchor in the url.
Given that we've taken on the resposibility of maintaining this forked
package I've added it to the Syncthing organization. We still vendor it
like an external package, because it's convenient to keep it as a fork
of upstream to easier merge and file pull requests towards them.
When dropping delta index IDs due to upgrade, only drop our local one.
Previously, when dropping all of them, we would trigger a full send in
both directions on first connect after upgrade. Then the other side
would upgrade, doing the same thing. Net effect is full index data gets
sent twice in both directions.
With this change we just drop our local ID, meaning we will send our
full index on first connect after upgrade. When the other side upgrades,
they will do the same. This is a bit less cruel.
Unignored files are marked as conflicting while scanning, which is then resolved
in the subsequent pull. Automatically reconciles needed items on send-only
folders, if they do not actually differ except for internal metadata.
This doesn't happen today, but it might in the future if the block size
were increased or made variable and we were talking to a client from the
future.
The current 500 "test failed" looks and sounds like a problem in the
relay pool server, while it actually indicates a problem on the
announcing side. Instead use 400 "connection test failed" to indicate
that the request was bad and what was the test.
* lib/db: Don't panic on negative counts (fixes#4659)
So, negative counts should never happen and hence the original idea to
panic. However, this sucks as the panic will happen in a folder runner,
be automatically swallowed by suture, and the runner gets restarted but
now we are in a bad state. (Related: #4758)
At the time of writing the global list is somewhat in flux (we've
changed how ignored files are handled, invalid bits, etc.) and I think
that can cause unusual conditions here. Hence just fixing up the numbers
instead until the next full recount.
* release:
lib/osutil: Fix priority lowering on Windows
lib/osutil: Don't attempt to reduce our niceness level (fixes#4681)
lib/osutil: Check PGID before trying to set it (fixes#4679)
When scanner.Walk detects a change, it now returns the new file info as well as the old file info. It also finds deleted and ignored files while scanning.
Also directory deletions are now always committed to db after their children to prevent temporary failure on remote due to non-empty directory.
This removes a number of timing related things, leaving just the total
test timeout now bumped to one minute. Normally we get the filesystem
events within a second or so, so this doesn't affect the test time in
the successfull case. If we don't actually get the events we expect
within a minute I think we are legitimately in "failed" territory.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4715
LGTM: imsodin, AudriusButkevicius
Since #4340 pulls aren't happening every 10s anymore and may be delayed up to 1h.
This means that no folder error event reaches the web UI for a long time, thus no
failed items will show up for a long time. Now errors are populated when the
web UI is opened.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4650
LGTM: AudriusButkevicius
This is a new revision of the discovery server. Relevant changes and
non-changes:
- Protocol towards clients is unchanged.
- Recommended large scale design is still to be deployed nehind nginx (I
tested, and it's still a lot faster at terminating TLS).
- Database backend is leveldb again, only. It scales enough, is easy to
setup, and we don't need any backend to take care of.
- Server supports replication. This is a simple TCP channel - protect it
with a firewall when deploying over the internet. (We deploy this within
the same datacenter, and with firewall.) Any incoming client announces
are sent over the replication channel(s) to other peer discosrvs.
Incoming replication changes are applied to the database as if they came
from clients, but without the TLS/certificate overhead.
- Metrics are exposed using the prometheus library, when enabled.
- The database values and replication protocol is protobuf, because JSON
was quite CPU intensive when I tried that and benchmarked it.
- The "Retry-After" value for failed lookups gets slowly increased from
a default of 120 seconds, by 5 seconds for each failed lookup,
independently by each discosrv. This lowers the query load over time for
clients that are never seen. The Retry-After maxes out at 3600 after a
couple of weeks of this increase. The number of failed lookups is
stored in the database, now and then (avoiding making each lookup a
database put).
All in all this means clients can be pointed towards a cluster using
just multiple A / AAAA records to gain both load sharing and redundancy
(if one is down, clients will talk to the remaining ones).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4648
Previous "reasonable resource limits" cause test failures with Go 1.9.
They were added to protect the previous generation of the build server
and no longer needed.
Fixes issue #4653.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4656
It turns out that ZFS doesn't do any normalization when storing files,
but does do normalization "as part of any comparison process".
In practice, this seems to mean that if you LStat a normalized filename,
ZFS will return the FileInfo for the un-normalized version of that
filename.
This meant that our test to see whether a separate file with a
normalized version of the filename already exists was failing, as we
were detecting the same file.
The fix is to use os.SameFile, to see whether we're getting the same
FileInfo from the normalized and un-normalized versions of the same
filename.
One complication is that ZFS also seems to apply its magic to os.Rename,
meaning that we can't use it to rename an un-normalized file to its
normalized filename. Instead we have to move via a temporary object. If
the move to the temporary object fails, that's OK, we can skip it and
move on. If the move from the temporary object fails however, I'm not
sure of the best approach: the current one is to leave the temporary
file name as-is, and get Syncthing to syncronize it, so at least we
don't lose the file. I'm not sure if there are any implications of this
however.
As part of reworking normalizePath, I spotted that it appeared to be
returning the wrong thing: the doc and the surrounding code expecting it
to return the normalized filename, but it was returning the
un-normalized one. I fixed this, but it seems suspicious that, if the
previous behaviour was incorrect, noone ever ran afoul of it. Maybe all
filesystems will do some searching and give you a normalized filename if
you request an unnormalized one.
As part of this, I found that TestNormalization was broken: it was
passing, when in fact one of the files it should have verified was
present was missing. Maybe this was related to the above issue with
normalizePath's return value, I'm not sure. Fixed en route.
Kindly tested by @khinsen on the forum, and it appears to work.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4646
This adds one new feature, that discovery servers can have ?nolookup to
be used only for announces. The default set of discovery servers is
changed to:
- discovery.s.n used for lookups. This is dual stack load balanced over
all discovery servers, and returns both IPv4 and IPV6 results when they
exist.
- discovery-v4.s.n used for announces. This has IPv4 addresses only and
the discovery servers will update the unspecified address with the IPv4
source address, as usual.
- discovery-v6.s.n which is exactly the same for IPv6.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4647
This no longer pokes at model internals, and only touches the config.
As a result, model handles this in CommitConfiguration, which restarts
the folders if things change, which repopulate m.folderDevice, m.deviceFolder
and other interal mappings.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4639
These files always have the symlink bit set, because they are reparse
points. Nonetheless they are not symlinks, and Lstat reports a size for
them. We use this fact to disambiguate, and hope fervently that nothing
else matches this description so it comes back to bite us...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4622
Just because there are a ton of people struggling to set env vars.
Perhaps this should live in advanced settings, and perhaps we should have a button to view the log.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4604
LGTM: calmh, imsodin
Also attempt to handle this nicer by ignoring the truncate failure when
it doesn't matter, and recover by deleting the temp file when it does.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4594
This keeps the data we need about sequence numbers and object counts
persistently in the database. The sizeTracker is expanded into a
metadataTracker than handled multiple folders, and the Counts struct is
made protobuf serializable. It gains a Sequence field to assist in
tracking that as well, and a collection of Counts become a CountsSet
(for serialization purposes).
The initial database scan is also a consistency check of the global
entries. This shouldn't strictly be necessary. Nonetheless I added a
created timestamp to the metadata and set a variable to compare against
that. When the time since the metadata creation is old enough, we drop
the metadata and rebuild from scratch like we used to, while also
consistency checking.
A new environment variable STCHECKDBEVERY can override this interval,
and for example be set to zero to force the check immediately.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4547
LGTM: imsodin
So STDEADLOCK seems to do the same thing as STDEADLOCKTIMEOUT, except in
the other package. Consolidate?
STDEADLOCKTHRESHOLD is actually called STLOCKTHRESHOLD, correct the help
text.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4598
Two issues since the filesystem migration;
- filepath.Base() doesn't do what the code expected when the path ends
in a slash, and we make sure the path ends in a slash.
- the return should be fully qualified, not just the last component.
With this change it works as before, and passes the new test for it.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4591
LGTM: imsodin
Fix the folder restart behavior (ignore Label), improve the API for that
(imho).
Also removes the tab switch animation in the settings modal, because
annoying.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4577
* release:
lib/connections: Trust the model to tell us if we are connected
build: More signatures, more better (ref #3420)
lib/model: Trigger a pull when ignore patterns change
This should address issue as described in https://forum.syncthing.net/t/stun-nig-party-with-paused-devices/10942/13
Essentially the model and the connection service goes out of sync in terms of thinking if we are connected or not.
Resort to model as being the ultimate source of truth.
I can't immediately pin down how this happens, yet some ideas.
ConfigSaved happens in separate routine, so it's possbile that we have some sort of device removed yet connection comes in parallel kind of thing.
However, in this case the connection exists in the model, and does not exist in the connection service and the only way for the connection to be removed
in the connection service is device removal from the config.
Given the subject, this might also be related to the device being paused.
Also, adds more info to the logs
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4533
This should address issue as described in https://forum.syncthing.net/t/stun-nig-party-with-paused-devices/10942/13
Essentially the model and the connection service goes out of sync in terms of thinking if we are connected or not.
Resort to model as being the ultimate source of truth.
I can't immediately pin down how this happens, yet some ideas.
ConfigSaved happens in separate routine, so it's possbile that we have some sort of device removed yet connection comes in parallel kind of thing.
However, in this case the connection exists in the model, and does not exist in the connection service and the only way for the connection to be removed
in the connection service is device removal from the config.
Given the subject, this might also be related to the device being paused.
Also, adds more info to the logs
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4533
We need to reset prevSeq so that we force a full check when someone
reconnects - the sequence number may not have changed due to the
reconnect. (This is a regression; we did this before f6ea2a7.)
Also add an optimization: we schedule a pull after scanning, but there
is no need to do so if no changes were detected. This matters now
because the scheduled pull actually traverses the database which is
expensive.
This, however, makes the pull not happen on initial scan if there were
no changes during the initial scan. Compensate by always scheduling a
pull after initial scan in the rwfolder itself.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4508
LGTM: imsodin, AudriusButkevicius
This is step one of a hundred fifty on the path to case insensitivity.
It brings in the basic case folding mechanism and adds it to the
mtimefs, as this is something outside the fileset that touches stuff in
the database based on name. No effort to convert or handle existing
entries when the insensitivity is changed, I don't think we need it...
Useless by itself but includes tests and will reduce the review load
along the way.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4521
This makes it OK to not have any listeners working. Specifically,
- We don't complain about an empty listener address
- We don't complain about not having anything to announce to global
discovery servers
- We don't send local discovery packets when there is nothing to
announce.
The last point also fixes a thing where the list of addresses for local
discovery was set at startup time and never refreshed.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4517
Well Tested(TM)
Introduces a potential issue where we always pick some connectable but dodgy connection that breaks
soon after the TLS handshake.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4489
Diff is large due to comment reformatting and indentation but all it
does is wrap the file mtime/size/permissions check in an "if
stat.IsRegular()".
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4507
This removes a significant, complex chunk of database code. The
"replace" operation walked both the old and new in lockstep and made the
relevant changes to make the new situation correct. But since delta
indexes we pretty much never need this - we just used replace to drop
the existing data and start over.
This makes that explicit and removes the complexity.
(This is one of those things that would be annoying to make case
insensitive, while the actual "drop and then insert" that we do is
easier.)
This is fairly well unit tested...
The one change to the tests is to cover the fact that previously replace
with something identical didn't bump the sequence number, while
obviously removing everything and re-inserting does. This is not
behavior we depend on anywhere.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4500
LGTM: imsodin, AudriusButkevicius
With VPNs and stuff we can get a single failure on an interface that
supposedly supports broadcasts without it being fatal.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4415
The folder marker conversion forgot to hide the .stfolder. This adds
that, for those who have not yet been converted.
Also adds Hide() calls to the folder start, to mend historical
unhidedness. (I'm sure this will upset someone who is manually managing
their .stignores in the other direction...)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4384
When STHASHING is set, don't benchmark as it's already decided. If weak
hashing isn't set to "auto", don't benchmark that either.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4349
These functions were very naive and slow. We haven't done much about
them because they pretty much don't matter at all for Syncthing
performance. They are however called very often in the discovery server
and these optimizations have a huge effect on the CPU load on the
public discovery servers.
The code isn't exactly obvious, but we have good test coverage on all
these functions.
benchmark old ns/op new ns/op delta
BenchmarkLuhnify-8 12458 1045 -91.61%
BenchmarkUnluhnify-8 12598 1074 -91.47%
BenchmarkChunkify-8 10792 104 -99.04%
benchmark old allocs new allocs delta
BenchmarkLuhnify-8 18 1 -94.44%
BenchmarkUnluhnify-8 18 1 -94.44%
BenchmarkChunkify-8 44 2 -95.45%
benchmark old bytes new bytes delta
BenchmarkLuhnify-8 1278 64 -94.99%
BenchmarkUnluhnify-8 1278 64 -94.99%
BenchmarkChunkify-8 42552 128 -99.70%
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4346
Currently all errors during pulling and the first of these errors again on
finishing are logged to info. Besides that the errors logged when finishing
are stored in f.errors. This PR moves all logging during pulling to the debug
channel (they might still be relevant in some obscure debugging case) and
uses the stored errors to log the main error per fail when all pulling
iterations are done and failed.
Additional instead of trying 11 times it now only tries 3 times.
This is the first part of what is discussed here:
https://forum.syncthing.net/t/reduce-verboseness-of-puller/10261
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4338
This updates kcp and uses our own fork which:
1. Keys sessions not just by remote address, but by remote address +
conversation id 2. Allows not to close connections that were passed directly
to the library. 3. Resets cache key if the session gets terminated.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4339
LGTM: calmh
Prior to this, the following is possible:
- Create a symlink "foo -> /somewhere", it gets synced
- Delete "foo", it gets versioned
- Create "foo/bar", it gets synced
- Delete "foo/bar", it gets versioned in "/somewhere/bar"
With this change, versioners should never version symlinks.
As of Go 1.8 it's valid to not have a GOPATH set. We ask the Go tool
what the GOPATH seems to be, and if we find ourselves (or a valid copy
of ourselves...) in that location we do not fiddle with the GOPATH.
This adds support for building with the source placed anywhere and no
GOPATH set. The build script handles this by creating a temporary GOPATH
in the system temp dir (or another specified location) and mirroring the
source there before building. The resulting binaries etc still end up in
the same place as usual, meaning at least the "build", "install", "tar",
"zip", "deb", "snap", "test", "vet", "lint", "metalint" and "clean"
commands work without a GOPATH. To this end these commands internally
use fully qualified package paths like
"github.com/syncthing/syncthing/cmd/..." instead of "./cmd/..." like
before.
There is a new command "gopath" that prepares and echoes the directory
of the temporary GOPATH. This can be used to run other non-build go
commands:
export GOPATH=$(go run build.go gopath) // GOPATH is now set
go test -v -race github.com/syncthing/syncthing/cmd/...
There is a new option "-no-build-gopath" that prevents the
check-and-copy step, instead assuming the temporary GOPATH is already
created and up to date. This is a performance optimization for build
servers running multiple builds commands in sequence:
go run build.go gopath // creates a temporary GOPATH
go run build.go -no-build-gopath -goos=... tar // reuses GOPATH
go run build.go -no-build-gopath -goos=... tar // reuses GOPATH
The temporary GOPATH is placed in the system temporary directory
(os.TempDir()) unless overridden by the STTMPDIR variable. It is named
after the hash of the current directory where build.go is run. The
reason for this is that the name should be unique to a source checkout
without risk for conflict, but still persistent between runs of
build.go.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4253
LGTM: AudriusButkevicius, imsodin
This moves a few things from script/ to a new directory meta/, and makes
them real Go tests. These are the authors, copyright, metalint and gofmt
checks. That means that they can now be run by
go test -v ./meta
and optionally filtered by the usual -run thing to go test. Also -short
will cut down on the metalint stuff and exclude the authors check (which
is slow because it runs git lots of times).
Mainly this makes everything easier on things like build servers where
we can now just run tests instead of do a bunch of scripting.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4252
Otherwise all the lines from includes will be shown in the web UI instead of
just the #include ... line. This problem was introduced in #3996.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4248
LGTM: calmh
This removes the special handling of minor versions as major when the
actual major is zero, and adds the special case that upgrades from 0.x
to 1.x are considered minor. 0.x to 2.x or 1.x to 2.x etc are still
considered major.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4226
This solves the erratic test failures on model.TestIgnores by ensuring
that the ignore patterns are reloaded even in the face of unchanged
timestamps.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4208
Things like the package name "syncthing" was hardcoded, which is not
awesome. With this in place we can build debs called syncthing-discosrv,
syncthing-relaysrv and syncthing-relaypoolsrv. I don't actually intend
to build and publish the relaypoolsrv, but the others can be good. Using
the "syncthing-" prefix to make the obvious "apt-cache search syncthing"
actually show them etc.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4206
LGTM: AudriusButkevicius, calmh
Starting stuff from init() is an antipattern, and the innerProcess
variable isn't 100% reliable. We should sort out the other uses of it as
well in due time.
Also removing the hack on innerProcess as I happened to see it and the
affected versions are now <1% users.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4185
The folder already knew how to stop properly, but the fs.Walk() didn't
and can potentially take a very long time. This adds context support to
Walk and the underlying scanning stuff, and passes in an appropriate
context from above. The stop channel in model.folder is replaced with a
context for this purpose.
To test I added an infiniteFS that represents a large amount of data
(not actually infinite, but close) and verify that walking it is
properly stopped. For that to be implemented smoothly I moved out the
Walk function to it's own type, as typically the implementer of a new
filesystem type might not need or want to reimplement Walk.
It's somewhat tricky to test that this actually works properly on the
actual sendReceiveFolder and so on, as those are started from inside the
model and the filesystem isn't easily pluggable etc. Instead I've tested
that part manually by adding a huge folder and verifying that pause,
resume and reconfig do the right things by looking at debug output.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4117
So, when first implementing the database layer I added panics on every
unexpected error condition mostly to be sure to flush out bugs and
inconsistencies. Then it became sort of standard, and we don't seem to
have many bugs here any more so the panics are usually caused by things
like checksum errors on read. But it's not an optimal user experience to
crash all the time.
Here I've weeded out most of the panics, while retaining a few "can't
happen" ones like errors on marshalling and write that we really can't
recover from.
For the rest, I'm mostly treating any read error as "entry didn't
exist". This should mean we'll rescan the file and correct the info (if
scanning) or treat it as a new file and do conflict handling (when
pulling). In some cases things like our global stats may be slightly
incorrect until a restart, if a database entry goes suddenly missing
during runtime.
All in all, I think this makes us a bit more robust and friendly without
introducing too many risks for the user. If the database is truly toast,
probably many other things on the system will be toast as well...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4118
This adds a pattern validator to the GUI listen port field that checks
for port numbers 1024 and above. Also adds a help link pointing to the
(new) page talking about GUI listen port numbers. That page has
information on how to work around the restriction, in general terms.
Also changes the header from "GUI Listen Addresses" to the singular
version, because we only support one listen address today.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4116
Given the saveConfig() is async, it might not have happened before we
try to save the ignores and unpause. Likewise must wait for saving
ignores before unpausing or the scan might start before ignores are on
disk.
Javsacript <3
Harmonize how we use batches in the model, using ProtoSize() to judge
the actual weight of the entire batch instead of estimating. Use smaller
batches in the block map - I think we might have though that batch.Len()
in the leveldb was the batch size in bytes, but it's actually number of
operations.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4114
The mechanism to disallow manual scans before the initial scan completed
(#3996) , had the side effect, that if the initial scan failed, no further
scans are allowed. So this marks the initial scan as finished regardless of
whether it succeeded or not.
There was also redundant code in rofolder and a pointless check for folder
health in scanSubsIfHealthy (happens in internalScanFolderSubdirs as well).
This also moves logging from folder.go to ro/rw-folder.go to include the
information about whether it is send-only or send-receive
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4104
This adds a parameter "events" to the /rest/events endpoint. It should
be a comma separated list of the events the consumer is interested in.
When not given it defaults to the current set of events, so it's
backwards compatible.
The API service then manages subscriptions, creating them as required
for each requested event mask. Old subscriptions are not "garbage
collected" - it's assumed that in normal usage the set of event
subscriptions will be small enough. Possibly lower than before, as we
will not set up the disk event subscription unless it's actually used.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4092
This deprecates the current minDiskFreePct setting and introduces
minDiskFree. The latter is, in it's serialized form, a string with a
unit. We accept percentages ("2.35%") and absolute values ("250 k", "12.5
Gi"). Common suffixes are understood. The config editor lets the user
enter the string, and validates it.
We still default to "1 %", but the user can change that to an absolute
value at will.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4087
LGTM: AudriusButkevicius, imsodin
This adds a new config AllowedNetworks per device, which when set should
contain a list of network prefixes (192.168.0.0/126 etc) that are
allowed for the given device. The connection service will not attempt
connections to addresses outside of the given networks and incoming
connections will be rejected as well.
I've added the config to the normal device editor and shown it (when
set) in the device summary on the main screen.
There's a unit test for the IsAllowedNetwork method, I've done some
manual sanity testing on top of that.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4073
One more step on the path of the great refactoring. Touches rwfolder a
little bit since it uses the Lstat from fs as well, but mostly this is
just on the scanner as rwfolder is scheduled for a later refactor.
There are a couple of usages of fs.DefaultFilesystem that will in the
end become a filesystem injected from the top, but that comes later.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4070
LGTM: AudriusButkevicius, imsodin
Click the transfer rate to toggle between binary-exponent bytes (KiB/s,
MiB/s) and metric based bits (kb/s, Mb/s). The setting is persisted in
browser local storage (best effort).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4074
The ng-list directive makes angular handle lists by doing comma
separation in the roundtrip. We could simplify our normal config dialog
this way as well...
Also adds devices that were inexplicably not available in the advanced
config.
This reveals some other uglyness, as the "devices" config of a folder
now shows as "[object Object], [object Object]" - previously it was
invisible. I think that's fine for now.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4071
This change sorts the language selection menu in Syncthing's home page so
that the languages are displayed alphabetically. The issue was discussed in
#3813. There were few ways of doing this. Sorting the language names in
transifix file did not work due to access control. So I sorted the languages
directly in languageSelectdirective.js by inverting and sorting the language
info retrieved from localeService.js.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4052
Adds a unit test to ensure we don't scan symlinks on Windows. For the
rwfolder, trusts that the logic in the invalid check is correct and that
the check is actually called from the need loop.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4042
Basically, if we don't care about the sync status of the file we should
not tag someone else out of sync because they don't have the latest
version. This solves *my* "Syncing - 100%" scenario at least.
The reason this happens seems to be like this, in my situation. I have
three devices, connected in a "line": A-B-C. A is a Mac and litters
.DS_Store files everywhere. I've ignored these, but some escaped into
the folders before I did so. I've also ignored them on B and C but at
different stages. B was flagging C as out of sync, because at the point
the ignores were introduced C had a lower version of .DS_Store than A.
Now none of them are sending updates about it any more since it's
ignored...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3981
Other routines use atomics, hence even if we are under a lock, we should
too.
We might atomically store with
Not sure how it happens, but it's between lines
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3974
After this change,
- Symlinks on Windows are always unsupported. Sorry.
- Symlinks are always enabled on other platforms. They are just a small
file like anything else. There is no need to special case them. If you
don't want to sync some symlinks, ignore them.
- The protocol doesn't differentiate between different "types" of
symlinks. If that distinction ever does become relevant the individual
devices can figure it out by looking at the destination when they
create the link.
It's backwards compatible in that all the old symlink types are still
understood to be symlinks, and the new SYMLINK type is equivalent to the
old SYMLINK_UNKNOWN which was always a valid way to do it.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3962
LGTM: AudriusButkevicius
Instead of just immediately dropping the event if the subscription isn't
ready to receive it, give it 15 ms to catch up. The value 15 ms is
grabbed out of thin air - it just seems reasonable to me.
The timer juggling makes the event send pretty much exactly twice as
slow as it was before, but we're still under a microsecond. I think it's
negligible compared to whatever event that just happened that we're
interested in logging (usually a file operation of some kind).
benchmark old ns/op new ns/op delta
BenchmarkBufferedSub-8 475 950 +100.00%
benchmark old allocs new allocs delta
BenchmarkBufferedSub-8 4 4 +0.00%
benchmark old bytes new bytes delta
BenchmarkBufferedSub-8 104 117 +12.50%
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3960
The monitor process should not set STNORESTART as this indicates the
intention from the user. Setting STMONITORED is enough, as this tells
the next Syncthing instance that it is running under the monitor
process.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3932
The Ping event is important, as it means that requests complete within
a sensible time. The disk events API didn't have the Ping event, so
if there were no disk events, the request would keep taking forever.
Unless, of course, there's a reverse proxy which times the request out
after a suitably large interval (or something else aborts it), in which
case Syncthing isn't very happy.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3929
Instead of
[I6KAH] 19:05:56 INFO: Single thread hash performance is 359 MB/s using minio/sha256-simd (354 MB/s using crypto/sha256).
it now says
[I6KAH] 19:06:16 INFO: Single thread SHA256 performance is 359 MB/s using minio/sha256-simd (354 MB/s using crypto/sha256).
[I6KAH] 19:06:17 INFO: Actual hashing performance is 299.01 MB/s
which is more informative. This is also the number it reports in usage
reporting.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3918
Can't do what I did, as the rolling function is not the same as the
non-rolling one. Instead this uses an improved version of the rolling
adler32 to accomplish the same thing. (PR filed on upstream, so should
be able to use that directly in the future.)
The rolling version of adler32 is just a wrapper around the standard
hash/adler32 when used in a non-rolling fashion, but it's inefficient as
it allocates a new hash instance for every Write(). This uses the
default version instead in the block hasher, and adds a test to verify
the result is the same as they were before. It reduces allocations by
88% and increases speed about 5%.
benchmark old ns/op new ns/op delta
BenchmarkHashFile-8 64434698 61303647 -4.86%
benchmark old MB/s new MB/s speedup
BenchmarkHashFile-8 276.65 290.78 1.05x
benchmark old allocs new allocs delta
BenchmarkHashFile-8 1238 150 -87.88%
benchmark old bytes new bytes delta
BenchmarkHashFile-8 17877363 49292 -99.72%
Syncthing adds some hidden files when a folder is added, but there is currently
no equivalent cleanup procedure. This change is conservative as not to
accidentally cause data loss.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3874
On Windows we would descend into SYMLINKD type links when we scanned
them successfully, as we would return nil from the walk function and the
filepath.Walk iterator apparently thought it OK to descend into the
symlinked directory.
With this change we always return filepath.SkipDir no matter what.
Tested on Windows 10 as admin, does what it should.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3875
Adds support for -auditfile= where is "-" for stdout, "--" for stderr, or a
filename. It can be left blank (or left out entirely) for the original
behaviour of creating a timestamped filename.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3860
Also tweaks the proto definitions:
- [packed=false] on the block_indexes field to retain compat with
v0.14.16 and earlier.
- Uses the vendored protobuf package in include paths.
And, "build.go setup" will install the vendored protoc-gen-gogofast.
This should ensure that a proto rebuild isn't so dependent on whatever
version of the compiler and package the developer has installed...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3864
The protobuf encoder now produces packed arrays for things like []int32,
which is actually correct according to the proto3 spec. However
Syncthing v0.14.16 and earlier doesn't support this. This reverts the
encoding change, but keeps the updated decoder so that we are both more
compatible with other proto3 implementations and can move to the updated
encoder in the future.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3856
This avoids unnecessary browser request failures and retries. Eg:
- Browser reuses existing HTTP connection for GUI refresh request
- Server closes connection with request in flight
- Browser retries GET request.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3854
Since we anyway need the folderConfig for this I'm skipping the copying
of all it's attributes that rwfolder did and just keeping the original
around instead.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3825
This adds support for AES_256_GCM_SHA384 (in there since Go 1.5, a bit
of a shame we missed it) and ChaCha20-Poly1305 (if built with Go 1.8;
ignored on older Gos).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3822
The test for the error string is fragile, and the error string changed
in Go 1.8 so the relevant part is no longer a prefix. This covers it
with a test though, so it should be fine in the future as well.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3818
Instead, trust (and test) that the temp file has appropriate permissions
from the start. The only place where this changes our behavior is for
ignores which go from 0644 to 0600. I'm OK with that.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3756
This changes the "seen" map that we're anyway keeping around to track
the modtimes of loaded files instead. When doing a Load() we check that
1) the file we are loading is in the modtime set, and 2) that none of
the files in the modtime set have changed modtimes. If that's the case
we do a quick return without parsing anything or clearing the cache.
This required adding two one seconds sleeps in the tests to make sure
the modtimes were updated when we expect cache reloads, because I'm on a
crappy filesystem with one second timestamp granularity. That also
proves it works...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3754
The current way is quite confusing for new users - we create a default
folder, but it's not usable with the default folder created somewhere
else. Instead, when setting up for the first time with two devices, the
default folder must be removed and recreated on one of them. This comes
up on IRC and the forum now and then.
I think this matches expectactions better.
Another alternative would be to remove it entirely (not create a default
folder), but then we should also add some guidance in the UI on how to
proceed.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3751
Fsyncing the file has a small performance penalty and seems unnecessary. The
file will be fsynced anyway, when the changes are commited to the database.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3749
This makes the device ID a real type that can be used in the protobuf
schema. That avoids the juggling back and forth from []byte in a bunch
of places and simplifies the code.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3695
Since delta indexes it's perfectly normal for us to need files that are
currently unavailable due to devices being disconnected. This doesn't
imply a failure, so we should not show the "Failed Items" line and
corresponding eternal spinner (since it would never be filled in, since
there is no failure).
We still show state "Out of Sync" (correct) and the list of files we
need (correct).
The discrepancy between global and local sizes is fine and expected in
the presence of ignores. This just moves the "we have ignore patterns"
indication to the actual local size metric, as an explanation of why it
may differ from the global size...
The ```syncthing-resume.service``` will show as a failed service in case
there are no syncthing processes running after resume but it can be
safely ignored because it makes no difference.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3630
This adds autodetection of the fastest hashing library on startup, thus
handling the performance regression. It also adds an environment
variable to control the selection, STHASHING=standard (Go standard
library version, avoids SIGILL crash when the minio library has bugs on
odd CPUs), STHASHING=minio (to force using the minio version) or unset
for the default autodetection.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3617
When the GUI/API is bound to localhost, we enforce that the Host header
looks like localhost. This can be disabled by setting
insecureSkipHostCheck in the GUI config.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3558
When files that were previously marked as deleted became ignored, we
used to do nothing at all. This changes that behavior to set the Invalid
bit (that we should rename to Ignored). This then becomes an update to
other devices that they should not trust our knowledge about the file in
question.
Read this diff without whitespace...
Tested by
- creating a bunch of files on s1
- letting them sync to s2
- shutting down s2
- deleting the files on s1 and rescanning
- adding the files to .stignore on s1 and rescanning
- starting up s2 and letting it sync
- observing the files are not deleted on s2, and it considers itself up
to date.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3557
We used to consider deleted files & directories 128 bytes large. After
the delta indexes change a bug slipped in where deleted files would be
weighted according to their old non-deleted size. Both ways are
incorrect (but the latest change made it worse), as if there are more
files deleted than remaining data in the repo the needSize can be
greater than the globalSize, resulting in a negative completion
percentage.
This change makes it so that deleted items are zero bytes large, which
makes more sense. Instead we expose the number of files that we need to
delete as a separate field in the Completion() result, and hack the
percentage down to 95% complete if it was 100% complete but we need to
delete files. This latter part is sort of ugly, but necessary to give
the user some sort of feedback.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3556
We previously set the mtime on the temp file, and then renamed it to the
real path. Unfortunately that means we'd save the real timestamp under
the under the temp name ".syncthing.foo.tmp" when the actual file that
we will look up on the next scan is "foo". This moves the Chtimes later,
ensuring that it gets recorded correctly under the right name.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3519
These are no longer required with Go 1.7. Change made by removing the
functions, doing a global s/osutil.Remove/os.Remove/.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3514
So there were some issues here. The main problem was that
model.Close(deviceID) was overloaded to mean "the connection was closed
by the protocol layer" and "i want to close this connection". That meant
it could get called twice - once *to* close the connection and then once
more when the connection *was* closed.
After this refactor there is instead a Closed(conn) method that is the
callback. I didn't need to change the parameter in the end, but I think
it's clearer what it means when it takes the connection that was closed
instead of a device ID. To close a connection, the new close(deviceID)
method is used instead, which only closes the underlying connection and
leaves the cleanup to the Closed() callback.
I also changed how we do connection switching. Instead of the connection
service calling close and then adding the connection, it just adds the
new connection. The model knows that it already has a connection and
makes sure to close and clean out that one before adding the new
connection.
To make sure to sequence this properly I added a new map of channels
that get created on connection add and closed by Closed(), so that
AddConnection() can do the close and wait for the cleanup to happen
before proceeding.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3490
The completion of remote devices was based only on the average of the percentages of all folders, which is irrelevant in case of two folders with very different sizes.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3481
LGTM: calmh, AudriusButkevicius
Furthermore:
1. Cleans configs received, migrates them as we receive them.
2. Clears indexes of devices we no longer share the folder with
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3478
It seems that it would be impossible to drop down to relay after establishing a direct connection
Also, we should not drop the existing connection until after we've passed the validation steps,
and it seems it's being dropped in two places unnecesserily at the moment.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3480
This adds a new nanoseconds field to the FileInfo, populates it during
scans and sets the non-truncated time in Chtimes calls.
The actual file modification time is defined as modified_s seconds +
modified_ns nanoseconds. It's expected that the modified_ns field is <=
1e9 (that is, all whole seconds should go in the modified_s field) but
not really enforced. Given that it's an int32 the timestamp can be
adjusted += ~2.9 seconds by the modified_ns field...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3431
This adds a config to enable debug functions on the API server, which is
by default disabled. When enabled, the /rest/debug things become
available and become available without requiring a CSRF token (although
authentication is required if configured).
We also add a new endpoint /rest/debug/cpuprof?duration=15s (with the
duration being configurable, defaulting to 30s). This runs a CPU profile
for the duration and returns it as a file. It sets headers so that a
browser will save the file with an informative name.
The same is done for heap profiles, /rest/debug/heapprof, which does not
take any parameters.
The purpose of this is that any user can enable debugging under
advanced, then point their browser to the endpoint above and get a file
that contains a CPU or heap profile we can use, with the filename
telling us what version and architecture the profile is from.
On the command line, this becomes
curl -O -J http://localhost:8082/rest/debug/cpuprof?duration=5s
curl: Saved to filename
'syncthing-cpu-darwin-amd64-v0.14.3+4-g935bcc0-110307.pprof'
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3467
Otherwise if the file grows during scanning the block list will be out
of sync with the stated size and things get confused. We could fixup the
size afterwards based on the block list, but then we might see other
inconsistencies as the mtime should have changed to reflect the new size
etc. Better stick to the original state and let the next scan pick up
the change.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3442
This used to happen by itself as the connecting device always sent an
Index message and we triggered on that. Nowadays there's no guarantee
for that, but we anyway need to send out one event to let listeners know
the state of folders shared with the device.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3438
We could have a file to sync with permissions rw------- but we'd create
the temp file with rw-rw-rw- minus umask, usually rw-r--r--. This
potentially exposes private data while the file is being synced.
Similarly, when ignorePerms was set and we were reusing a temp files we
would set the permissions to rw-r--r-- explicitly, potentially
overriding a strict umask that would otherwise have had the file be
rw-------.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3437
The previous commit loosened the locking around database updates.
Apparently that was not fine - what happens is that parallell updates
to the same file for different devices stomp on each others updates to
the global index, leaving it missing one of the two devices.
This lets us add message types in the future, for authentication or
other purposes, without completely breaking old clients. I see this as
similar behavior to adding fields to messages - newer clients must
simple be aware that older ones may ignore the message and act
accordingly.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3390
This slightly changes the interface used for committing configuration
changes. The two parts are now:
- VerifyConfiguration, which runs synchronously and locked, and can
abort the config change. These callbacks shouldn't *do* anything
apart from looking at the config changes and saying yes or no. No
change from previously.
- CommitConfiguration, which runs asynchronously (one goroutine per
call) *after* replacing the config and releasing any locks. Returning
false from these methods sets the "requires restart" flag, which now
lives in the config.Wrapper.
This should be deadlock free as the CommitConfiguration calls can take
as long as they like and can wait for locks to be released when they
need to tweak things. I think this should be safe compared to before as
the CommitConfiguration calls were always made from a random background
goroutine (typically one from the HTTP server), so it was always
concurrent with everything else anyway.
Hence the CommitResponse type is gone, instead you get an error back on
verification failure only, and need to explicitly check
w.RequiresRestart() afterwards if you care.
As an added bonus this fixes a bug where we would reset the "requires
restart" indicator if a config that did not require restart was saved,
even if we already were in the requires-restart state.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3386
* relaypoolsrv/master: (32 commits)
Fetch deps of deps X_x
Here we go with gvt bugs
Screw godep
Add solaris support back in
Add font awesome
No value is less than zero
Screw solaris
Godeps
Refactor javascript, always show table, add sorting
Add local geoip
Update dependencies
Hey look, had to check all code out on linux to fix the deps
Update godeps, reduce amount of time spent testing a relay. Goddamit godeps.
Add timeouts, deal with overlapping markers, add a table, increase circle radiuses
Fix a couple of issues with the relays map (geoip, 'data unavailable')
Rate infos are in kbps, not kBps
Add support for header holding IP address
Update relay parameters even if it already exists (fixes#3)
Add missing space
Add homepage
...
A random "instance ID" is generated on each start of the local discovery
service. The instance ID is included in the announcement. When we see a
new instance ID we treat is a new device and respond with an
announcement of our own. Hence devices get to know each other quickly on
restart.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3385
This changes the BEP protocol to use protocol buffer serialization
instead of XDR, and therefore also the database format. The local
discovery protocol is also updated to be protocol buffer format.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3276
LGTM: AudriusButkevicius
This is a supplement patch to commit a58f69b which only fixed global
discovery. This patch adds the missing parts for the local discovery.
If the listen address scheme is set to tcp4:// or tcp6:// and no
explicit host is specified, an address should not be considered if the
source address does not match this scheme.
This prevents invalid URIs like tcp4://<IPv6 address>:<port> or tcp6://<IPv4
address>:<port> for local discovery.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3380
If the listen address scheme is set to tcp4:// or tcp6://, it needs to be
made sure that the remote address matches this scheme before it is added to
the database.
This prevents invalid URIs like tcp4://<IPv6 address>:<port> or tcp6://<IPv4
address>:<port>.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3378
This contains the following behavioral changes:
- Duplicate folder IDs is now fatal during startup
- Invalid folder flags in the ClusterConfig is fatal for the connection
(this will go away soon with the proto changes, as we won't have any
unknown flags any more then)
- Empty path is a folder error reported at runtime
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3370
Events API consumers rely on being able to detect that events were skipped
by the fact that the event ID has increased by more than 1. This is
documented, and is absolutely necessary when trying to maintain a local
model of Syncthing's state.
With the introduction of LocalChangeDetected, which is not exposed to the
Events API, this contract was broken.
This commit introduces separate concepts of a "Global ID" and a
"Subscription ID". The Global ID of an event is unique across all
subscriptions. The Subscription ID is local to a particular subscription,
and always increments by 1. They are both exposed over the Events API, but
the Subscription ID uses the key "id" for backwards compatibility, and
the "?since=xx" parameter refers to the Subscription ID (making the Global
ID for information only).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3351
LGTM: calmh
Also fixes an issue where the discovery cache call would only return the
newest cache entry for a given device instead of the merged addresses
from all cache entries (which is more useful).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3344
The various path cleaning operations done in in cleanedPath() removes
it, so we make sure it's added again at the end. This makes adding the
slash in prepare() unnecessary, but keep it anyway for display purposes
(people looking at the config).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3342
While attempting to fix#2782 I thought the problem was the
CheckFolderHealth method, so I cleaned it up. That turned out not to be
the case, but I think this is better anyhow.
It also moves the "create folder and marker if the folder was empty in
the index" code to StartFolder where I think it makes better sense.
This is covered by a number of existing tests.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3343
This adds a metric for "committed items" to the database instance that I
use in the test code, and a couple of tests that ensure that scans that
don't change anything also don't commit anything.
There was a case in the scanner where we set the invalid bit on files
that are ignored, even though they were already ignored and had the
invalid bit set. I had assumed this would result in an extra database
commit, but it was in fact filtered out by the Set... Anyway, I think we
can save some work on not pushing that change to the Set at all.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3298
This is in preparation for future changes, but also improves the
handling when talking to pre-v0.13 clients. It breaks out the Hello
message and magic from the rest of the protocol implementation, with the
intention that this small part of the protocol will survive future
changes.
To enable this, and future testing, the new ExchangeHello function takes
an interface that can be implemented by future Hello versions and
returns a version indendent result type. It correctly detects pre-v0.13
protocols and returns a "too old" error message which gets logged to the
user at warning level:
[I6KAH] 09:21:36 WARNING: Connecting to [...]:
the remote device speaks an older version of the protocol (v0.12) not
compatible with this version
Conversely, something entirely unknown will generate:
[I6KAH] 09:40:27 WARNING: Connecting to [...]:
the remote device speaks an unknown (newer?) version of the protocol
The intention is that in future iterations the Hello exchange will
succeed on at least one side and ExchangeHello will return the actual
data from the Hello together with ErrTooOld and an even more precise
message can be generated.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3289
I run a lot of builds. They're quite slow now:
jb@syno:~/s/g/s/syncthing $ BUILDDEBUG=1 ./build.sh
... snipped commands ...
runError: gometalinter --disable-all --deadline=60s --enable=varcheck . ./cmd/... ./lib/...
... in 13.00592726s
... build completed in 15.392265235s
That's 15 s total build time, 13 s of which is the varcheck call. The
build server is welcome to run it, but I don't want to on each build. :)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3285
The purpose of this operation is to separate the serving of GUI assets a
bit from the serving of the REST API. It's by no means complete. The end
goal is something like a combined server type that embeds a statics
server and an API server and wraps it in authentication and HTTPS and
stuff, plus possibly a named pipe server that only provides the API and
does not wrap in the same authentication etc.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3273
This sacrifices the ability to return an error when creating the service
for being more persistent in keeping it running.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3270
LGTM: AudriusButkevicius, canton7
As noted in the ticket I no longer agree that dev builds should not auto
upgrade. The main reason is that we give dev builds to users to test
specific fixes, and noone is happier by them being inadvertently stuck
on that version when a newer version including the fix is released.
For developers, it's first of all probably unlikely that development is
happening on a build that's older than release, and secondly STNOUPGRADE
can be set in the environment once and for all if it an issue.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3244
* relaysrv/master: (60 commits)
Add new dependencies
Add more logging in the case of relaypoolsrv internal server error
Dependency update
Update deps
Update packages, fix testutil. Goddamit godep.
Typo
Add signal handlers (fixes#15)
Update readme (fixes#16)
Limit number of connections (fixes#23)
Enable extra logging in pool.go even when -debug not specified
Add Antony Male to CONTRIBUTORS
Allow extAddress to be set from the command line
URLs should have Go units
Add CORS headers
Fix units
Expose provided by in status endpoint
Add ability to advertise provider
Change the URL
Rename relaysrv binary, see #11
Jail the whole thing a bit more
...
* discosrv/master: (64 commits)
Use atomics for statistics handling (fixes#45)
Lower case JSON fields are nicer
Change v13 to v2
Remove explicit relay handling
Update vendored github.com/cznic/ql (fixes#34)
Defer fd.Close() (fixes#37)
There is no "get dependencies" step
Add vendor/golang.org/x/net/context
Use Go 1.5 vendoring instead of Godeps
Add debug performance logging per request
Must close result sets
Set Retry-After header
Ignores
lru.Cache is not concurrency safe
We need a limit on the number of PostgreSQL connections
Correct example DSN (fixes#29)
Allow plain HTTP serving behind a proxy
Fix Query/Answer stats
Reduce our patience with slow clients somewhat
Discovery server should print device ID of certificate at startup
...
Git didn't really understand the multiple email addresses in the NICKS
file the same way I expected it to, and this fixes that. It also makes
AUTHORS the "master" file that everything else depends on, so it
now includes all of name, nickname and email addresses.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3243
Pull issue information from Github to show both the resolved issue
subject and the commit subject. Also show reviewer, when different from
author.
* #3201: api: /rest/system/browse behaves strangely on Windows
lib/osutil: Fix globbing at root (by @AudriusButkevicius, reviewed by
@calmh)
* #3174: Ignore patterns with non-ASCII characters causes out of memory
crash
vendor: Update github.com/gobwas/glob (by @calmh)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3228
With the previous setup, browsers were free to use a local cache for any
length of time they pleased: we didn't set an 'Expires' header (or max-age
directive), and Cache-Control just said "you're free to cache this".
Therefore be more explicit: we don't mind if browsers cache things, but they
MUST revalidate everything on every request.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3221
The log bar is hidden by CSS, but will appear briefly while the page is
loading (after the html is fetched, but before dev.css is fetched).
Hide it by using an inline style instead, so this does not happen.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3220
1. For the same internal port we ask for the same external port on all devices. This can be a problem if one device speaks over two protocols.
2. Always add a nil address even if we managed to get external address of the gateway, just because the gateway might be in DMZ behind another gateway.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3196
The intention for this package is to provide a combination of the
security of crypto/rand and the convenience of math/rand. It should be
the first choice of random data unless ultimate performance is required
and the usage is provably irrelevant from a security standpoint.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3186
The math/rand package contains lots of convenient functions, for example
to get an integer in a specified range without running into issues
caused by just truncating a number from a different distribution and so
on. But it's insecure, and we use if for things that benefit from being
more secure like session IDs, CSRF tokens and API keys.
This implements a math/rand.Source that reads from crypto/rand.Reader,
this bridging the gap between them. It also updates our RandomString to
use the new source, thus giving us secure session IDs and CSRF tokens.
Some future work remains:
- Fix API keys by making the generation in the UI use this code as well
- Refactor out these things into an actual random package, and audit
our use of randomness everywhere
I'll leave both of those for the future in order to not muddy the waters
on this diff...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3180
We need to set these properties *before* Angular starts making requests,
and doing that from the response to a request is too late. The obvious
choice (to me) would be to use the angular $cookies service, but that
service isn't available until after initialization so we can't use it.
Instead, add a special file that is loaded by index.html and includes
the info we need before the JS app even starts running.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3152
Switch to my forked version which contains a fix for this issue. I'll
track upstream in the future if things update there, and attempt to
contribute back fixes...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3149
Without this the summary service doesn't know to recalculate completion
percentage for remote devices when DownloadProgress messages come in.
That means that completion percentage isn't updated in the GUI while
transfers of large files are ongoing. With this change, it updates
correctly.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3144
I think this better reflects what it means. Also tweaks the verbose
format to be more like our other things and lightly refactors the code
to not have the boolean and include the folder in the event.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3121
This was fixed upstream due to our ticket, so we no longer need the
manual handling of commas. Keep the tests and better debug output around
though.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3081
The old usage pattern was to create a Walker with a bunch of attributes,
then call Walk() on it and nothing else. This extracts the attributes
into a Config struct and exposes a Walk(cfg Config) method instead, as
there was no reason to expose the state-holding walker type.
Also creates a few no-op implementations of the necessary interfaces
so that we can skip nil checks and simiplify things here and there.
Definitely look at this diff without whitespace.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3060
Just an optimization. Required exposing the priority from the factory,
so made that an interface with an extra method instead of just a func
type.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3071
This fixes the deadlock by reducing where we hold the various locks. To
start with it splits up the existing "mut" into a "listenersMut" and a
"curConMut" as these are the two things being protected and I can see no
relation between them that requires a shared lock. It also moves all
model calls outside of the lock, as I see no reason to hold the lock
while calling the model (and it's risky, as proven).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3069
When doing prefix scans in the database, "foo" should not be considered
a prefix of "foo2". Instead, it should match "foo" exactly and also
strings with the prefix "foo/". This is more restrictive than what the
standard leveldb prefix scan does so we add some code to enforce it.
Also exposes the initialScanCompleted on the rwfolder for testing, and
change it to be a channel (so we can wait for it from another
goroutine). Otherwise we can't be sure when the initial scan has
completed, and we need to wait for that or it might pick up changes
we're doing at an unexpected time.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3067
The VersioningConfig change is because it defaults to nil but gets
deserialized to map[string]string{}. Now prepare() enforces a single
representation of the empty map.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3065
Because json.NewDecoder(r).Decode(&v) doesn't necessarily consume all
data on the reader, that means an HTTP connection can't be reused. We
don't do a lot of HTTP traffic where we read JSON responses, but the
discovery is one such place. The other two are for POSTs from the GUI,
where it's not exactly critical but still nice if the connection still
can be keep-alive'd after the request as well.
Also ensure that we call req.Body.Close() for clarity, even though this
should by all accounts not really be necessary.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3050
New signature is the HMAC of archive name (which includes the release
version and architecture) plus the contents of the binary. This is
expected in a new file "release.sig" which may be present in a
subdirectory. The new release tools put this in [.]metadata/release.sig.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3043
1. Removes separate relay lists and relay clients/services, just makes it a listen address
2. Easier plugging-in of other transports
3. Allows "hot" disabling and enabling NAT services
4. Allows "hot" listen address changes
5. Changes listen address list with a preferable "default" value just like for discovery
6. Debounces global discovery announcements as external addresses change (which it might alot upon starting)
7. Stops this whole "pick other peers relay by latency". This information is no longer available,
but I don't think it matters as most of the time other peer only has one relay.
8. Rename ListenAddress to ListenAddresses, as well as in javascript land.
9. Stop serializing deprecated values to JSON
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2982
This happens automatically in the background anyway, and it can take a
long time on low powered devices at an inconvenient time. We just want
to get up and running as quickly as possible.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3000
A potential practical use is to encode a short version of the hostname
at the beginning of the device ID.
For example:
jb@syno:~/s/g/s/s/c/stvanity $ stvanity abc
Want 15 bits for prefix "ABC", about 3.3e+04 certs to test (statistically speaking)
Found ABCFPWS-JKDIFV3-E5IUAQW-DK53WVR-HY7XWBS-56H33GR-CJQI67Q-VGXRMAW
Saved to cert.pem, key.pem
jb@syno:~/s/g/s/s/c/stvanity $ stvanity $(hostname)
Want 20 bits for prefix "SYNO", about 1e+06 certs to test (statistically speaking)
Trying 554 certs/s, tested 8307 so far in 15s, expect ~32m total time to complete
Trying 543 certs/s, tested 16277 so far in 30s, expect ~32m total time to complete
...
The rest is just a matter of patience.
jb@syno:~/s/g/s/s/c/stvanity $ stvanity syncthing
Want 50 bits for prefix "SYNCTHI-NG", about 1.1e+15 certs to test (statistically speaking)
Trying 529 certs/s, tested 7941 so far in 15s, expect ~67443 years total time to complete
...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2986
When run without parameters, attempts to listen for local discovery
announcements just like Syncthing, and prints them.
With -send, it also sends fake discovery packets. This can be used on
two or more computers simultaneously to verify that they can see each
other.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/2985
Previously the code failed in that it would return top-level plus a sub,
i.e. ["", "foo"], and it would consider "usr/lib" a prefix of
"usr/libexec" which it is not.
By using data-original-title the tooltips live update without reapplying the
js code, such as .tooltip('fixTitle') each time the content changes. This
method also works well with angular expressions:
data-original-title="{{'Download Rate' | translate}}"
This example provides a bootstrap tooltip saying 'Download Rate' that changes
automatically when the language is updated.
Before this change, issuing either
systemctl --user help syncthing[.service]
or
systemctl help syncthing@user[.service]
gave the message
Can't show: http://docs.syncthing.net/
Following this change the syncthing man page is displayed
More prominent positions are given to authors with more commits, in
steps of magnitude. Authors with 100-999 commits are listed before
authors with 10-99 commits. Yes, this puts me at the head of the list
and is a slight ego trip, but I still think it's the right thing to do.
People want to add themselves to AUTHORS. That's fine, but it's not
enough as it also needs to be added to NICKS and script/authors.go needs
to be run. I'd rather have us do this and do it correctly so lets
document that people should not worry about it.
Change made by:
- running "gvt fetch" on each of the packages mentioned in
Godeps/Godeps.json
- `rm -rf Godeps`
- tweaking the build scripts to not mention Godeps
- tweaking the build scripts to test `./lib/...`, `./cmd/...` explicitly
(to avoid testing vendor)
- tweaking the build scripts to not juggle GOPATH for Godeps and instead
set GO15VENDOREXPERIMENT.
This also results in some updated packages at the same time I bet.
Building with Go 1.3 and 1.4 still *works* but won't use our vendored
dependencies - the user needs to have the actual packages in their
GOPATH then, which they'll get with a normal "go get". Building with Go
1.6+ will get our vendored dependencies by default even when not using
our build script, which is nice.
By doing this we gain some freedom in that we can pick and choose
manually what to include in vendor, as it's not based on just dependency
analysis of our own code. This is also a risk as we might pick up
dependencies we are unaware of, as the build may work locally with those
packages present in GOPATH. On the other hand the build server will
detect this as it has no packages in it's GOPATH beyond what is included
in the repo.
Recommended tool to manage dependencies is github.com/FiloSottile/gvt.
This systemd service restarts Syncthing after resume from suspend
via sending SIGHUP. By default Syncthing detects resume from sleep
on its own by looking for jumps in the system clock. Since systemd
knows exactly when the system resumes from sleep let's trigger
the Syncthing restart from there. Doing this in systemd eliminates
some annoying delay, as the service is restarted immediately after
resume. Also, using the systemd dependency mechanism syncthing-inotify
is restarted as well.
$ journalctl -e --identifier syncthing --identifier syncthing-inotify --identifier systemd
Feb 22 09:44:27 kronos systemd[1]: Reached target Sleep.
Feb 22 09:44:27 kronos systemd[1]: Starting Suspend...
Feb 22 09:44:33 kronos systemd[1]: Time has been changed
Feb 22 09:44:33 kronos systemd[963]: Time has been changed
Feb 22 09:44:33 kronos systemd[1]: Started Suspend.
Feb 22 09:44:33 kronos systemd[1]: sleep.target: Unit not needed anymore. Stopping.
Feb 22 09:44:33 kronos systemd[1]: Stopped target Sleep.
Feb 22 09:44:33 kronos systemd[1]: Reached target Suspend.
Feb 22 09:44:33 kronos systemd[1]: suspend.target: Unit is bound to inactive unit systemd-suspend.service. Stopping, too.
Feb 22 09:44:33 kronos systemd[1]: Stopped target Suspend.
Feb 22 09:44:33 kronos systemd[1]: Starting Restart Syncthing after resume...
Feb 22 09:44:33 kronos syncthing[2561]: [35K66] OK: Exiting
Feb 22 09:44:33 kronos systemd[1]: Started Restart Syncthing after resume.
Feb 22 09:44:34 kronos systemd[963]: syncthing.service: Service hold-off time over, scheduling restart.
Feb 22 09:44:34 kronos systemd[963]: Stopping Syncthing Inotify File Watcher...
Feb 22 09:44:34 kronos systemd[963]: Stopped Syncthing Inotify File Watcher.
Feb 22 09:44:34 kronos systemd[963]: Stopped Syncthing - Open Source Continuous File Synchronization.
Feb 22 09:44:34 kronos systemd[963]: Started Syncthing - Open Source Continuous File Synchronization.
Feb 22 09:44:34 kronos systemd[963]: Started Syncthing Inotify File Watcher.
Feb 22 09:44:34 kronos syncthing[2836]: [35K66] INFO: syncthing v0.12.19 "Beryllium Bedbug" (go1.5.3 linux-amd64) builduser@svetlemodry 2016-02-14 19:26:33 UTC
This system service has to be located in "/etc/systemd/system/syncthing-resume.service",
and for packages in "/usr/lib/systemd/system/syncthing-resume.service". It can be
enabled using "systemctl enable syncthing-resume.service".
Template for newly created issues. We want this text to be short and clear and request the required information from the user, and also be clear that it is a template and should be removed/replaced in the actual issue before saving...
Fixes#2151.
Since Walk.walkAndHashFiles ignores .stfolder and .stignore, they will
never be found by fs.Get(protocol.LocalDeviceID, sub) in
Model.internalScanFolder. As a result, when asked to scan those subs
we end up scanning the whole folder.
This reverts the change introduced in 9b9fe0d Reduce scanning effort.
That commit caused us to automatically ignore the basename of the
specified subs and instead scan closest known root folder. For
example, in a folder that looks like:
Sync/
├── 00
│ ├── one
│ ├── three
│ └── two
├── 01
│ ├── one
│ ├── three
│ └── two
├── 02
│ ├── one
│ ├── three
│ └── two
└── one
calling '/rest/db/scan?folder=default&sub=01' called filepath.Walk on
the whole Sync/ folder instead of just the desired subfolder. This
contradicts the scan behavior promised by the documentation.
This is related to #2151.
This fixes both a race condition where we could assign s.stop from one
goroutine and then read it from another without locking, and handles the
fact that listener may be nil at shutdown if we've had a bad
CommitConfiguration call in the meantime.
We only need to protect the integrity of the "finders" and "caches"
slices, and for that we only need an RLock except while actually
appending to them. The actual finders and caches are concurrency safe on
their own.
This updates the modified time of the config file before archiving it
during an update so that the clean up routine doesn't delete it if it's
too old, preventing the user from being able to rollback after an
upgrade.
This gets rid of redundant checks and centralizes the logic of loading
the config files so that we don't have to keep doing the same thing in
multiple places.
After the first media break (under 1200px), the footer is too long to
fit in a single line, taking up too much space in small screen devices.
This makes it so that it will stop being fixed at the bottom, freeing up
valuable screen real estate.
Safari has its own standard for handling icons for pinned tabs,
which requires a black-and-white .svg and a special tag.
Without using this, pinning a tab to localhost will show just
a blank square, instead of a pre-generated letter.
Checks the existing blocks that can be reused when downloading a file so
that it only requires the space corresponding to the missing blocks.
This will prevent syncthing from claiming the folder doesn't have enough
space when resuming download of large files after they have been
partially downloaded.
I'm not really sure under what circumstances MkdirAll returns a nil
error but a subsequent stat fails, but apparently it can happen and we
need to handle it. The "mode >= 0" was a no-op, and we never call
ensureDir anyway without the intention of ensuring the mode, so removed
that.
* pr/2594:
Move event timestamp to header for share folder prompt
Edit device after accepting new connection (fixes#1929)
Move new device timestamp to box header
Fix unmatching closing span tag
This will open the "edit device" dialogue after accepting a new device
connection. This will allow the user to specify the name of the device
or leave it blank in case they want to accept whatever the device
advertises once it connects.
When upgrade info is not available and the "Automatic Upgrades" options
is hidden, then "Global Discovery Server" wraps around and gets
misaligned. This fixes all that.
This avoids the double negative of having noConsole = false to represent
not hiding the console. It is also consistent with the action performed
by osutils.
This takes advantage of the newly created parseCommandLineOptions()
function and makes it work so that it now returns a nice struct of
options rather than relying on global variables.
There are a few global variables left, but they will take a bit more
refactoring in order to be removed, so it'll happen in later commits.
ensureDir() did not handle one last error case and there was some logic
in the main() function that belonged to ensureDir() as well. It was also
creating a directory with a hardcoded 0700 mode, regardless of what mode
was passed to it.
This refactors it a little to fix the broken behavior, avoid redundant
checks by taking advantage of the behavior of MkdirAll, and move the
extra logic from main() into ensureDir().
The main() function is growing too big (142 lines as of the date of this
commit), so this attempts to extract some functionality out of there and
into their own functions to make it easier to reason about them and keep
functions short and concise.
--browser-only assumes syncthing is already running and will open the
user's browser in the URL:port currently used in the configuration if
the GUI is enabled.
Every time a JSON object is returned in an HTTP response, the
appropriate header needs to be set and the object itself needs to be
encoded. Doing this in every function is repetitive and error prone
(getDBFile and postDBScan, for instance, never set any headers).
This adds a helper function to centralize the appropriate JSON response
handling.
PR #2578 enables us to remove the exit code 2 from the list of
success status codes, because SIGINT will be handled properly.
I have also converted STNORESTART to --no-restart for the sake
of consistency.
With this change, the behavior is as follows:
- SIGTERM / SIGINT: Exit cleanly (exit code 0)
- SIGHUP: Restart
This is the case both when hitting the monitor process or the inner
process, or if running with NORESTART (but then we won't restart,
obviously, just exit with code exitRestarting).
It enables "pkill -HUP syncthing" to do the right thing to restart all
Syncthings on package upgrade, for example.
Previously, when unmarshing the SOAP error code data we would overwrite
the original err, typically with null since the parsing of the error
code information succeeds. If we don't have a upnp 725 error, we would fall
back to returning null or no error. This broke our upnp error handling
logic for AddPortMappings as it would think it succeeds if it gets a 718
permission error.
Also fixes what I think migh thave been a bug where we did not use the
proxy for usage reports. And removes the BuildEnv field that we don't
need any more.
This is the same issue as #2014/#2062. Bootstrap doesn't like having two dialogs
open at once: it marks the body has having no dialogs open when the first dialog
is closed, regardless of whether the second dialog is still open.
This means that scrolling doesn't happen properly, and the user cannot
scroll to the dialog's 'close' button.
Work around this by making sure the first dialog (the settings page) is fully closed
before the second dialog (usage preview) is opened.
This replaces the current 3072 bit RSA certificates with 384 bit ECDSA
certificates. The advantage is these certificates are smaller and
essentially instantaneous to generate. According to RFC4492 (ECC Cipher
Suites for TLS), Table 1: Comparable Key Sizes, ECC has comparable
strength to 3072 bit RSA at 283 bits - so we exceed that.
There is no compatibility issue with existing Syncthing code - this is
verified by the integration test ("h2" instance has the new
certificate).
There are browsers out there that don't understand ECC certificates yet,
although I think they're dying out. In the meantime, I've retained the
RSA code for the HTTPS certificate, but pulled it down to 2048 bits. I
don't think a higher security level there is motivated, is this matches
current industry standard for HTTPS certificates.
- Move to ipinfo.io for geoip, rather than Telize. Telize has been closed
down. ipinfo.io has apparently got decent availability, and allows
1,000 requests per day on the free tier. Since requests are made by the
client, this should be more than enough (and the total across all clients
should still be less than this).
- Fix issue where one nonresponsive relay would cause 'data unavailable'
to be shown for many relays. This was caused by the relay status
promise not being correctly added to the list of things being waited
for before the map was rendered. Any delayed relay status requests
would therefore occur after the map was rendered, which was too late.
This reverts commit 81bc6bf34b, reversing
changes made to 7de736e8d0.
Unfortunately this tricks the upgrade system into picking the wrong
binary. We need to fix the upgrade system before merging this.
Knowing why a relay server failed to join the pool can be important. This
is typically an issue which must be investigated after it occurred, so
having logs available is useful.
Running with -debug permanently enabled is impractical, due to the amount
of traffic that is generated, particularly when data is being transferred.
Logging is limited to at most one message per minute, although one message
per hour is more likely.
This allows relaysrv to listen on an unprivileged port, with port
forwarding directing traffic from 443, thus providing an alternative
to using setcap cap_net_bind_service=+ep
We're going to need the db.Instance to keep some state, and for that to
work we need the same one passed around everywhere. Hence this moves the
leveldb-specific file opening stuff into the db package and exports the
dbInstance type.
By using copyBuffer we avoid a buffer allocation for each block we hash,
and by allocating space for the hashes up front we get one large backing
array instead of a small one for each block. For a 17 MiB file this
makes quite a difference in the amount of memory allocated:
benchmark old ns/op new ns/op delta
BenchmarkHashFile-8 102045110 100459158 -1.55%
benchmark old allocs new allocs delta
BenchmarkHashFile-8 415 144 -65.30%
benchmark old bytes new bytes delta
BenchmarkHashFile-8 4504296 48104 -98.93%
'AlwaysLocalNets' was getting printed, but was getting used
when setting up connections. Now, the nets that should be
considered local are printed and used.
Did some manual tests in the playground, such as kicking off two clients in parallel, first connecting,
second one getting a message about already being connected, falling back to the second address.
Overwriting configuration files is likely to happen if a
user syncs their home directories across computers. In this
case, the biggest risk is that all nodes will end up with
the same certificate and thus Device ID.
When the model prepares a folder for syncing, it checks to
see if the configuration files this instance is using are
getting synced. If the are getting synced, and they aren't
getting ignored, a warning is emitted. The model is used
so that when a new folder is added dynamically, a warning
is also emitted.
This will not prevent a user from shooting themselves in
the foot, and will not cover all cases (e.g. symlinks).
It should provide _something_ for many users in this
situation to go on, though.
Just incase we want to show some stats in the future, such as a Geo-IP based map of where relays are, their dot size being proportional to global rate limits,
together with potentially how much data in total has been transferred, and how many sessions there by crawling relay status pages etc ;)
Add WorkingDirectory to create and use the certificates within
/var/lib/syncthing-relaysrv. Add RootDirectory to chroot(2) the whole
thing into that directory.
This implements a new debug/trace infrastructure based on a slightly
hacked up logger. Instead of the traditional "if debug { ... }" I've
rewritten the logger to have no-op Debugln and Debugf, unless debugging
has been enabled for a given "facility". The "facility" is just a
string, typically a package name.
This will be slightly slower than before; but not that much as it's
mostly a function call that returns immediately. For the cases where it
matters (the Debugln takes a hex.Dump() of something for example, and
it's not in a very occasional "if err != nil" branch) there is an
l.ShouldDebug(facility) that is fast enough to be used like the old "if
debug".
The point of all this is that we can now toggle debugging for the
various packages on and off at runtime. There's a new method
/rest/system/debug that can be POSTed a set of facilities to enable and
disable debug for, or GET from to get a list of facilities with
descriptions and their current debug status.
Similarly a /rest/system/log?since=... can grab the latest log entries,
up to 250 of them (hardcoded constant in main.go) plus the initial few.
Not implemented in this commit (but planned) is a simple debug GUI
available on /debug that shows the current log in an easily pasteable
format and has checkboxes to enable the various debug facilities.
The debug instructions to a user then becomes "visit this URL, check
these boxes, reproduce your problem, copy and paste the log". The actual
log viewer on the hypothetical /debug URL can poll regularly for new log
entries and this bypass the 250 line limit.
The existing STTRACE=foo variable is still obeyed and just sets the
start state of the system.
Not necessarily the easiest way to fix just this bug, but the root cause
was using the (at that point uninitialized) cfg variable, so it seemed
sensible to just get rid of it to avoid that kind of crap.
An error on opening .stignore will satisfy os.IsNotExist() and not be
reported. Other errors will be reported and stop the folder, including
is-not-exist errors from #include as these are passed through fmt.Errorf.
Also fixes minor issue where we would not print cause of folder stopping
to the log.
Also fixes minor issue with capitalization of errors.
I just wanted to add the freenode webchat link, because people who are
not used to irc can join the chatroom instantly. I tried to clean up the
markdown file a bit and removed the links to the footer; that makes the
"source code" less ugly.
The connections service no longer depends directly on the
syncthing model object, but on an interface instead. This
makes it drastically easier to write clients that handle
the model differently, but still want to benefit from
existing and future connections changes in the core.
This was motivated by burkemw3's interest in creating a
FUSE client that can present a view of the global model,
but not have all of the file data locally.
The actual decoupling was done by adding a connections.Model
interface. This interface is effectively an extension of the
protocol.Model interface that also handles connections
alongside the modified service.
The XDR encoder doesn't understart slices of strings very well. It can
encode and decode them, but there's no way to set limits on the length
of the strings themselves (only on the length of the slice), and the
generated diagrams are incorrect. This trivially works around this,
while also documenting what the string actually is (a URL).
This should resolve the spurious ping timeouts we've had on low powered
boxes. Those errors are the result of us requiring a timely Pong
response to our Pings. However this is unnecessarily strict - as long as
we've received *anything* recently, we know the other peer is alive. So
the new mechanism removes the Pong message entirely and separates the
ping check into two routines:
- One that makes sure to send ping periodically, if nothing else has
been sent. This guarantees a message sent every 45-90 seconds.
- One that checks how long it was since we last received a message. If
it's longer than 300 seconds, we trigger an ErrTimeout.
So we're guaranteed to detect a connection failure in 300 + 300/2
seconds (due to how often the check runs) and we may detect it much
sooner if we get an actual error on the ping write (a connection reset
or so).
This is more sluggish than before but I think that's an OK price to pay
for making it actually work out of the box.
This removes the configurability of it, as the timeout on one side is
dependent on the send interval on the other side. Do we still need it
configurable?
This makes it so we can initialize the relay management and then give
that to the connection management, instead of the other way around.
This is important to me in the discovery revamp I'm doing, as otherwise
I get a circular dependency when constructing stuff, with relaying
depending on connection, connection depending on discovery, and
discovery depending on relaying.
With this fixed, discovery will depend on relaying, and connection will
depend on both discovery and relaying.
Instead, make sure we do the check as part of CheckFolderHealth before
pulling, and individually per file to try to not run out of space at
that stage.
(The latter is far from fool proof as we may pull lots of stuff in
parallell, but it's worth a try.)
This makes it possible to run multiple instances on the same box, all
receiving local discovery packets. Tested on Mac, Windows, supposed to
work on at least Linux too. For Windows, there may be issues with XP and
earlier, but meh...
Either Angular or the browser sometimes returns cached repsonse header,
causing a flap between requests that return the new version and requests
that return the old one. Here, instead, we trust the actual data
returned by the uncached /rest/system/version call.
Resetting the timeout doesn't fully cut it, as it may timeout after we
got an event and be delivered later. This should fix it well enough for
the moment. https://github.com/golang/go/issues/11513
* syncthing/pr/1995:
Add switch to disable relays
Do not start relay service unless explicitly asked for, or global announcement server is running
Add dynamic relay lookup (DDoS relays.syncthing.net!)
Discovery clients now take an announcer, global discovery is delayed
Expose connection type and relay status in the UI
Add dependencies (fixes#1364)
Check relays for available devices
Add incoming connection relay service
Add unsubscribe to config
Connections have types
Large refactoring/feature commit
1. Change listen addresses to URIs
2. Break out connectionSvc to support listeners and dialers based on schema
3. Add relay announcement and lookups part of discovery service
I figured we're missing out on being cool and awesome by not having an
alphabetically based release code name like the big guys. This commit
fixes that. I've unilaterally decided on a theme of "$metal $bug"
because metals are kind of cool, and bugs, well, ...
The XDR unmarshaller allocates a []T when it sees a slice type and reads
the expected length, so we must always limit the length in order to
avoid allocating too much memory when encountering corruption.
We allocate a []byte to read the message into, so if the header says the
messages is several gigabytes large we may run into trouble. In reality,
a message should never be that large so we impose a limit.
This fixes a corner case I discovered in the symlink branch, where we
unexpectedly succeed in "replacing" an entire non-empty directory tree
with a file or symlink. This happens when archiving is in use, as we
then just move the entire tree away into the archive. This is wrong as
we should just archive files and fail on non-empty dirs in all cases.
New handling first checks what the (old) thing is, and if it's a
directory or symlink just does the delete, otherwise does conflict
handling or archiving as appropriate.
This will decrease the risk of running out of file descriptors for the
database and other bad things, which could otherwise potentially happen
if we're serving lots of requests and scanning in parallel, etc.
Windows doesn't have a per process open file limit like Unix so we don't
need to worry about it there.
The number of copiers and pullers is set to default at config loading
time, but the new folder configuration doesn't pass through config
loading so we start up with 0 copiers and 0 pullers and hence get stuck.
I moved the default handling to the puller itself instead. I think this
way is also cleaner as we get to keep the 0 in the config and the puller
gets to decide the defaults on it's own.
- remove Glyphicon assets and customize bootstrap CSS
- add Font Awesome v4.4.0 assets
- replace Glyphicons with Font Awesome icons in HTML
- add icons to modal headers
- add attribution for Font Awesome
- format HTML source code for buttons
- Move the Go files into script/ instead of random places
- Rewrite check-contrib.sh into check-authors.go and check-copyright.go
- Clean up build.sh a little bit
* v0.11:
Translations and docs update
Enable browser caching of static resources
Handle multiple case insensitivity prefixes in ignores (fixes#2134)
Make rescan available for unshared folders
Add timeout for peek (fixes#1035)
Fix TestReset when Syncthing shuts down too fast
Clarify password in integration tests
Properly rename config files during integration tests (fixes#1769)
This sends the Cache-Control header to allow caching of static resources,
and checks the If-Modified-Since header to allow browser to use the
cached resource on refresh.
* pr-2066:
Configurable home disk percentage, translations
Add minimum disk free percentage to GUI
Stop folder when running out of disk space (fixes#2057)
Setting "Sync Protocol Listen Addresses" and "Global Discovery
Server" in the Settings dialog does not work. This bug seems to
have been introduced in 26d52be.
This sends the Cache-Control header to allow caching of static resources,
and checks the If-Modified-Since header to allow browser to use the
cached resource on refresh. Also fixes some paths that caused redirects
(core//foo -> core/foo)
The very deep tree structure didn't really aggree with me, sorry. This
makes the core module rather large, but on the other hand that just
highlights that it is rather large.
Without this, as soon as we'd touched 1000 files in the last minute
(which can happen), we got stuck doing cache cleaning all the time,
burning a lot of CPU time.
This captures the common pattern of writing to a temp file and moving it
to it's real name only if everything went well. It reduces the amount of
code in some places where we do this, but maybe not as much as I
expected because the upgrade thing is still a special snowflake...
commit d60fbce311
Author: Jacek Szafarkiewicz <szafar@linux.pl>
Date: Mon Jun 1 11:16:36 2015 +0200
Correct order of deb files
commit 3b2ecfcc45
Merge: f4daebbc23a601
Author: Jacek Szafarkiewicz <szafar@linux.pl>
Date: Mon Jun 1 11:15:06 2015 +0200
Merge github.com:syncthing/syncthing
Conflicts:
build.go
commit f4daebb851
Author: Jacek Szafarkiewicz <szafar@linux.pl>
Date: Tue May 26 12:58:25 2015 +0200
Add me to AUTHORS
commit 9e77f4bea0
Author: Jacek Szafarkiewicz <szafar@linux.pl>
Date: Tue May 26 12:57:40 2015 +0200
Add systemd files to deb packate
class modal-open is applied to <body>, which ultimately means that the
browser will scoll to the modal's content. However, #editFolder was
finishing its close animation (and removing this modal-open class)
after #editIgnores had set modal-open (and had started its open
animation). The end result is that <body> ends up without modal-open
when #editIgnores is open, and so the browser doesn't properly scroll.
Instead, only open the #editIgnores once #editFolder has finished closing.
This captures the common pattern of writing to a temp file and moving it
to it's real name only if everything went well. It reduces the amount of
code in some places where we do this, but maybe not as much as I
expected because the upgrade thing is still a special snowflake...
Upgrading to bootstrap 3.3.5 meant that checkboxes inside a div with
column-count:3 set would be unclickable in Chrome: in fact, the entire
div appears to sit on top of its contents, making interaction impossible.
This affected both the 'show folder with these devices' and 'these devices
can access this folder' sections of the UI.
I'm not sure what the the underlying cause is, but moving to Bootstrap's
grid system appears work around the issue. Devices/folders have to be
explicitly split out into rows, otherwise the final element appears offset.
To do this grouping by row, a new filter (groupFilter) has been added, which
turns an input of e.g. [1, 2, 3, 4, 5] with a groupSize of 3 into
[[1, 2, 3], [4, 5]]. However altering the collection in this way throws
Angular into an infinite watch loop, terminating in infdig. m59peacemaker's
pmkr.filterStabilize (MIT) was added to work around this issue.
This also has the nice side-effect of wrapping the list of devices/folders
when the screen width decreases.
See also:
- #2027 (bootstrap update which triggered this issue)
- #1121 (last time it happened)
Added myself to the AUTHORS, NICKS and GUI contributors files. Also
fixed the sort order in AUTHORS and NICKS when adding myself as there
were a couple of entries in both that were not quite in alphabetical
order.
Renaming them puts the user in a difficult situation as they can't
rename them back in the GUI. This way, they need to fix the config in
the same way it got broken (manual editing or external tool).
The reset of all folders failed when there was no data for a given
folder, as it was not returned by db.ListFolders then. But we don't
really care about that, we can "reset" it anyway...
The override should look like we detected the changes locally, or the
GUI and other things won't update correctly.
This is/was caught by the Override integration test, on my newly
refactored integration test suite which is soon ready for prime time, so
a test is coming. :)
The API never got the first few events ("Starting" etc) as it subscribed
too late. Instead, set up a subscription for it early on. If the API is
configured not to run this is unnecessary but doesn't hurt very much.
This skips the warning about "unsupported symlinks" for invalid or
deleted symlinks (and as a side effect also accepts them into the index,
which should be fine). It also prints the affected file paths to the
log. This should be in the hypothetical list of "errored files" we
should present instead of the cryptic "puller stopped" message in the
future...
My theory is that some error condition on the socket results in it
blocking for writes, which maybe also blocks reads... This separates the
two into separate services with their own socket, with restarts and
retries as appropriates on write timeouts and read/write errors. It
should be more robust, hopefully, but I have a hard time testing the
actual error conditions...
We introduced the dbUpdater routine to handle many small files
efficiently, but the folder stats call is almost equally expensive as it
results in two distinct write transactions to the database. This moves
it to the same routine.
(Doesn't make a *huge* difference with leveldb actually, but reduces the
50k-files benchmark time by 25% on my experimental bolt branch...)
This is surprisingly memory expensive when Poll gets called a lot, such
as when syncing lots of small files generating itemstarted/itemfinished
events. It's line number three in this heap profile on the
TestBenchmarkManyFiles test:
jb@syno:~/src/github.com/syncthing/syncthing/test (master) $ go tool pprof ../bin/syncthing heap-13194.pprof
Entering interactive mode (type "help" for commands)
(pprof) top
80.91MB of 83.05MB total (97.42%)
Dropped 1024 nodes (cum <= 0.42MB)
Showing top 10 nodes out of 85 (cum >= 1.75MB)
flat flat% sum% cum cum%
32MB 38.53% 38.53% 32.01MB 38.54% github.com/syndtr/goleveldb/leveldb/memdb.New
22.16MB 26.68% 65.21% 22.16MB 26.68% github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
13.02MB 15.68% 80.89% 13.02MB 15.68% time.NewTimer
6.94MB 8.35% 89.24% 6.94MB 8.35% github.com/syndtr/goleveldb/leveldb/memdb.(*DB).Put
3.18MB 3.82% 93.06% 3.18MB 3.82% github.com/calmh/xdr.(*Reader).ReadBytesMaxInto
With this change the allocation is removed entirely.
Integers are for numbers, enabling arithmetic like subtractions and for
loops without getting shot in the foot. Unsigneds are for bitfields.
- "int" for numbers that will always be laughably smaller than four
billion, and where we don't care about the serialization format.
- "int32" for numbers that will always be laughably smaller than four
billion, and will be serialized to four bytes.
- "int64" for numbers that may approach four billion or will be
serialized to eight bytes.
- "uint32" and "uint64" for bitfields, depending on required number of
bits and serialization format. Likewise "uint8" and "uint16", although
rare in this project since they don't exist in XDR.
- "int8", "int16" and plain "uint" are almost never useful.
This change makes sure that things work smoothly when "we" are a newer
version than our peer and have more fields in our messages than they do.
Missing fields will be left at zero/nil.
(The other side will ignore our extra fields, for the same effect.)
Request to terminate currently ongoing downloads and jump to the bumped file
incoming in 3, 2, 1.
Also, has a slightly strange effect where we pop a job off the queue, but
the copyChannel is still busy and blocks, though it gets moved to the
progress slice in the jobqueue, and looks like it's in progress which it isn't
as it's waiting to be picked up from the copyChan.
As a result, the progress emitter doesn't register on the task, and hence the file
doesn't have a progress bar, but cannot be replaced by a bump.
I guess I can fix progress bar issue by moving the progressEmiter.Register just
before passing the file to the copyChan, but then we are back to the initial
problem of a file with a progress bar, but no progress happening as it's stuck
on write to copyChan
I checked if there is a way to check for channel writeability (before popping)
but got struck by lightning just for bringing the idea up in #go-nuts.
My ideal scenario would be to check if copyChan is writeable, pop job from the
queue and shove it down handleFile. This way jobs would stay in the queue while
they cannot be handled, meaning that the `Bump` could bring your file up higher.
New node ID:s contain four Luhn check digits and are grouped
differently. Code uses NodeID type instead of string, so it's formatted
homogenously everywhere.
description:Please describe the behavior you'd like to see.
validations:
required:true
- type:textarea
id:problem-usecase
attributes:
label:Problem or use case
description:Please explain which problem this would solve, or what the use case is for the feature. Keep in mind that it's more likely to be implemented if it's generally useful for a larger number of users.
validations:
required:true
- type:textarea
id:alternatives
attributes:
label:Alternatives or workarounds
description:Please describe any alternatives or workarounds you have considered and, possibly, rejected.
description:If you're actually looking for support instead, see "I need help / I have a question".
labels:["bug","needs-triage"]
type:Bug
body:
- type:markdown
attributes:
value:|
:no_entry_sign: If you want to report a security issue, please see [our Security Policy](https://syncthing.net/security/) and do not report the issue here.
:interrobang: If you are not sure if there is a bug, but something isn't working right and you need help, please [use the forum](https://forum.syncthing.net/).
- type:textarea
id:what-happened
attributes:
label:What happened?
description:Also tell us, what did you expect to happen, and any steps we might use to reproduce the problem.
placeholder:Tell us what you see!
validations:
required:true
- type:input
id:version
attributes:
label:Syncthing version
description:What version of Syncthing are you running?
placeholder:v1.27.4
validations:
required:true
- type:input
id:platform
attributes:
label:Platform & operating system
description:Onwhat platform(s) are you seeing the problem?
placeholder:Linux arm64
validations:
required:true
- type:input
id:browser
attributes:
label:Browser version
description:If the problem is related to the GUI, describe your browser and version.
placeholder:Safari 17.3.1
- type:textarea
id:logs
attributes:
label:Relevant log output
description:Please copy and paste any relevant log output or crash backtrace. This will be automatically formatted into code, so no need for backticks.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
