Currently, the QR code image has no size specified in the HTML. This
causes other elements of the modal to jump around when opening it for
the first time and waiting for the QR image to generate. The jumping is
especially noticeable when the GUI is accessed remotely and there is a
delay while loading the elements due to slow connection.
In addition, capitalise "QR" in the alt text. This is necessary for
accessibility reasons, e.g. when using a screen reader, which may read
upper- and lowercase characters differently.
Lastly, allow to translate the alt text itself.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Currently, a custom JS script is used to select the whole device ID on
click. However, the current script isn't compatible with all browsers
(and in IE in particular), making it impossible to select the ID in them
at all. Additionally, the same functionality is already available in CSS
with no such drawbacks, as the whole selection process is handled by the
Web browser natively, which is lightweight and does not require custom
code.
Thus, remove the currently used JS script completely, replacing it with
a new CSS class that can be added to an element when required. If the
browser does not support the CSS, the user can still select the element
manually, which makes it safer than the current behaviour that can block
the user from being able to select the element at all.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The restore function of Trash Can ran a rename at the end regardless of whether there was anything to rename. In this case, when the file-to-be-restored did not exist in the destination folder, this resulted in an error. I added a simple check, keeping track of whether the file existed prior to restoring it in the destination folder and depending on this value it will now return nil after the restoration to prevent the renaming function to kick off. Added a test for this specific edge-case as well.
For consistency with syncOwnership explanation right above this one, the
sentence should talk about "sending ownership information", and not just
"sending ownership".
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Duplicate the regular expression for single and double quotes.
Support additional arguments (string substitution) in both variants.
Simplify the translation string group matching by using a lazy
quantifier instead of excluding the quote itself.
This adds support for syncing extended attributes on supported
filesystem on Linux, macOS, FreeBSD and NetBSD. Windows is currently
excluded because the APIs seem onerous and annoying and frankly the uses
cases seem few and far between. On Unixes this also covers ACLs as those
are stored as extended attributes.
Similar to ownership syncing this will optional & opt-in, which two
settings controlling the main behavior: one to "sync" xattrs (read &
write) and another one to "scan" xattrs (only read them so other devices
can "sync" them, but not apply any locally).
Co-authored-by: Tomasz Wilczyński <twilczynski@naver.com>
The cleaning logic in util.go was used by Simple and Trashcan but only
really suited Trashcan since it works based on mtimes which Simple does
not use. The cleaning logic in util.go was moved to trashcan.go.
Staggered and Simple seemed to be able to benefit from the same base so
util.go now has the base for those two with an added parameter which
takes a function so it can still handle versioner-specific logic to
decide which files to clean up. Simple now also correctly cleans files
based on their time-stamp in the title together with a specific maximum
amount to keep. The Archive function in Simple.go was changed to get rid
of duplicated code.
Additionally the trashcan testcase which was used by Trashcan as well as
Simple was moved from versioner_test.go to trashcan_test.go to keep it
clean, there was no need to keep it in a separate test file
Currently, the code contains a "mobile phone" fix to allow wrapping of
long lines in table heading and cells. However, the fix is applied to
all screen sizes equal or below 768 px wide, which causes the layout to
break on tablet-sized screens.
The commit moves the "mobile" fix to the actual mobile media query,
which is applied to screens up to 419 px wide. It is only really needed
there, where it synergises with the existing fix that changes table cell
display to "block". There is no need to wrap the text on larger screens,
as there is more than enough space to display the lines in full on them.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Add a commented entry to the systemd service file templates to point
the user in the right direction when using syncOwnership and starting
via systemd. Which is more upgrade-friendly than setting caps on the
executable directly, as mentioned in the docs.
The current code checks whether the same-named item exists in the tree,
and when it does, it re-uses it when adding new children to it. However,
the code doesn't check whether the existing item is a folder or a file.
It rather assumes that it is always a folder, which is not necessarily
the case.
This commit adds a new check to the code, so that the existing element
is reused only when it is a folder, and ignored otherwise.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Currently, the error message is often quite long and thus it appears
truncated with no possibility for the user to view the full string.
Thus, add a tooltip that displays the message in full on hover. This
follows the convention used in other parts of the GUI in similar cases.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This replaces old style errors.Wrap with modern fmt.Errorf and removes
the (direct) dependency on github.com/pkg/errors. A couple of cases are
adjusted by hand as previously errors.Wrap(nil, ...) would return nil,
which is not what fmt.Errorf does.
Locally paused folders will fail on checkFolderRunningLocked() and
therefore abort the loop. Avoid this by skipping paused folders
directly.
Co-authored-by: Jakob Borg <jakob@kastelo.net>
These strings no longer exist in the GUI, hence there is no need to keep
them in the translation json files either.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
* lib/locations: Fix enum values camelCase.
* lib/locations: Remove unused FailuresFile.
* cmd/syncthing: Turn around role of locations storage.
Previously the locations package was used to provide default paths,
possibly with an overridden home directory. Extra paths supplied on
the command line were handled and passed around in the options object.
To make the changed paths available to any other interested package,
override the location setting from the option if supplied, instead of
vice versa when not supplied. Adapt code using this to read from the
locations package instead of passing through the options object.
* lib/locations: Refactor showPaths to locations package.
Generate a reusable string in locations.PrettyPrintPaths().
Enumerating all possible locations in different packages is error
prone, so add a new public function to generate the listing as a
string in the locations package. Adapt cmd/syncthing --paths to use
that instead of its own console output.
* lib/locations: Include CSRF token in pretty printed paths.
* lib/api: New endpoint /rest/system/paths.
The paths should be available for troubleshooting from a running
instance. Using the --paths CLI option is not easy in some
environments, so expose the locations mapping to a JSON endpoint.
Add utility function ListExpandedPaths() that also filters out any
entries which still contain variable placeholders.
* gui: List runtime paths in separate log viewer tab.
* Wrap paths.
* lib/syncthing: Utilize locations.Get() instead of passing an arg.
* Include base directories, move label to table caption.
* gui: Switch to hard-coded paths instead of iterating over all.
* gui: Break aboutModalView into tabs.
Use tabs to separate authors from included third-party software.
* gui: Move paths from log viewer to about modal.
* lib/locations: Adjust pretty print output order to match GUI.
* gui, authors: Remove additional bot names and fix indent.
The indentation changed because of the tabbed about dialog, fix the
authors script to respect that.
Skip Syncthing*Automation in authors list as well.
* Update AUTHORS list to remove bot names.
* Revert AUTHORS email order change.
* Do not emphasize DB and log file locations.
* Review line wrapping.
* review part 1: strings.Builder, naming
* Rename and extend locations.Set() with error handling.
Remodel the Override() function along the existing SetBaseDir() and
rename it to simply Set(). Make sure to use absolute paths when given
log file or GUI assets override options. Add proper error reporting
if that goes wrong.
* Remove obsolete comment about empty logfile option.
* Don't filter out unexpanded baseDir placeholders, only ${timestamp}.
* Restore behavior regarding special "-" logfile argument.
If the option is given, but with empty value, assume the no log
file (same as "-"). Don't try to convert the special value to an
absolute path though and document this fact in a comment for the Set()
function.
* Use template to check for location key validity.
* Don't filter out timestamp placeholders.
* lib/api: Remove paths from /rest/system/status.
* lib/ur: Properly initialize map in failure data (fixes#8479)
Co-authored-by: Jakob Borg <jakob@kastelo.net>
Currently the `Run()` function of the CLI always uses `os.Args` directly.
This change adds an additional `RunWithArgs()` function that allows passing
arguments as `[]string` instead. This is useful when linking against
Syncthing as a library to be able to also expose the CLI.
Also adds idle time and keepalive parameters because how this is
configured has changed in the new package version. The values are those
that seems like might already be default, if keep-alives were enabled,
which is not obvious from the doc comments.
Also, Go 1.19 gofmt reformatting of comments.
all: Add package runtimeos for runtime.GOOS comparisons
I grew tired of hand written string comparisons. This adds generated
constants for the GOOS values, and predefined Is$OS constants that can
be iffed on. In a couple of places I rewrote trivial switch:es to if:s,
and added Illumos where we checked for Solaris (because they are
effectively the same, and if we're going to target one of them that
would be Illumos...).
Currently, there are two issues with the detailed staggered versioning
information displayed in the folder info. Firstly, the maxAge value of
365d is displayed despite matching the default. Secondly, there is no
consideration for the special case of maxAge equal to 0, meaning that
versions are kept forever.
This commit fixes both of those issues, so that the default maxAge is
not displayed and its value of 0 is displayed as "forever".
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds support for syncing ownership on Unixes and on Windows. The
scanner always picks up ownership information, but it is not applied
unless the new folder option "Sync Ownership" is set.
Ownership data is stored in a new FileInfo field called "platform data". This
is intended to hold further platform-specific data in the future
(specifically, extended attributes), which is why the whole design is a
bit overkill for just ownership.
* lib/model: Override scan config for auto-accepted encrypted folders.
Encrypted folders should not have the fs watcher enabled and rarely
benefit from a scheduled rescan. The GUI adjusts the suggested
settings (watcher disabled, one day rescan interval) when accepting a
receive-encrypted folder. Mirror that behavior to the auto-accept
case where the GUI is not involved.
Versioning also does not work well for encrypted folders, same
treatment.
Currently, the filesystem watcher explanation in the Advanced tab in the
Edit Folder modal window is split into two parts. The first is location
in a tooltip that is visible only when hovering the mouse over the small
checkbox, which makes it not easily accessible, e.g. for new or touch
screen users.
No other settings in the Edit Folder window have their explanation
presentend like that. When needed, it is always displayed in their
respective help blocks, which are always visible on the screen. Thus,
move the filesystem watcher explanation from the tooltip to the help
block, merging it with the other part of the explanation in the process.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* gui: Use discovered IDs from cache when adding a new remote device.
The GUI already does refreshDiscoveryCache() when it comes online and
on every 10 second refresh cycle. Querying the same information
freshly in addDevice() seems unnecessary and adds some more round-trip
delay on a possibly slow network link. Instead use the IDs from the
existing discoveryCache property to populate the suggested ID list.
* Increase maximum suggested device ID count to 100.
For the auto-completion list, which is hidden by default and filtered
by the browser, we can offer more discovered device IDs without
causing much confusion. The list of suggested "nearby" devices is
still limited to the first five.
* Rename $scope.discoveryUnknown.
The old name "discovery" was pretty ambiguous..
This commit updates Fork Awesome from version 1.1.2 from 2018 to version
1.2.0 from 2021. The changes are few and non-breaking, consisting mainly
of new icon additions.
It is worth to note that the new version includes also a Syncthing icon,
which was not present before.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
An off-by-one error could cause tokens to be forgotten. Suppose
tokens := []string{"foo", "bar", "baz", "quux"}
i := 2
token := tokens[i] // token == "baz"
Then, after
copy(tokens[1:], tokens[:i+1])
tokens[0] = token
we have
tokens == []string{"baz", "foo", "bar", "baz"}
The short test actually relied on this bug.
Similar to the "remote has not accepted sharing" message, add a
footnote number 2 to indicate a folder which will not sync with a
certain device because the remote has paused it. Applies to the edit
folder / device modals' sharing tab, as well as the "shared with"
listing and tooltips under device and folder details.
Use the proper encoding function in the relay server when constructing
the URL. In the pool server, parse and re-encode the query values to
sanitize whatever the client sent.
Currently, the "Latest Change" translation string is hard-coded to use
the English-like word order of V ("deleted") + N ("file"). As such, it
is incompatible with languages that require to use a different word
order, e.g. Korean or Japanese. In other words, a proper translation of
the string to those languages is currently impossible.
This commit changes the translation string, so that it includes the file
variable, and thanks to this, it can be easily adopted to languages with
different word order than English.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Showing all folders from disconnected or paused remote devices as
unaccepted would be a lot of false positives. As we cannot know
whether the remote has accepted while it doesn't have an active
connection, let's better report false negatives, as in assuming the
folders are accepted.
* lib/api: Note ItemStarted and ItemFinished for default filtering.
The reasoning why LocalChangeDetected and RemoteChangeDetected events
are not included in the event stream by default (without explicit
filter mask requested) also holds for the ItemStarted and ItemFinished
events. They should be excluded as well when we start to break the
API compatibility for some reason.
* gui: Enumerate unused event types in the eventService.
Define constants for the unused event types as well, for completeness'
sake. They are intentionally not handled in the GUI currently.
* cmd/syncthing: Harmonize uppercase CLI argument placeholders.
Use ALL-UPPERCASE and connecting dashes to distinguish argument
placeholders from literal argument options (e.g. "cpu" or "heap" for
profiling). The dash makes it clear which words form a single
argument and where a new argument starts.
This style is already used for the "syncthing cli debug file" command.
* lib/model: Simplify event data structure.
Using map[string]interface{} is not necessary when all values are
known to be strings.
Having a separate mutex for the three or four instructions needed to
fetch and increment nextID means the overhead exceeds the cost of this
operation. nextID is now handled inside the critical section for
awaiting instead, while the more expensive channel creation has been
moved outside it.
This is mostly a simplification, though it may have minor performance
benefits in some situations. The single-threaded sender benchmark shows
no significant difference:
name old speed new speed delta
RequestsRawTCP-8 55.3MB/s ± 7% 56.6MB/s ± 6% ~ (p=0.190 n=10+10)
RequestsTLSoTCP-8 20.5MB/s ±20% 20.8MB/s ± 8% ~ (p=0.604 n=10+9)
Add a link next to each setting's label to its explanation in the Docs.
This way, it is easy to quickly check what the setting is about without
going to the Docs site separately and searching for it manually.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Right now, the Trash Can versioning info text is displayed for both the
Trash Can and Simple versioning. However, the Simple versioning has its
own info text that is also displayed, which results in two very similar
sentences being shown on the screen.
This commit limits the Trash Can info text to be displayed only when the
Trash Can versioning is selected, and moves the Simple versioning info
text up to the same location as the other one.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Even though technically possible, CJK languages normally don't use
italic text at all, as not only does it make the characters/letters look
unnatural, but also, in the case of complex characters, unreadable too.
For these reasons, it is usually recommended not to use the italic font
style at all [1][2].
This commit changes the default font-style of the i element for Chinese,
Japanese, and Korean langauge to "normal" instead of "italic". In order
to do so, the HTML lang attribute is also changed following each change
of the GUI language.
[1] https://bobtung.medium.com/best-practice-in-chinese-layout-f933aff1728f
[2] https://devblogs.microsoft.com/oldnewthing/20060914-02/?p=29743
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Staggered File Versioning used to have its own cleanInterval that
controlled how often file versions were cleaned. Nowadays, there is a
seperate setting called cleanupIntervalS responsible for the cleanup,
which applies to all File Versioning (except External). Thus, remove the
unneeded code and don't set the param up on new folders anymore.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* gui: Allow to translate and fix incorrect Versions date filter ranges
Translate the previously English-only ranges used in Versions date
filter. In the process, fix the currently incorrect range calculation.
For instance, let us say it is 08:05. Selecting "today" should set the
range to start at 00:00 and end at 08:05. However, what really happens
is that both start and end are set to 08:05, and as a result "today" is
never shown to the user. The case is the same for "yesterday", which in
contrary to "today" is shown, but its range is fixed at 08:05 on the
previous day.
This commit fixes the above by always calculating "today" starting at
00:00 up to the current moment, and calculating "yesterday" from 00:00
on the previous to 00:00 on the current day.
When it comes to "last x days", the commit fixes the calculation so that
it actually covers the whole day, which is done by moving the start date
to 00:00 on the first day.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The ignoreDelete warning code originally comprised of two sentences,
which in total were much longer than the current one. Because of that,
some additional fixes were added to make it present itself better, e.g.
white-space was set to normal to allow for text wrap on narrow screens,
and the help link was moved to a new line to make sure both the anchor
and the question mark always stayed together.
However, the code fixes remained despite the final text being much
shorter than the original. Therefore, remove the unnecessary white-space
formatting, and also move the help link right next to the warning. This
makes it clear that the link is related to this particular warning when
both it and the ignore patterns warning are displayed at the same time.
Ref: https://github.com/syncthing/syncthing/pull/8054#issuecomment-978148117
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* gui: Translate fancytree messages in Versions modal
Currently, the default fancytree info/error messages are used. This
means that a) they are English-only, and b) they are very generic. With
this commit, the messages are added to the translatable strings, and
they are also more specifically related to file versioning.
On a side note, the "moreData" string has been left out on purpose, as
it is not used in the current code. It can be added later if needed.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* gui: Make ID read-only and hide nearby devices when adding pending devices (fixes#8083)
Currently, there is no distinction between adding "new" and "pending"
devices. For this reason, the user is always presented with a list of
"nearby devices" to choose from. This commit adds such distinction to
the code, and in the case of "pending" device, both the device ID is
made read-only and the nearby devices list is hidden.
As a by-product of the function rename and clean-up, this commit also
hides the non-functional "remove" button that is shown when editing
device defaults.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The intention was that if no peers are given, we shouldn't start the
listener. We did that anyway, because:
- splitting an empty string on comma returns a slice with one empty
string in it
- parsing the empty string as a device ID returns the empty device ID
so we end up with a valid replication peer which is the empty device ID.
* lib/protocol: Require at least 3.125% savings from compression
The new lz4 library doesn't need its output buffer to be the maximum
size, unlike the old one (which would allocate if it weren't). It can
take a buffer that is of a smaller size and will report if compressed
data can fit inside the buffer (with a small chance of reporting a false
negative). Use that property to our advantage by requiring compressed
data to be at most n-n/32 = .96875*n bytes long for n input bytes.
* lib/protocol: Remove unused receivers
To make DeepSource happy.
* lib/protocol: Micro-optimize lz4Compress
Only write the length if compression was successful. This is a memory
write, so the compiler can't reorder it.
Only check the return value of lz4.CompressBlock. Length-zero inputs
are always expanded by LZ4 compression (the library documents this),
so the check on len(src) isn't needed.
When GOBIN is set, 'go install' cannot install cross-compilied binaries.
To satisfy cross-compilation, it's necessary to add the '-o' to build
target, otherwise 'go build' will discarding the resulting objects when
compiling multiple packages.
Signed-off-by: bekcpear <i@bitbili.net>
* lib/model: Remove bogus fields from connections API endpoint.
Switch the returned data type for the /rest/system/connections element
"total" to use only the Statistics struct. The other fields of the
ConnectionInfo struct are not populated and misleading.
* Lowercase JSON field names.
* lib/model: Get rid of ConnectionInfo.MarshalJSON().
It was missing the StartedAt field from the embedded Statistics
struct. Just lowercasing the JSON attribute names can be done just as
easily with annotations.
* lib/model: Remove bogus startedAt field from totals.
Instead of using the Statistics type with one field empty, just switch
to a free-form map with the three needed fields.
* cmd/syncthing: Remove unnecessary function arguments.
The openGUI() function does not need a device ID to work, and there is
only one caller anyway which uses EmptyDeviceID.
The loadOrDefaultConfig() function is always called with the same
dummy values.
* cmd/syncthing: Avoid misleading info messages from monitor process.
In order to check whether panic reporting is enabled, the monitor
process utilizes the loadOrDefaultConfig() function. In case there is
no config file yet, info messages may be logged during creation if the
config Wrapper, which is discarded immediately after.
Stop using the DefaultConfig() utility function from lib/syncthing and
directly generate a minimal config instead to avoid these.
Add comments to loadOrDefaultConfig() explaining its limited purpose.
* cmd/syncthing/generate: Always write updated config file.
Previously, an existing config file was left untouched unless either
of the --gui-user or --gui-password options was given. Remove that
condition and simplify the checking code.
* lib/config: Factor out ProbeFreePorts().
* cmd/syncthing: Add option --skip-port-probing.
Applies to both the "generate" and "serve" subcommands, as well as the
deprecated --generate option, just as the --no-default-folder flag.
* cmd/syncthing: Remove unnecessary function arguments.
The openGUI() function does not need a device ID to work, and there is
only one caller anyway which uses EmptyDeviceID.
The loadOrDefaultConfig() function is always called with the same
dummy values.
* cmd/syncthing: Avoid misleading info messages from monitor process.
In order to check whether panic reporting is enabled, the monitor
process utilizes the loadOrDefaultConfig() function. In case there is
no config file yet, info messages may be logged during creation if the
config Wrapper, which is discarded immediately after.
Stop using the DefaultConfig() utility function from lib/syncthing and
directly generate a minimal config instead to avoid these.
Add comments to loadOrDefaultConfig() explaining its limited purpose.
* cmd/syncthing/generate: Always write updated config file.
Previously, an existing config file was left untouched unless either
of the --gui-user or --gui-password options was given. Remove that
condition and simplify the checking code.
Use indexOf instead of startsWith to make the now translatable theme
names appear correctly in IE11. This also prevents console log error
spam in the browser. The same problem was previously reported in #6940.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
When pathSep is a constant, the compiler precomputes pathSep+pathSep and
".."+pathSep instead of emitting function calls to compute "//" and
"../". Benchmark results in lib/osutil:
name old time/op new time/op delta
TraversesSymlink-8 8.86µs ± 3% 8.53µs ± 4% -3.79% (p=0.000 n=18+20)
name old alloc/op new alloc/op delta
TraversesSymlink-8 1.06kB ± 0% 1.06kB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
TraversesSymlink-8 15.0 ± 0% 15.0 ± 0% ~ (all equal)
The locking protocol in nat.Mapping was racy:
* Mapping.addressMap RLock'd, but then returned a map shared between
caller and Mapping, so the lock didn't do anything.
* Operations inside Service.{verifyExistingMappings,acquireNewMappings}
would lock the map for every update, but that means callers to
Mapping.ExternalAddresses can be looping over the map while the
Service methods are concurrently modifying it. When the Go runtime
detects that happening, it panics.
* Mapping.expires was read and updated without locking.
The Service methods now lock the map once and release the lock only when
done.
Also, subscribers no longer get the added and removed addresses, because
none of them were using the information. This was changed for a previous
attempt to retain the fine-grained locking and not reverted because it
simplifies the code.
Accept a subcommand as an alternative to the --generate option. It
accepts a custom config directory through either the --home or
--config options, using the default location if neither is given.
Add the options --gui-user and --gui-password to "generate", but not
the "serve --generate" option form. If either is given, an existing
config will not abort the command, but rather load, modify and save it
with the new credentials. The password can be read from standard
input by passing only a single dash as argument.
Config modification is skipped if the value matches what's already in
the config.
* cmd/syncthing: Utilize lib/locations package in generate().
Instead of manually joining paths with "magic" file names, get them
from the centralized locations helper lib.
* cmd/syncthing: Simplify logging for --generate option.
Visible change: No more timestamp prefixes.
What hash is used to store the password should ideally be an
implementation detail, so that every user of the GUIConfiguration
object automatically agrees on how to handle it. That is currently
distribututed over the confighandler.go and api_auth.go files, plus
tests.
Add the SetHasedPassword() / CompareHashedPassword() API to keep the
hashing method encapsulated. Add a separate test for it and adjust
other users and tests. Remove all deprecated imports of the bcrypt
package.
LoadOrGenerateCertificate() takes two file path arguments, but then
uses the locations package to determine the actual path. Fix that
with a minimally invasive change, by using the arguments instead.
Factor out GenerateCertificate().
The only caller of this function is cmd/syncthing, which passes the
same values, so this is technically a no-op.
* lib/tlsutil: Make storing generated certificate optional. Avoid
temporary cert and key files in tests, keep cert in memory.
Consistently use double dashes and fix typos -conf, -data-dir and
-verify.
Applies also to tests running the syncthing binary for consistency.
* Fix mismatched option name --conf in cli subcommand.
According to the source code comments, the cli option flags should
mirror those from the serve subcommand where applicable. That one is
actually called --config though.
* cli: Fix help text option placeholders.
The urfave/cli package uses the Value field of StringFlag to provide a
default value, not to name the placeholder. That is instead done with
backticks around some part of the Usage field.
* cli: Add missing --data flag in subcommand help text.
The urfave/cli based option parsing uses a fake flags collection to
generate help texts matching the used global options. But the --data
option was omitted from it, although it is definitely required when
using --config as well. Note that it cannot just be ignored, as some
debug stuff actually uses the DB:
syncthing cli --data=/bar --config=/foo debug index dump
By truncating time.Time to an int64 nanosecond count, we lose the
ability to precisely order timestamps before 1678 or after 2262, but we
gain (linux/amd64, Go 1.17.1):
name old time/op new time/op delta
JobQueuePushPopDone10k-8 2.85ms ± 5% 2.29ms ± 2% -19.80% (p=0.000 n=20+18)
JobQueueBump-8 34.0µs ± 1% 29.8µs ± 1% -12.35% (p=0.000 n=19+19)
name old alloc/op new alloc/op delta
JobQueuePushPopDone10k-8 2.56MB ± 0% 1.76MB ± 0% -31.31% (p=0.000 n=18+13)
name old allocs/op new allocs/op delta
JobQueuePushPopDone10k-8 23.0 ± 0% 23.0 ± 0% ~ (all equal)
Results for BenchmarkJobQueueBump are with the fixed version, which no
longer depends on b.N for the amount of work performed. rand.Rand.Intn
is cheap at ~10ns per iteration.
Like Windows and Mac, Android is also an interactive operating system.
On top of that, it usually runs on much slower hardware than the other
two. Because of that, it makes sense to limit the number of hashes used
by default there too.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Add each subdirectory of the guiDir as a translation candidate string.
The key is prefixed with "theme-name-" and the default English
translation corresponds to the directory name turned to title case.
Disable the automatic name mangling in the GUI JS code in favor of
just looking up the translation.
Registry.Get used a full sort to get the minimum of a list, and the sort
was broken because util.AddressUnspecifiedLess assumed it could find out
whether an address is IPv4 or IPv6 from its Network method. However,
net.(TCP|UDP)Addr.Network always returns "tcp"/"udp".
Currently, the default sorting of the file list in the Restore Versions
modal in both case sensitive, and it also does not distinguish between
files and folders. With this change, which uses a slightly modified code
from the fancytree itself (with added folder prioritisation), the sort
becomes case insensitive and also always places folders on top. This is
to make it behave more similar to what file managers do in the commonly
used operating systems (including popular Linux desktop environments).
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This reverts commit cca17f5306.
Using textarea instead of input makes it possible to enter new lines,
which then are added to config.xml as "
". This breaks the whole
script, because these characters are passed to the command line as "\n".
Therefore, the script should rather remain a single-lined input field.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The current detection is flawed, because it looks for a few specific
file systems like "msdos" or "fat" to set the mtime window, while in
reality Android seems to report names like "fuseblk", which can stand
for fat, ext4, or even f2fs.
At the moment, we set the mtime window only for a few known names used
for the fat filesystem. With this change, we take a safer approach of
always setting the time window unless we explicitly detect file systems
like ext2/ext3/ex4, which are known not to experience issues with moving
timestamps on Android.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The list of unshared devices is built from deviceList(), which sorts
the result. But the shared devices are listed in undefined order from
the unsorted currentFolder.devices array. Apply the same sorting
after building the array used in the dialog.
Currently, the dismiss button is displayed as the first of the three
buttons. However, the most common action that the user wants to do when
sharing a new folder is to add it on a different device. Because of
this, the add button should be displayed first as the most prominent of
the three. The ignore button is the opposite of the add button, and also
results in a parmenent action, hence it makes sense to lump the two
together. Thus, the dismiss button should be moved to the last place as
an alternative to the two main actions, when the user is yet unsure what
they want to do with the notification.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
1. Change each modal title text to match the action that is being
executed (i.e. "Revert" to "Revert Local Additions", "Override" to
"Override Changes", "Delete" to "Delete Unexpected Items").
2. Change the icons to match the icons used by each action (i.e. arrow-
circle-down for Revert, arrow-circle-up for Override). Replace the
broken lock icon for Delete with minus-circle.
3. Rearrange the order in the modal HTML code to simplify it a little.
Before this patch, IPv4-compatible addresses (::ffff:aaa.bbb.ccc.ddd)
may be used if a quic6://some.domain:port is specified and both IPv4 and
IPv6 addresses exist for that domain name.
This changes the GC mechanism so that the first pass (which reads all
FileInfos to populate bloom filters with block & version hashes) can
happen concurrently with normal database operations.
The big gcMut still exists, and we grab it temporarily to block all
other modifications while we set up the bloom filters. We then release
the lock and let other things happen, with those other things also
updating the bloom filters as required. Once the first phase is done we
again grab the gcMut, knowing that we are the sole modifier of the
database, and do the cleanup.
I also removed the final compaction step.
* cmd/strelaypoolsrv: Fix minor grammar, use https in links
Add a few minor grammatical/stylistic fixes. Use `https` instead of
`http` in the MaxMind link in the footer.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* wip
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Disable the Versions button when the folder is paused, because it does
not work, i.e. the versioned files are not loaded. The folder needs to
be unpaused to actually be able to view the versioned file list.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
* Trigger connection loop on config device addition (fixes#7600)
* Also check for device address equality
* Move EqualStrings from api_test to utils, and use in connections/service.go
* Make sure CommitConfiguration cannot block due on the deviceAddressesChanged channel
* Update lib/connections/service.go
Co-authored-by: Jakob Borg <jakob@kastelo.net>
* cmd/syncthing: Don't fail early on api setup error (fixes 7558)
* switch to factory pattern
* refactor config command to show help on nothing
* wip
* wip
* already abort in before
This makes us use TLS 1.3+ on sync connections by default. A new option
`insecureAllowOldTLSVersions` exists to allow communication with TLS
1.2-only clients (roughly Syncthing 1.2.2 and older). Even with that
option set you get a slightly simplified setup, with the cipher suite
order fixed instead of auto detected.
* cmd/syncthing: Don't fail early on api setup error (fixes 7558)
* switch to factory pattern
* refactor config command to show help on nothing
* wip
* wip
* already abort in before
No longer hide the web UI controls for the new untrusted/encrypted
device feature. Testing hasn't been very widespread, but there has been
some and quite a few bugs have been caught and fixed. I believe its time
to not hide it anymore, and cautiously recommend usage. E.g. mention
that the feature hasn't been widely used yet and anyone using it is an
early adopter, but drop the bit about not using it with production data.
We can maybe stress the need for backups in general and especially
using this.
Remove the animation due to its excessive CPU usage, especially when a
large number of files is being downloaded and listed at the same time.
Also, remove the stripes, as they serve no purpose in the now-static
progress bar.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
There was a logic mistake, so the limit in question wasn't used. On my
macOS this doesn't seem to matter, the hard limit returned is 2^63-1 and
setting the soft limit to that works. However I'm assuming that's not
the case for older macOSes since it was so nicely documented, so we
should still have this working. (10240 FDs should be enough for
anybody.)
This is a mostly pointless change to make security scanners and static
analysis tools happy, as they all hate seeing md5. None of our md5 uses
were security relevant, but still. Only visible effect of this change is
that our temp file names for very long file names become slightly longer
than they were previously...
The current text gives an impression that we are currently using a
Receive Encrypted folder, even if we are not. Thus, make the current
text displayed only when the folder is in fact Receive Encrypted, and
add a new string to be displayed when using different folder types.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds a couple of dummy asset files protected by the "noassets"
build tag. The purpose is that it should be possible for, for example,
CI tools and static analysis things to compile and analyze the source
tree without our custom asset generation step. Also makes `go test -tags
noassets ./...` work without building assets first.
Disabled options are currently barely distinguishable from enabled
ones. This changes their background to grey, following the Bootstrap
defaults already used for disabled <select>.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This truncates times meant for API consumption to second precision,
where fractions won't typically matter or add any value. Exception to
this is timestamps on logs and events, and of course I'm not touching
things like file metadata.
I'm not 100% certain this is an exhaustive change, but it's the things I
found by grepping and following the breadcrumbs from lib/api...
I also considered general-but-ugly solutions, like having the API
serializer itself do reflection magic or even regexps on returned
objects, but decided against it because aurgh...
This adds a button in the top right that changes the config back to the
default theme.
Code wise, it takes the header that was previously a part of the
dashboard component and moves it to the app component, and then adds the
button there. Possibly the header should be a component of it's own, but
that's more of refactor that can happen separately I think.
The config change uses the new config API to just patch the relevant
setting.
I'm not doing an automatic reload because 1) I don't want to figure out
how to do it correctly and 2) this doesn't work reliably anyway, as
for example the current gen GUI does a reload and ends up with
connection refused as the API service is still reloading...
* Provide a sysctl config to raise max UDP buffer size
* Add sysctl config to deb
* Check if `deb-systemd-invoke` is available
Co-authored-by: otbutz <tbutz@optitool.de>
This loosens the ‘is this localhost?’ check to include *.localhost host
names.
This allows for clearer (hence better) names to be used in browsers,
e.g. when accessing a remote syncthing instance ‘foo’ using a ssh port
forward, one can use foo.localhost to remind oneself which one is which.
💡 Without these changes, Syncthing shows a ‘Host check error’ when
pointing a browser at http://foo.localhost/, and with these changes, the
interface loads as usual.
The .localhost top level domain is a reserved top-level domain (RFC 2606):
> The ".localhost" TLD has traditionally been statically defined in
> host DNS implementations as having an A record pointing to the
> loop back IP address and is reserved for such use. Any other use
> would conflict with widely deployed code which assumes this use.
> – https://tools.ietf.org/html/rfc2606
As Wikipedia puts it:
> This allows the use of these names for either documentation purposes
or in local testing scenarios. – https://en.wikipedia.org/wiki/.localhost
On Linux systems, systemd-resolved resolves *.localhost, on purpose:
https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html
See also #4815, #4816.
* release:
all: Fix Microsoft documentation links in code comments (#7387)
gui: Handle info labels that are longer than available space (fixes#944) (#7386)
gui: Show "Last seen" at the top when device is disconnected (ref #7166) (#7373)
gui: Remove cleanupIntervalS leftovers from External Versioning (ref #7360) (#7378)
gui: Allow setting custom path for all versioning except external (#7377)
lib/fs: Consider options in case-fs caching (fixes#7371) (#7381)
lib/scanner: Pass on errors while hashing (#7380)
build: Update pfilter (#7376)
gui: Hide the Rescan All button when no folders exist (#7367)
lib/connections: Allow QUIC with Go 1.16 (#7372)
gui: Hide non-functional Versions button when using External Versioning (#7365)
gui, man, authors: Update docs, translations, and contributors
gui: Fix setting external versioning command (fixes#7361) (#7362)
lib/versioner: Improve error messages (fixes#7354) (#7357)
gui: Disable Cleanup Interval with External File Versioning as unsupported (#7360)
Apply to table headers the same code as already used for table data.
This way, the headers will be either pushed to the next line, or cut
with an ellipsis if the single word is too long.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Move the "Last seen" field to the very top in the device information.
This way, if a device has disconnected unexpectly, we can quickly check
the time when it was last available. Right now, due to the very long
address field, it is usually necessary to scroll down in order to view
the "Last seen" field.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
If there are no folders present, show only the "Add Folder" button, and
hide the "Rescan All" button. Only show the latter when at least one
folder exists.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Apply to table headers the same code as already used for table data.
This way, the headers will be either pushed to the next line, or cut
with an ellipsis if the single word is too long.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Move the "Last seen" field to the very top in the device information.
This way, if a device has disconnected unexpectly, we can quickly check
the time when it was last available. Right now, due to the very long
address field, it is usually necessary to scroll down in order to view
the "Last seen" field.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
If there are no folders present, show only the "Add Folder" button, and
hide the "Rescan All" button. Only show the latter when at least one
folder exists.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The button does nothing when the External Versioning is being used, so
it should not be displayed at all to avoid confusing the users.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
As for right now, the External File Versioning does not support Cleanup
Interval. Therefore, the option should no be available at all when using
it.
Ref: https://forum.syncthing.net/t/16346
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
The button does nothing when the External Versioning is being used, so
it should not be displayed at all to avoid confusing the users.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
As for right now, the External File Versioning does not support Cleanup
Interval. Therefore, the option should no be available at all when using
it.
Ref: https://forum.syncthing.net/t/16346
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
An untrusted device will receive padded info for small blocks, and hence
sometimes request a larger block than actually exists on disk.
Previously we let this pass because we didn't have a hash to compare to
in that case and we ignored the EOF error based on that.
Now the untrusted device does pass an encrypted hash that we decrypt and
verify. This means we can't check for len(hash)==0 any more, but on the
other hand we do have a valid hash we can apply to the data we actually
read. If it matches then we don't need to worry about the read
supposedly being a bit short.
Benchmark results on Linux/amd64, using updated benchmark for old and
new:
name old time/op new time/op delta
HashFile-8 88.6ms ± 1% 88.3ms ± 1% -0.33% (p=0.046 n=19+19)
name old speed new speed delta
HashFile-8 201MB/s ± 1% 202MB/s ± 1% +0.33% (p=0.044 n=19+19)
name old alloc/op new alloc/op delta
HashFile-8 59.4kB ± 0% 46.1kB ± 0% -22.47% (p=0.000 n=14+20)
name old allocs/op new allocs/op delta
HashFile-8 29.0 ± 0% 27.0 ± 0% -6.90% (p=0.000 n=20+20)
Co-authored-by: greatroar <@>
* lib/db: Add ExpirePendingFolders().
Use-case is to drop any no-longer-pending folders for a specific
device when parsing its ClusterConfig message where previously offered
folders are not mentioned any more.
The timestamp in ObservedFolder is stored with only second precision,
so round to seconds here as well. This allows calling the function
within the same second of adding or updating entries.
* lib/model: Weed out pending folders when receiving ClusterConfig.
Filter the entries by timestamp, which must be newer than or equal to
the reception time of the ClusterConfig. For just mentioned ones,
this assumption will hold as AddOrUpdatePendingFolder() updates the
timestamp.
* lib/model, gui: Notify when one or more pending folders expired.
Introduce new event type FolderOfferCancelled and use it to trigger a
complete refreshCluster() cycle. Listing individual entries would be
much more code and probably just as much work to answer the API
request.
* lib/model: Add comment and rename ExpirePendingFolders().
* lib/events: Rename FolderOfferCancelled to ClusterPendingChanged.
* lib/model: Reuse ClusterPendingChanged event for cleanPending()
Changing the config does not necessarily mean that the
/resut/cluster/pending endpoints need to be refreshed, but only if
something was actually removed. Detect this and indicate it through
the ClusterPendingChanged event, which is already hooked up to requery
respective endpoints within the GUI.
No more need for a separate refreshCluster() in reaction to
ConfigSaved event or calling refreshConfig().
* lib/model: Gofmt.
* lib/db: Warn instead of info log for failed removal.
* gui: Fix pending notifications not loading on GUI start.
* lib/db: Use short device ID in log message.
* lib/db: Return list of expired folder IDs after deleting them.
* lib/model: Refactor Pending...Changed events.
* lib/model: Adjust format of removed pending folders enumeration.
Use an array of objects with device / folder ID properties, matching
the other places where it's used.
* lib/db: Drop invalid entries in RemovePendingFoldersBeforeTime().
* lib/model: Gofmt.
My local gofmt did not complain here, strangely...
* gui: Handle PendingDevicesChanged event.
Even though it currently only holds one device at a time, wrap the
contents in an array under the "added" property name.
* lib/model: Fix null values in PendingFoldersChanged removed member.
* gui: Handle PendingFoldersChanged event.
* lib/model: Simplify construction of expiredPendingList.
* lib/model: Reduce code duplication in cleanPending().
Use goto and a label for the common parts of calling the DB removal
function and building the event data part.
* lib/events, gui: Mark ...Rejected events deprecated.
Extend comments explaining the conditions when the replacement event
types are emitted.
* lib/model: Wrap removed devices in array of objects as well.
* lib/db: Use iter.Value() instead of needless db.Get(iter.Key())
* lib/db: Add comment explaining RemovePendingFoldersBeforeTime().
* lib/model: Rename fields folderID and deviceID in event data.
* lib/db: Only list actually expired IDs as removed.
Skip entries where Delete() failed as well as invalid entries that got
removed automatically.
* lib/model: Gofmt
This splits the ignore getting to two methods, one that loads from disk
(the old one) and one that just returns whatever is already loaded (the
new one). The folder summary service which is just interested in stats
now uses the latter method. This means that it, and API calls that call
it, does not get blocked by folder I/O.
This adds two new configuration options:
// The number of connections at which we stop trying to connect to more
// devices, zero meaning no limit. Does not affect incoming connections.
ConnectionLimitEnough int
// The maximum number of connections which we will allow in total, zero
// meaning no limit. Affects incoming connections and prevents
// attempting outgoing connections.
ConnectionLimitMax int
These can be used to limit the number of concurrent connections in
various ways.
This adds a statistic to track the last connection duration per device.
It isn't used for much in this PR, but it's available for #7223 to use
in deciding how to order device connection attempts (deprioritizing
devices that just dropped our connection the last time).
The test would fail if the umask on UNIX is greater than 0022, because
the OS transparently subtracts it from the mode passed to Mkdir(), as
the Go documentation confirms.
Our goal here is not to test os.Mkdir(), so just make sure the desired
mode is actually set by forcing it afterwards.
This breaks out some methods from the connection loop to make it simpler
to manage and understand.
Some slight simplifications to remove the `seen` variable (we can filter
`nextDial` based on times are in the future or not, so we don't need to
track `seen`) and adding a minimum loop interval (5s) in case some
dialer goes haywire and requests a 0s redial interval or such.
Otherwise no significant behavioral changes.
This does two things:
- Exclude QUIC from go1.16 builds, automatically, for now, since it
doesn't work and just panics.
- Provide some fake listeners and dialers when QUIC is disabled.
These fake listeners and dialers indicate that they are disabled and
unsupported, which silences "Dialing $address: unknown address scheme:
quic" type of stuff which is not super helpful to the user.
This fixes a regression by restoring two functions removed in commit
a20d85d451, but still used from the HTML
template. Adjust the restored versions to access the new flag storage
structure.
The replacement functions selectAllSharedFolders() and
selectAllUnrelatedFolders() are not functional, so this is just a
quick fix to restore a working state before making the new functions
actually work sensibly.
Just like the respective functions for sharing device selections, give
it a boolean argument instead of writing two almost identical
functions.
In line with the other selectAll...() functions, avoid calling
angular.forEach() and iterate over the folders object directly.
This fixes a regression by restoring two functions removed in commit
a20d85d451, but still used from the HTML
template. Adjust the restored versions to access the new flag storage
structure.
The replacement functions selectAllSharedFolders() and
selectAllUnrelatedFolders() are not functional, so this is just a
quick fix to restore a working state before making the new functions
actually work sensibly.
Just like the respective functions for sharing device selections, give
it a boolean argument instead of writing two almost identical
functions.
In line with the other selectAll...() functions, avoid calling
angular.forEach() and iterate over the folders object directly.
When using a Web browser with JavaScript either disabled or unavailable,
show a warning to let the user know that the Web GUI requires JS in
order to operate.
To achieve this, add a <div> that wraps both the navbar and the main
content, and then move the CSS class ng-cloak from the <html> element to
that <div>. This way, only the JavaScript-dependent part is hidden when
JS is unavailable, and not the whole website, as it is the case right
now. Then, add a <noscript> element right at the start of the <body>
element, so that the warning is also shown right away in text-based Web
browsers. The <noscript> element includes a stripped down version of the
navbar showing only the Syncthing logo, and then a container with the
warning itself. Lastly, leave the footer untouched and always visible,
because it does not rely on JavaScript at all.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This removes the switch for using a Badger database, because it has bugs
that it seems there is no interest in fixing, and no actual bug tracker
to track them in.
It retains the actual implementation for the sole purpose of being able
to do the conversion back to LevelDB if anyone is actually running with
USE_BADGER. At some point in a couple of versions we can remove the
implementation as well.
Add specific errors for the failures, resulting in this rather than just
the generic "invalid filename":
[MRIW7] 08:50:50 INFO: Puller (folder default, item "NUL"): syncing: filename is invalid: name is reserved
[MRIW7] 08:50:50 INFO: Puller (folder default, item "fail."): syncing: filename is invalid: name ends with space or period
[MRIW7] 08:50:50 INFO: Puller (folder default, item "sup:yo"): syncing: filename is invalid: name contains reserved character
[MRIW7] 08:50:50 INFO: default: Failed to sync 3 items
Use the IoctlFileClone and IoctlFileCloneRange ioctl wrappers and the
FileCloneRange type provided by golang.org/x/sys/unix instead of
locally implementing them. This also allows to re-enable the code for
ppc/ppc64/ppc64le again (see commit 758a1a6a37 ("lib/fs: Disable ioctl
on ppc (fixes#6898) (#6901)")) since golang.org/x/sys/unix internally
uses the correct FICLONE and FICLONERANGE values depending on $GOARCH.
Most notably, it now detects all-lowercase files and returns these
as-is. The tests have been expanded with two cases and are now used
as a benchmark (admittedly a rather trivial one).
name old time/op new time/op delta
UnicodeLowercaseMaybeChange-8 4.59µs ± 2% 4.57µs ± 1% ~ (p=0.197 n=10+10)
UnicodeLowercaseNoChange-8 3.26µs ± 1% 3.09µs ± 1% -5.27% (p=0.000 n=9+10)
This changes the cache to cache less things, yet retain the required
efficiency for our walk usecase. This uses less memory.
Specifically, instead of keeping result and child caches for each path
level, only keep a single cached child. In practice our operations are
depth-first, or almost depth-first, and then we retain the same hit
ratio for a smaller cache size.
I improved the benchmark so that it counts the Lstat and DirNames
operations performed, and they do not change significantly. The amount
of allocated memory is reduced by 20% and the walk itself is actually
slightly faster.
This also removes the clear based on number of cached names (as that is
not a thing any more) and the timer based clear (which was unused). This
means we'll retain the last cache state forever until it's cleared by a
write operation, but we did that before too and that state is now a lot
smaller...
The overhead compared to not using a casefs, for our typical "double
walk" (walk the tree then stat everything again) is 2x the dirnames we
would otherwise call, and no overhead on the stats (unchanged from old
implementation)
```
name old time/op new time/op delta
WalkCaseFakeFS100k/rawfs-8 306ms ± 1% 305ms ± 2% ~ (p=0.182 n=9+10)
WalkCaseFakeFS100k/casefs-8 579ms ± 5% 557ms ± 1% -3.77% (p=0.000 n=10+10)
name old B/entry new B/entry delta
WalkCaseFakeFS100k/rawfs-8 590 ± 0% 590 ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 1.09k ± 0% 0.87k ± 0% -19.98% (p=0.000 n=10+10)
name old DirNames/entry new DirNames/entry delta
WalkCaseFakeFS100k/rawfs-8 0.51 ± 0% 0.51 ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 1.02 ± 0% 1.02 ± 0% ~ (all equal)
name old DirNames/op new DirNames/op delta
WalkCaseFakeFS100k/rawfs-8 51.2k ± 0% 51.2k ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 102k ± 0% 102k ± 0% ~ (all equal)
name old Lstat/entry new Lstat/entry delta
WalkCaseFakeFS100k/rawfs-8 3.02 ± 0% 3.02 ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 3.02 ± 0% 3.02 ± 0% ~ (all equal)
name old Lstat/op new Lstat/op delta
WalkCaseFakeFS100k/rawfs-8 302k ± 0% 302k ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 302k ± 0% 302k ± 0% ~ (all equal)
name old allocs/entry new allocs/entry delta
WalkCaseFakeFS100k/rawfs-8 15.7 ± 0% 15.7 ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 27.5 ± 0% 26.1 ± 0% -5.09% (p=0.000 n=10+10)
name old ns/entry new ns/entry delta
WalkCaseFakeFS100k/rawfs-8 2.02k ± 1% 2.02k ± 2% ~ (p=0.163 n=9+10)
WalkCaseFakeFS100k/casefs-8 3.83k ± 5% 3.68k ± 1% -3.77% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
WalkCaseFakeFS100k/rawfs-8 89.2MB ± 0% 89.2MB ± 0% ~ (p=0.364 n=9+10)
WalkCaseFakeFS100k/casefs-8 164MB ± 0% 131MB ± 0% -19.97% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
WalkCaseFakeFS100k/rawfs-8 2.38M ± 0% 2.38M ± 0% ~ (all equal)
WalkCaseFakeFS100k/casefs-8 4.16M ± 0% 3.95M ± 0% -5.05% (p=0.000 n=10+10)
```
Since iterators must be released before committing or discarding a
transaction we have the pattern of both deferring a release plus doing
it manually. But we can't release twice because we track this with a
WaitGroup that will panic when there are more Done()s than Add()s. This
just adds a boolean to let an iterator keep track.
Use indexOf() == 0 instead of startsWith() to maintain compatibility
and prevent JavaScript error console spam in the Web GUI when used in
Internet Explorer 11 under Windows 7.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
We created a new fileset before stopping the folder during restart. When
we create that fileset it loads the current metadata and sequence
numbers from the database. But the folder may have time to update those
before stopping, leaving the new fileset with bad data.
This would cause wrong accounting (forgotten files) and potentially
sequence reuse causing files not sent to other devices.
This change reuses the fileset on restart, skipping the issue entirely.
It also moves the creation of the fileset back under the lock so there
should be no chance of concurrency issues here.
We created a new fileset before stopping the folder during restart. When
we create that fileset it loads the current metadata and sequence
numbers from the database. But the folder may have time to update those
before stopping, leaving the new fileset with bad data.
This would cause wrong accounting (forgotten files) and potentially
sequence reuse causing files not sent to other devices.
This change reuses the fileset on restart, skipping the issue entirely.
It also moves the creation of the fileset back under the lock so there
should be no chance of concurrency issues here.
The FileSet.Drop operation in there needs to potentially update a whole lot of global lists, which can take a while (longer than the deadlock interval apparently)
When cap(permanentRelays) >= len(permanentRelays) + len(knownRelays),
append(permanentRelays, knownRelays...)
returns a slice of the array underlying permanentRelays. The subsequent
rand.Shuffle then mixes the permanent and known relays. Sequential
requests may cause strelaypoolsrv to forget its permanent relays. Worse,
concurrent requests may cause shuffling of the same slice on multiple
processors concurrently.
Co-authored-by: greatroar <@>
* Add clean up for Simple File Versioning pt.1
created test
* Add clean up for Simple File Versioning pt.2
Passing the test
* stuck on how javascript communicates with backend
* Add trash clean up for Simple File Versioning
Add trash clean up functionality of to allow the user to delete backups
after specified amount of days.
* Fixed html and js style
* Refactored cleanup test cases
Refactored cleanup test cases to one file and deleted duplicated code.
* Added copyright to test file
* Refactor folder cleanout to utility function
* change utility function to package private
* refactored utility function; fixed build errors
* Updated copyright year.
* refactor test and logging
* refactor html and js
* revert style change in html
* reverted changes in html and some js
* checkout origin head version edit...html
* checkout upstream master and correct file
Quoting the manual:
-trimpath
remove all file system paths from the resulting executable.
Instead of absolute file system paths, the recorded file names
will begin with either "go" (for the standard library),
or a module path@version (when using modules),
or a plain import path (when using GOPATH).
That is, when we panic, instead of:
goroutine 1 [running]:
main.main()
/Users/jb/dev/syncthing/syncthing/cmd/syncthing/main.go:272 +0x116
we get:
goroutine 1 [running]:
main.main()
github.com/syncthing/syncthing@/cmd/syncthing/main.go:272 +0x116
(Module path and file path within module.)
Our authentication is based on device ID (certificate fingerprint) but
we also check the certificate name for ... historical extra security
reasons. (I don't think this adds anything but it is what it is.) Since
that check breaks in Go 1.15 this change does two things:
- Adds a manual check for the peer certificate CommonName, and if they
are equal we are happy and don't call the more advanced
VerifyHostname() function. This allows our old style certificates to
still pass the check.
- Adds the cert name "syncthing" as a DNS SAN when generating the
certificate. This is the correct way nowadays and makes VerifyHostname()
happy in Go 1.15 as well, even without the above patch.
Apparently our Tags field depended on having specific files react to
tags and add themselves there. This, instead, works for all tags.
Also, pass tags to the test command line.
The QUIC package is notorious for being incompatible with either too
old or too new Go releases. Currently it doesn't build with Go 1.15 RC
and I want to test the rest with Go 1.15. With this I can do `go run
build.go --tags noquic` to do that.
With this change we emulate a case sensitive filesystem on top of
insensitive filesystems. This means we correctly pick up case-only renames
and throw a case conflict error when there would be multiple files differing
only in case.
This safety check has a small performance hit (about 20% more filesystem
operations when scanning for changes). The new advanced folder option
`caseSensitiveFS` can be used to disable the safety checks, retaining the
previous behavior on systems known to be fully case sensitive.
Co-authored-by: Jakob Borg <jakob@kastelo.net>
Prompted by https://forum.syncthing.net/t/infinite-filesystem-recursion-detected/15285. In my opinion the filesystem shouldn't throw warnings but pass on errors for the caller to decide what's to be happening with it. Right now in this PR an infinite recursion is a normal scan error, i.e. folder is in failed state and displays failed items, but no warning. I think that's appropriate but if deemed appropriate an additional warning can be thrown in the scanner.
This fixes the change in #6674 where the weak hash became a deciding
factor. Now we again just use it to accept a block, but don't take a
negative as meaning the block is bad.
This fixes the change in #6674 where the weak hash became a deciding
factor. Now we again just use it to accept a block, but don't take a
negative as meaning the block is bad.
If the GC finds a key k that it wants to keep, it records that in a
Bloom filter. If a key k' can be removed but its hash collides with k,
it will be kept. Since the old Bloom filter code was completely
deterministic, the next run would encounter the same collision, assuming
k must still be kept.
A randomized hash function that uses all the SHA-256 bits solves this
problem: the second run has a non-zero probability of removing k', as
long as the Bloom filter is not completely full.
* Fix ui, hide report date
* Undo Goland madness
* UR now web scale
* Fix migration
* Fix marshaling, force tick on start
* Fix tests
* Darwin build
* Split "all" build target, add package name as a tag
* Remove pq and sql dep from syncthing, split build targets
* Empty line
* Revert "Empty line"
This reverts commit f74af2b067.
* Revert "Remove pq and sql dep from syncthing, split build targets"
This reverts commit 8fc295ad00.
* Revert "Split "all" build target, add package name as a tag"
This reverts commit f4dc889951.
* Normalise contract types
* Fix build add more logging
This is shorter, skips two allocations, makes the function inlineable
and is safer, since the compiler now check whether
DeviceIDLength == sha256.Size.
Currently a random dev version has a version string like this:
v1.7.0-rc.1+23-gef3441bd6
That is, the tag name followed by a plus sign and the git describe
metadata (number of commits and hash) plus -dirty or -branchname in some
cases.
We introduced the plus sign in #473, where a dev version would
previously be called v0.9.0-42-gwhatever which is considered older than
v0.9.0 and hence caused a downgrade. The problem with the plus is that
per semver everything after the plus is ignored as build metadata, which
means we won't upgrade from v1.7.0-rc.1 to v1.7.0-rc.1+22-g946170f3f.
With this change the we instead either just add a dev suffix (if we're
already on a prerelease version) or we wind the patch version and add a
dev suffix.
v1.7.0-rc.1+23-gef3441bd6 => v1.7.0-rc.1.dev.23.gef3441bd6
v1.6.1+80-gef3441bd6 => v1.6.2-dev.80.gef3441bd6
This should preserve the ordering and keep versions semver-ish.
This changes the error handling in loading ignores slightly:
- There is a new ParseError type that is returned as the error
(somewhere in the chain) when the problem was not an I/O error loading
the file, but some issue with the contents.
- If the file was read successfully but not parsed successfully we still
return the lines read (in addition to nil patterns and a ParseError).
- In the API, if the error IsParseError then we return a successful
HTTP response with the lines and the actual error included in the JSON
object.
- In the GUI, as long as the HTTP call to load the ignores was
successful we can edit the ignores. If there was an error we show this
as a validation error on the dialog.
Also some cleanup on the Javascript side as it for some reason used
jQuery instead of Angular for this editor...
crypto/rand output is cryptographically secure by the Go library
documentation's promise. That, rather than strength (= passes randomness
tests) is the property that Syncthing needs).
This makes Go 1.15 test/vet happy, avoiding "conversion from untyped int
to string yields a string of one rune" warning where we do
string(KeyTypeWhatever) in namespaced.go.
It also clarifies and enforces the currently allowed range of these
numbers so I think it's fine.
This matches the convention of the stdlib and avoids ambiguity: when
customErr{} and &customErr{} both implement error, client code needs to
check for both.
Memory use should remain the same, since storing a non-pointer type in
an interface value still copies the value to the heap.
This extracts the extra tags from any `[foo]` stuff at the end of the
version and sends them to Sentry for indexing.
If I need to modify that regexp again I'll probably write a from scratch
tokenizer and parser for our version string instead...
This adds some env vars to the long version string as if they were build
tags. The purpose is to better understand what code was running or not
in the version output, usage reporting and crash reports. In order to
prevent possible privacy issues the actual value of the variable is not
reported, just the fact that it was set to something non-empty.
Example:
% ./bin/syncthing --version
syncthing v1.6.1+47-g1eb104f3-buildtags "Fermium Flea" (go1.14.3 darwin-amd64) jb@kvin.kastelo.net 2020-06-03 07:25:46 UTC [stnoupgrade, use_badger]
Group the global list of files by version, instead of having one flat list for all devices. This removes lots of duplicate protocol.Vectors.
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This reduces the size of our write batches before we flush them. This
has two effects: reducing the amount of data lost if we crash when
updating the database, and reducing the amount of memory used when we do
large updates without checkpoint (e.g., deleting a folder).
I ran our SyncManyFiles benchmark as it is the one doing most
transactions, however there was no relevant change in any metric (it's
limited by our fsync I expect). This is good as any visible change would
just be a decrease in performance.
I don't have a benchmark on deleting a large folder, taking that part on
trust for now...
* cmd/stindex: Unify access to key from cached variable.
Avoid calling the Key() method from the iterator each time the value
is needed. Just reuse the cache variable already assigned before the
switch block.
* cmd/stindex: Display the prefix byte value for unknown key types.
Make it easier to diagnose corrupt / unknown key type entries by
showing their decimal value, correlating with the definitions in
keyer.go.
* cmd/stindex: Add missing KeyType values in stindex dump code.
Recently added DB key prefixes KeyTypeBlockListMap and KeyTypeVersion
were unknown to the stindex dumping tool. Add basic parsing to dump
their key structure.
If we fail to take the rename shortcut we may crash on a later loop,
because we do trickiness with the indexes but the original buckets[key]
in "range buckets[key]" isn't re-evaluated so i exceeds the max index.
This adds indirection of large version vectors in the same manner as we
already to block lists. The effect is the same: less duplicated data in
some situations.
To mitigate the impact for when this indirection
wouldn't be needed I've added an indirection cutoff for both blocks and
the new version vector stuff: we don't do the indirection at all for
small block lists or small version vectors, instead storing it directly
like we used to do. This is faster for small files and small setups.
Storing assets as []byte requires every compiled-in asset to be copied
into writable memory at program startup. That currently takes up 1.6MB
per syncthing process. Strings stay in the RODATA section and should be
shared between processes running the same binary.
This makes version vector values clock based instead of just incremented
from zero. The effect is that a vector that is created from scratch
(after database reset) will have a higher value for the local device
than what it could have been previously, causing a conflict. That is, if
we are A and we had
{A: 42, B: 12}
in the old scheme, a reset and rescan would give us
{A: 1}
which is a strict ancestor of the older file (this might be wrong). With
the new scheme we would instead have
{A: someClockTime, b: otherClockTime}
and the new version after reset would become
{A: someClockTime+delta}
which is in conflict with the previous entry (better).
In case the clocks are wrong (current time is less than the value in the
vector) we fall back to just simple increment like today.
This scheme is ineffective if we suffer a database reset while at the
same time setting the clock back far into the past. It's however no
worse than what we already do.
This loses the ability to emit the "added" event, as we can't look for
the magic 1 entry any more. That event was however already broken
(#5541).
Another place where we infer meaning from the vector itself is in
receive only folders, but there the only criteria is that the vector is
one item long and includes just ourselves, which remains the case with
this change.
* wip
This changes the build script to build all the things in one go
invocation, instead of one invocation per cmd. This is a lot faster
because it means more things get compiled concurrently. It's especially
a lot faster when things *don't* need to be rebuilt, possibly because it
only needs to build the dependency map and such once instead of once per
binary.
In order for this to work we need to be able to pass the same ldflags to
all the binaries. This means we can't set the program name with an
ldflag.
When it needs to rebuild everything (go clean -cache):
( ./old-build -gocmd go1.14.2 build all 2> /dev/null; ) 65.82s user 11.28s system 574% cpu 13.409 total
( ./new-build -gocmd go1.14.2 build all 2> /dev/null; ) 63.26s user 7.12s system 1220% cpu 5.766 total
On a subsequent run (nothing to build, just link the binaries):
( ./old-build -gocmd go1.14.2 build all 2> /dev/null; ) 26.58s user 7.53s system 582% cpu 5.853 total
( ./new-build -gocmd go1.14.2 build all 2> /dev/null; ) 18.66s user 2.45s system 1090% cpu 1.935 total
This makes the GC runner a service that will stop fairly quickly when
told to.
As a bonus, STTRACE=app will print the service tree on the way out,
including any errors they've flagged.
* release:
lib/db: Don't get blocklists on drop and missing continue (ref #6457) (#6502)
Revert "cmd/syncthing: Do auto-upgrade before startup (fixes#6384) (#6385)"
lib/ur: Correct freaky error handling (fixes#6499) (#6500)
So, in a funny plot twist, it turns out that WriteFile in Go 1.13
doesn't actually set the read only bit on Windows when called with
permissions 0444 so my test was broken. With an improved test it turns
out that Rename does not, in fact, overwrite a read-only file on
Windows. This adds a fix for that.
(Rename might get improved in Go 1.15...)
Successful LRU cache lookups modify the cache's recency list, so
RWMutex.RLock isn't enough protection.
Secondarily, multiple concurrent lookups with the same key should not
create separate rate limiters, so release the lock only when presence
of the key in the cache has been ascertained.
Co-authored-by: greatroar <@>
This adds the functionality to run a user search with a filter for LDAP
authentication. The search is done after successful bind, as the binding
user. The typical use case is to limit authentication to users who are
member of a group or under a certain OU. For example, to only match
users in the "Syncthing" group in otherwise default Active Directory
set up for example.com:
<searchBaseDN>CN=Users,DC=example,DC=com</searchBaseDN>
<searchFilter>(&(sAMAccountName=%s)(memberOf=CN=Syncthing,CN=Users,DC=example,DC=com))</searchFilter>
The search filter is an "and" of two criteria (with the ampersand being
XML quoted),
- "(sAMAccountName=%s)" matches the user logging in
- "(memberOf=CN=Syncthing,CN=Users,DC=example,DC=com)" matches members
of the group in question.
Authentication will only proceed if the search filter matches precisely
one user.
The previous implementation was very generic; its tests didn't cover the
actual alphabet for device IDs.
Benchmark results on amd64:
name old time/op new time/op delta
Luhnify-8 1.00µs ± 1% 0.28µs ± 4% -72.38% (p=0.000 n=9+10)
Unluhnify-8 992ns ± 2% 274ns ± 1% -72.39% (p=0.000 n=10+9)
We set the STRESTART environment when starting the inner process after
the first time, but this didn't persist when restarting the monitor
process. Now it does.
As of the latest database checker we are again putting files without
blocks. I'm not 100% convinced that's a great idea, but we also do it
for ignored files apparently so it looks like we probably should support
it. This adds an escape hatch that must be manually enabled...
As of the latest database checker we are again putting files without
blocks. I'm not 100% convinced that's a great idea, but we also do it
for ignored files apparently so it looks like we probably should support
it. This adds an escape hatch that must be manually enabled...
* release:
lib/db, lib/syncthing: Repair db once on upgrade (ref #6425, #6427) (#6429)
lib/db: Fix removeFromGlobal and no filenames in error (fixes#6427) (#6428)
lib/db: Remove emptied global list in checkGlobals (fixes#6425) (#6426)
- In the few places where we wrap errors, use the new Go 1.13 "%w"
construction instead of %s or %v.
- Where we create errors with constant strings, consistently use
errors.New and not fmt.Errorf.
- Remove capitalization from errors in the few places where we had that.
If we decide to recalculate the metadata we shouldn't start from
whatever we loaded from the database, as that data is wrong. We should
start from a clean slate.
If we decide to recalculate the metadata we shouldn't start from
whatever we loaded from the database, as that data is wrong. We should
start from a clean slate.
I was working on indirecting version vectors, and that resulted in some
refactoring and improving the existing block indirection stuff. We may
or may not end up doing the version vector indirection, but I think
these changes are reasonable anyhow and will simplify the diff
significantly if we do go there. The main points are:
- A bunch of renaming to make the indirection and GC not about "blocks"
but about "indirection".
- Adding a cutoff so that we don't actually indirect for small block
lists. This gets us better performance when handling small files as it
cuts out the indirection for quite small loss in space efficiency.
- Being paranoid and always recalculating the hash on put. This costs
some CPU, but the consequences if a buggy or malicious implementation
silently substituted the block list by lying about the hash would be bad.
I was working on indirecting version vectors, and that resulted in some
refactoring and improving the existing block indirection stuff. We may
or may not end up doing the version vector indirection, but I think
these changes are reasonable anyhow and will simplify the diff
significantly if we do go there. The main points are:
- A bunch of renaming to make the indirection and GC not about "blocks"
but about "indirection".
- Adding a cutoff so that we don't actually indirect for small block
lists. This gets us better performance when handling small files as it
cuts out the indirection for quite small loss in space efficiency.
- Being paranoid and always recalculating the hash on put. This costs
some CPU, but the consequences if a buggy or malicious implementation
silently substituted the block list by lying about the hash would be bad.
One of the causes of "panic: database is closed" is that we try to send
summaries after it's been closed. Calculating summaries can take a long
time and if we have a lot of folders it's not unreasonable to think
that we might be stopped in this loop, so prepare to bail here.
* push
During NAT discovery we block for 10s (NatTimeoutS) before returning.
This is mostly noticeable when Ctrl-C:ing a Syncthing directly after
startup as we wait for those ten seconds before shutting down. This
makes it check the context a little bit more frequently.
This adds metadata updates to the same write batch as the underlying
file change. The odds of a metadata update going missing is greatly
reduced.
Bonus change: actually commit the transaction in recalcMeta.
lib/db: Recover sequence number and metadata on startup (fixes#6335)
If we crashed after writing new file entries but before updating
metadata in the database the sequence number and metadata will be wrong.
This fixes that.
We could potentially get a snapshot and then fail to get a releaser,
leaking the snapshot. This takes the releaser first and makes sure to
release it on snapshot error.
The readWriteTransaction offered both commit() (the one to use) and
Commit() (via embedding) where the latter didn't close the read
transaction. This removes the lower cased variant in order to prevent
the mistake.
The only place where the mistake was made was the new gc runner, where
it would leave a read snapshot open forever.
During some other work I discovered these tests weren't great, so I've
rewritten them to be a little better. The real changes here are:
- Don't play games with not starting the folder and such, and don't
construct a fake folder instance -- just use the one the model has. The
folder starts and scans but the folder contents are empty at this point
so that's fine.
- Use a fakefs instead of a temp dir.
- To support the above, implement a fakefs option `?content=true` to
make the fakefs actually retain written content. Use sparingly,
obviously, but it means the fakefs can usually be used instead of an
on disk real directory.
This adds a new config with the simple and concise name
maxConcurrentIncomingRequestKiB. This limits how many bytes we have "in
the air" in the form of response data being read and processed.
After some testing I think that not having this limiter is seldom a
great idea and thus I propose a default value of 256 MiB for this new
setting.
I also refactored the folder IO limiter to be a model/folder attribute
instead of a package global.
Adds a new folder state "Waiting to Sync" in the same vein as the
existing "Waiting to Scan". This vastly improves performances in the
rare cases when there are lots and lots of folders operating.
Also retain the interval over restarts by storing last GC time in the
database. This to make sure that GC eventually happens even if the
interval is configured to a long time (say, a month).
* lib/db: Deduplicate block lists in database (fixes#5898)
This moves the block list in the database out from being just a field on
the FileInfo to being an object of its own. When putting a FileInfo we
marshal the block list separately and store it keyed by the sha256 of
the marshalled block list. When getting, if we are not doing a
"truncated" get, we do an extra read and unmarshal for the block list.
Old block lists are cleared out by a periodic GC sweep. The alternative
would be to use refcounting, but:
- There is a larger risk of getting that wrong and either dropping a
block list in error or keeping them around forever.
- It's tricky with our current database, as we don't have dirty reads.
This means that if we update two FileInfos with identical block lists in
the same transaction we can't just do read/modify/write for the ref
counters as we wouldn't see our own first update. See above about
tracking this and risks about getting it wrong.
GC uses a bloom filter for keys to avoid heavy RAM usage. GC can't run
concurrently with FileInfo updates so there is a new lock around those
operation at the lowlevel.
The end result is a much more compact database, especially for setups
with many peers where files get duplicated many times.
This is per-key-class stats for a large database I'm currently working
with, under the current schema:
```
0x00: 9138161 items, 870876 KB keys + 7397482 KB data, 95 B + 809 B avg, 1637651 B max
0x01: 185656 items, 10388 KB keys + 1790909 KB data, 55 B + 9646 B avg, 924525 B max
0x02: 916890 items, 84795 KB keys + 3667 KB data, 92 B + 4 B avg, 192 B max
0x03: 384 items, 27 KB keys + 5 KB data, 72 B + 15 B avg, 87 B max
0x04: 1109 items, 17 KB keys + 17 KB data, 15 B + 15 B avg, 69 B max
0x06: 383 items, 3 KB keys + 0 KB data, 9 B + 2 B avg, 18 B max
0x07: 510 items, 4 KB keys + 12 KB data, 9 B + 24 B avg, 41 B max
0x08: 1349 items, 12 KB keys + 10 KB data, 9 B + 8 B avg, 17 B max
0x09: 194 items, 0 KB keys + 123 KB data, 5 B + 634 B avg, 11484 B max
0x0a: 3 items, 0 KB keys + 0 KB data, 14 B + 7 B avg, 30 B max
0x0b: 181836 items, 2363 KB keys + 10694 KB data, 13 B + 58 B avg, 173 B max
Total 10426475 items, 968490 KB keys + 9202925 KB data.
```
Note 7.4 GB of data in class 00, total size 9.2 GB. After running the
migration we get this instead:
```
0x00: 9138161 items, 870876 KB keys + 2611392 KB data, 95 B + 285 B avg, 4788 B max
0x01: 185656 items, 10388 KB keys + 1790909 KB data, 55 B + 9646 B avg, 924525 B max
0x02: 916890 items, 84795 KB keys + 3667 KB data, 92 B + 4 B avg, 192 B max
0x03: 384 items, 27 KB keys + 5 KB data, 72 B + 15 B avg, 87 B max
0x04: 1109 items, 17 KB keys + 17 KB data, 15 B + 15 B avg, 69 B max
0x06: 383 items, 3 KB keys + 0 KB data, 9 B + 2 B avg, 18 B max
0x07: 510 items, 4 KB keys + 12 KB data, 9 B + 24 B avg, 41 B max
0x09: 194 items, 0 KB keys + 123 KB data, 5 B + 634 B avg, 11484 B max
0x0a: 3 items, 0 KB keys + 0 KB data, 14 B + 17 B avg, 51 B max
0x0b: 181836 items, 2363 KB keys + 10694 KB data, 13 B + 58 B avg, 173 B max
0x0d: 44282 items, 1461 KB keys + 61081 KB data, 33 B + 1379 B avg, 1637399 B max
Total 10469408 items, 969939 KB keys + 4477905 KB data.
```
Class 00 is now down to 2.6 GB, with just 61 MB added in class 0d.
There will be some additional reads in some cases which theoretically
hurts performance, but this will be more than compensated for by smaller
writes and better compaction.
On my own home setup which just has three devices and a handful of
folders the difference is smaller in absolute numbers of course, but
still less than half the old size:
```
0x00: 297122 items, 20894 KB keys + 306860 KB data, 70 B + 1032 B avg, 103237 B max
0x01: 115299 items, 7738 KB keys + 17542 KB data, 67 B + 152 B avg, 419 B max
0x02: 1430537 items, 121223 KB keys + 5722 KB data, 84 B + 4 B avg, 253 B max
...
Total 1947412 items, 151268 KB keys + 337485 KB data.
```
to:
```
0x00: 297122 items, 20894 KB keys + 37038 KB data, 70 B + 124 B avg, 520 B max
0x01: 115299 items, 7738 KB keys + 17542 KB data, 67 B + 152 B avg, 419 B max
0x02: 1430537 items, 121223 KB keys + 5722 KB data, 84 B + 4 B avg, 253 B max
...
0x0d: 18041 items, 595 KB keys + 71964 KB data, 33 B + 3988 B avg, 101109 B max
Total 1965447 items, 151863 KB keys + 139628 KB data.
```
* wip
* wip
* wip
* wip
Makes the logic a bit clearer and safer. This also sneakily redefines
the 0 interval to also mean disabled, whereas it previously meant ...
sometimes default to 1s, sometimes just spin.
* lib/db, lib/protocol: Compact FileInfo and BlockInfo alignment
This fixes the following two lint warnings
FileInfo: struct of size 160 bytes could be of size 136 bytes
BlockInfo: struct of size 48 bytes could be of size 40 bytes
by reordering fields in alignment order (64 bit fields, then 32 bit
fields, then 16 bit fields (if any), then small ones). The end result is
a slightly less aesthetically pleasing struct field order, but since
these are the objects we often juggle in bulk and keep large queues of I
think it's worth it.
It's a micro optimization, but a cheap one.
This adds error returns to model methods called by the protocol layer.
Returning an error will cause the connection to be torn down as the
message couldn't be handled. Using this to signal that a folder isn't
currently available will then cause a reconnection a few moments later,
when it'll hopefully work better.
Tested manually by running with STRECHECKDBEVERY=0 on a nontrivially
sized setup. This panics reliably before this patch, but just causes a
disconnect/reconnect now.
As foretold by the prophecy, "once the database refactor is merged, then
shall appear a request to propagate errors from the store known
throughout the land as the NamedspacedKV, and it shall be good".
This PR does two things, because one lead to the other:
- Move the leveldb specific stuff into a small "backend" package that
defines a backend interface and the leveldb implementation. This allows,
potentially, in the future, switching the db implementation so another
KV store should we wish to do so.
- Add proper error handling all along the way. The db and backend
packages are now errcheck clean. However, I drew the line at modifying
the FileSet API in order to keep this manageable and not continue
refactoring all of the rest of Syncthing. As such, the FileSet methods
still panic on database errors, except for the "database is closed"
error which is instead handled by silently returning as quickly as
possible, with the assumption that we're anyway "on the way out".
Since we've taken upon ourselves to create a log file by default on
Windows, this adds proper management of that log file. There are two new
options:
-log-max-old-files="3" Number of old files to keep (zero to keep only current).
-log-max-size="10485760" Maximum size of any file (zero to disable log rotation).
The default values result in four files (syncthing.log, synchting.0.log,
..., syncthing.3.log) each up to 10 MiB in size. To not use log rotation
at all, the user can say --log-max-size=0.
* lib/versioner: Reduce surface area
This is a refactor while I was anyway rooting around in the versioner.
Instead of exporting every possible implementation and the factory and
letting the caller do whatever, this now encapsulates all that and
exposes a New() that takes a config.VersioningConfiguration.
Given that and that we don't know (from the outside) how a versioner
works or what state it keeps, we now just construct it once per folder
and keep it around. Previously it was recreated for each restore
request.
* unparam
* wip
- The 'help-box' with nearby devices will appear only when there are
any nearby devices
- IDs of nearby devices will look like links (i.e. underlined when
hovered over)
We incorrectly gave a too small buffer to lz4.Compress, causing it to
allocate in some cases (when the data actually becomes larger when
compressed). This then panicked when passed to the buffer pool.
This ensures a buffer that is large enough, and adds tripwires closer to
the source in case this ever pops up again. There is a test that
exercises the issue.
* gui, lib/api: Adds support for prefers-color-scheme on default theme (fixes#6115)
- Renames current default theme into a new "light" theme
- Modifies assets serving to allow getting assets from different themes
* lib/api: Serve assets from arbitrary theme when path starts with "theme-assets"
* lib/api: Moves constant out of function
* Loads light theme in browsers without support for prefers-color-scheme
* gui: Disables dark theme when printing
* Prevents repeated injection and adds support for older browsers
The CSS is always loaded if there is no support for `matchMedia`.
This is apparently an old benchmarking tool. I'd forgotten about it.
Since 67b8ef1f3e the build script tries to
build all binaries explicitly by default, and this fails on Windows as
this tool doesn't build on Windows.
Kill it with fire.
* Fix bufferpool puts (ref #4976)
There was a logic error in Put() which made us put all large blocks into
segment zero, where we subsequently did not look for them.
I also added a lowest threshold, as we otherwise allocate a 128KiB
buffer when we need 24 bytes for a header and such.
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* smaller stress
* cap/len
* wip
* wip
This makes sure addresses are sorted when coming in from the API. The
database merge operation still checks for correct ordering (which is
quick) and sorts if it isn't correct (legacy database record or
replication peer), but then does a copy first.
Tested with -race in production...
This adds a certificate lifetime parameter to our certificate generation
and hard codes it to twenty years in some uninteresting places. In the
main binary there are a couple of constants but it results in twenty
years for the device certificate and 820 days for the HTTPS one. 820 is
less than the 825 maximum Apple allows nowadays.
This also means we must be prepared for certificates to expire, so I add
some handling for that and generate a new certificate when needed. For
self signed certificates we regenerate a month ahead of time. For other
certificates we leave well enough alone.
Without the desktop plug, launching the syncthing snap will not launch a browser window to the admin UI. Adding this one line will fix that. Tested here on my Ubuntu system with a build from tip of master.
### DO NOT REPORT SECURITY ISSUES IN THIS ISSUE TRACKER
Instead, contact security@syncthing.net directly - see
https://syncthing.net/security.html for more information.
### DO NOT POST SUPPORT REQUESTS OR GENERAL QUESTIONS IN THIS ISSUE TRACKER
Please use the forum at https://forum.syncthing.net/ where a large number of
helpful people hang out. This issue tracker is for reporting bugs or feature
requests directly to the developers. Worst case you might get a short
"that's a bug, please report it on GitHub" response on the forum, in which
case we thank you for your patience and following our advice. :)
### Please use the correct issue tracker
If your problem relates to a Syncthing wrapper or [sub-project](https://github.com/syncthing) such as [Syncthing for Android](https://github.com/syncthing/syncthing-android/issues), [SyncTrayzor](https://github.com/canton7/synctrayzor) or the [documentation](https://github.com/syncthing/docs/issues), please use their respective issue trackers.
### Does your log mention database corruption?
If your Syncthing log reports panics because of database corruption it is most likely a fault with your system's storage or memory. Affected log entries will contain lines starting with `panic: leveldb`. You will need to delete the index database to clear this, by running `syncthing -reset-database`.
### Please do post actual bug reports and feature requests.
If your issue is a bug report, replace this boilerplate with a description
of the problem, being sure to include at least:
- what happened,
- what you expected to happen instead, and
- any steps to reproduce the problem.
Also fill out the version information below and add log output or
screenshots as appropriate.
If your issue is a feature request, simply replace this template text in
If you believe that you've found a Syncthing-related security vulnerability, please report it by sending email to the address security@syncthing.net. The PGP key for security@syncthing.net (B683AD7B76CAB013) below can be used to send encrypted mail or to verify responses received from that address.
If you believe that you've found a Syncthing-related security vulnerability,
please report it by sending email to the address security@syncthing.net. The
[PGP key for security@syncthing.net
(B683AD7B76CAB013)](https://syncthing.net/security-key.txt) can be used to
send encrypted mail or to verify responses received from that address.
message:`panic: filling Blocks: read C:\Users\Serv-Resp-Tizayuca\AppData\Local\Syncthing\index-v0.14.0.db\006561.ldb: Error de datos (comprobación de redundancia cíclica).`,
exp:`panic: filling Blocks: read x: Error de datos (comprobación de redundancia cíclica).`,
fmt.Fprintf(tw,"0x%02x:\t%d items,\t%d KB keys +\t%d KB data,\t%d B +\t%d B avg,\t%d B max\t\n",t,counts[t],ksizes[t]/1000,dsizes[t]/1000,ksizes[t]/counts[t],dsizes[t]/counts[t],max[t])
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
