Unbreak build script

README.md

@@ -25,178 +25,11 @@ making sure large swarms of selfish agents behave and somehow work
 towards a common goal. Here we have a much smaller swarm of cooperative
 agents and a simpler approach will suffice.
 
-Features
---------
+Documentation
+=============
 
-> To request features and file bugs, see [the issue tracker][issues].
-
-The following features are _currently implemented and working_:
-
- * The formation of a cluster of nodes, certificate authenticated and
-   communicating over TLS over TCP.
-
- * Synchronization of a single directory among the cluster nodes.
-
- * Change detection by periodic scanning of the local repository.
-
- * Static configuration of cluster nodes.
-
- * Automatic discovery of cluster nodes. See [discover.go][discover.go]
-   for the protocol specification. Discovery on the LAN is performed by
-   broadcasts, Internet wide discovery is performed with the assistance
-   of a global server.
-
- * Handling of deleted files. Deletes can be propagated or ignored per
-   client.
-
- * Synchronizing multiple unrelated directory trees by following
-   symlinks directly below the repository level.
-
- * HTTP GUI.
-
-The following features are _not yet implemented but planned_:
-
- * Change detection by listening to file system notifications instead of
-   periodic scanning.
-
-The following features are _not implemented but may be implemented_ in
-the future:
-
- * Syncing multiple directories from the same syncthing instance.
-
- * Automatic NAT handling via UPNP.
-
- * Conflict resolution. Currently whichever file has the newest
-   modification time "wins". The correct behavior in the face of
-   conflicts is open for discussion.
-
-[discover.go]: https://github.com/calmh/syncthing/blob/master/discover/discover.go
-[issues]: https://github.com/calmh/syncthing/issues
-
-Security
---------
-
-Security is one of the primary project goals. This means that it should
-not be possible for an attacker to join a cluster uninvited, and it
-should not be possible to extract private information from intercepted
-traffic. Currently this is implemented as follows.
-
-All traffic is protected by TLS. To prevent uninvited nodes from joining
-a cluster, the certificate fingerprint of each node is compared to a
-preset list of acceptable nodes at connection establishment. The
-fingerprint is computed as the SHA-1 hash of the certificate and
-displayed in BASE32 encoding to form a compact yet convenient string.
-Currently SHA-1 is deemed secure against preimage attacks.
-
-Incoming requests for file data are verified to the extent that the
-requested file name must exist in the local index and the global model.
-
-Installing
-==========
-
-Download the appropriate precompiled binary from the
-[releases](https://github.com/calmh/syncthing/releases) page. Untar and
-put the `syncthing` binary somewhere convenient in your `$PATH`.
-
-If you are a developer and have Go 1.2 installed you can also install
-the latest version from source. `go get` works as expected but builds
-a binary without GUI capabilities. Use the included `build.sh` script
-without parameters to build a syncthing with GUI.
-
-Usage
-=====
-
-Check out the options:
-
-```
-$ syncthing --help
-Usage:
-  syncthing [options]
-
-...
-```
-
-Run syncthing to let it create it's config directory and certificate:
-
-```
-$ syncthing
-11:34:13 main.go:85: INFO: Version v0.1-40-gbb0fd87
-11:34:13 tls.go:61: OK: Created TLS certificate file
-11:34:13 tls.go:67: OK: Created TLS key file
-11:34:13 main.go:66: INFO: My ID: NCTBZAAHXR6ZZP3D7SL3DLYFFQERMW4Q
-11:34:13 main.go:90: FATAL: No config file
-```
-
-Take note of the "My ID: ..." line. Perform the same operation on
-another computer to create another node. Take note of that ID as well,
-and create a config file `~/.syncthing/syncthing.ini` looking something
-like this:
-
-```
-[repository]
-dir = /Users/jb/Synced
-
-[nodes]
-NCTBZAAHXR6ZZP3D7SL3DLYFFQERMW4Q = 172.16.32.1:22000 192.23.34.56:22000
-CUGAE43Y5N64CRJU26YFH6MTWPSBLSUL = dynamic
-```
-
-This assumes that the first node is reachable on either of the two
-addresses listed (perhaps one internal and one port-forwarded external)
-and that the other node is not normally reachable from the outside. Save
-this config file, identically, to both nodes.
-
-If the nodes are running on the same network, or reachable on port 22000
-from the outside world, you can set all addresses to "dynamic" and they
-will find each other using automatic discovery. (This discovery,
-including port numbers, can be tweaked or disabled using command line
-options.)
-
-Start syncthing on both nodes. For the cautious, one side can be set to
-be read only.
-
-```
-$ syncthing --ro
-13:30:55 main.go:85: INFO: Version v0.1-40-gbb0fd87
-13:30:55 main.go:102: INFO: My ID: NCTBZAAHXR6ZZP3D7SL3DLYFFQERMW4Q
-13:30:55 main.go:149: INFO: Initial repository scan in progress
-13:30:59 main.go:153: INFO: Listening for incoming connections
-13:30:59 main.go:157: INFO: Attempting to connect to other nodes
-13:30:59 main.go:247: INFO: Starting local discovery
-13:30:59 main.go:165: OK: Ready to synchronize
-13:31:04 discover.go:113: INFO: Discovered node CUGAE43Y5N64CRJU26YFH6MTWPSBLSUL at 172.16.32.24:22000
-13:31:14 main.go:296: INFO: Connected to node CUGAE43Y5N64CRJU26YFH6MTWPSBLSUL
-13:31:19 main.go:345: INFO: Transferred 139 KiB in (14 KiB/s), 139 KiB out (14 KiB/s)
-13:32:20 model.go:94: INFO: CUGAE43Y5N64CRJU26YFH6MTWPSBLSUL: 263.4 KB/s in, 69.1 KB/s out
-13:32:20 model.go:104: INFO:  18289 files,  24.24 GB in cluster
-13:32:20 model.go:111: INFO:  17132 files,  22.39 GB in local repo
-13:32:20 model.go:117: INFO:   1157 files,   1.84 GB to synchronize
-...
-```
-You should see the synchronization start and then finish a short while
-later. Add nodes to taste.
-
-GUI
----
-
-The web based GUI is disabled per default. To enable and access it you
-must start syncthing with the `--gui` command line option, giving a
-listen address. For example:
-
-```
-$ syncthing --gui 127.0.0.1:8080
-```
-
-You then point your browser to the given address.
-
-Excluding Files
----------------
-
-syncthing looks for files named `.stignore` while walking the
-repository. The file is expected to contain glob patterns of file names
-to ignore. Patterns are matched on file name only and apply to files in
-the same directory as the `.stignore` file and in directories lower down
-in the hierarchy.
+The syncthing documentation is kept on the
+[GitHub Wiki](https://github.com/calmh/syncthing/wiki).
 
 License
 =======

build.sh

@@ -7,6 +7,14 @@ if [[ -z $1 ]] ; then
 	go test ./...
 	go build -ldflags "-X main.Version $version" \
 	&& nrsc syncthing gui
+elif [[ $1 == "tar" ]] ; then
+	go test ./...
+	go build -ldflags "-X main.Version $version" \
+	&& nrsc syncthing gui \
+	&& mkdir syncthing-dist \
+	&& cp syncthing README.md LICENSE syncthing-dist \
+	&& tar zcvf syncthing-dist.tar.gz syncthing-dist \
+	&& rm -rf syncthing-dist
 else
 	go test ./... || exit 1
 
@@ -39,7 +47,7 @@ else
 			go build -ldflags "-X main.Version $version" \
 				&& nrsc syncthing.exe gui \
 				&& mkdir -p "$name" \
-				&& cp syncthing.exe "$buildDir/$name.exe" \
+				&& mv syncthing.exe "$buildDir/$name.exe" \
 				&& cp README.md LICENSE "$name" \
 				&& zip -qr "$buildDir/$name.zip" "$name" \
 				&& rm -r  "$name"

main.go

@@ -24,8 +24,8 @@ type Options struct {
 	ConfDir    string           `short:"c" long:"cfg" description:"Configuration directory" default:"~/.syncthing" value-name:"DIR"`
 	Listen     string           `short:"l" long:"listen" description:"Listen address" default:":22000" value-name:"ADDR"`
 	ReadOnly   bool             `short:"r" long:"ro" description:"Repository is read only"`
-	Delete     bool             `short:"d" long:"delete" description:"Delete files deleted from cluster"`
 	Rehash     bool             `long:"rehash" description:"Ignore cache and rehash all files in repository"`
+	NoDelete   bool             `long:"no-delete" description:"Never delete files"`
 	NoSymlinks bool             `long:"no-symlinks" description:"Don't follow first level symlinks in the repo"`
 	NoStats    bool             `long:"no-stats" description:"Don't print model and connection statistics"`
 	GUIAddr    string           `long:"gui" description:"GUI listen address" default:"" value-name:"ADDR"`
@@ -164,13 +164,13 @@ func main() {
 	// Routine to pull blocks from other nodes to synchronize the local
 	// repository. Does not run when we are in read only (publish only) mode.
 	if !opts.ReadOnly {
-		if opts.Delete {
-			infoln("Deletes from peer nodes are allowed")
-		} else {
+		if opts.NoDelete {
 			infoln("Deletes from peer nodes will be ignored")
+		} else {
+			infoln("Deletes from peer nodes are allowed")
 		}
 		okln("Ready to synchronize (read-write)")
-		m.StartRW(opts.Delete, opts.Advanced.FilesInFlight, opts.Advanced.RequestsInFlight)
+		m.StartRW(!opts.NoDelete, opts.Advanced.FilesInFlight, opts.Advanced.RequestsInFlight)
 	} else {
 		okln("Ready to synchronize (read only; no external updates accepted)")
 	}

model/model.go

@@ -50,6 +50,9 @@ type Model struct {
 	delete         bool
 
 	trace map[string]bool
+
+	fileLastChanged   map[string]time.Time
+	fileWasSuppressed map[string]int
 }
 
 const (
@@ -57,6 +60,9 @@ const (
 
 	idxBcastHoldtime = 15 * time.Second  // Wait at least this long after the last index modification
 	idxBcastMaxDelay = 120 * time.Second // Unless we've already waited this long
+
+	minFileHoldTimeS = 60  // Never allow file changes more often than this
+	maxFileHoldTimeS = 600 // Always allow file changes at least this often
 )
 
 var ErrNoSuchFile = errors.New("no such file")
@@ -66,15 +72,17 @@ var ErrNoSuchFile = errors.New("no such file")
 // for file data without altering the local repository in any way.
 func NewModel(dir string) *Model {
 	m := &Model{
-		dir:          dir,
-		global:       make(map[string]File),
-		local:        make(map[string]File),
-		remote:       make(map[string]map[string]File),
-		need:         make(map[string]bool),
-		nodes:        make(map[string]*protocol.Connection),
-		rawConn:      make(map[string]io.ReadWriteCloser),
-		lastIdxBcast: time.Now(),
-		trace:        make(map[string]bool),
+		dir:               dir,
+		global:            make(map[string]File),
+		local:             make(map[string]File),
+		remote:            make(map[string]map[string]File),
+		need:              make(map[string]bool),
+		nodes:             make(map[string]*protocol.Connection),
+		rawConn:           make(map[string]io.ReadWriteCloser),
+		lastIdxBcast:      time.Now(),
+		trace:             make(map[string]bool),
+		fileLastChanged:   make(map[string]time.Time),
+		fileWasSuppressed: make(map[string]int),
 	}
 
 	go m.broadcastIndexLoop()
@@ -304,6 +312,7 @@ func (m *Model) Request(nodeID, name string, offset uint64, size uint32, hash []
 }
 
 // ReplaceLocal replaces the local repository index with the given list of files.
+// Change suppression is applied to files changing too often.
 func (m *Model) ReplaceLocal(fs []File) {
 	m.Lock()
 	defer m.Unlock()
@@ -391,6 +400,28 @@ func (m *Model) AddConnection(conn io.ReadWriteCloser, nodeID string) {
 	}()
 }
 
+func (m *Model) shouldSuppressChange(name string) bool {
+	sup := shouldSuppressChange(m.fileLastChanged[name], m.fileWasSuppressed[name])
+	if sup {
+		m.fileWasSuppressed[name]++
+	} else {
+		m.fileWasSuppressed[name] = 0
+		m.fileLastChanged[name] = time.Now()
+	}
+	return sup
+}
+
+func shouldSuppressChange(lastChange time.Time, numChanges int) bool {
+	sinceLast := time.Since(lastChange)
+	if sinceLast > maxFileHoldTimeS*time.Second {
+		return false
+	}
+	if sinceLast < time.Duration((numChanges+2)*minFileHoldTimeS)*time.Second {
+		return true
+	}
+	return false
+}
+
 // protocolIndex returns the current local index in protocol data types.
 // Must be called with the read lock held.
 func (m *Model) protocolIndex() []protocol.FileInfo {

model/model_puller.go

@@ -121,6 +121,11 @@ func (m *Model) pullFile(name string) error {
 		return err
 	}
 
+	err = os.Chmod(tmpFilename, os.FileMode(globalFile.Flags&0777))
+	if err != nil {
+		return err
+	}
+
 	err = os.Rename(tmpFilename, filename)
 	if err != nil {
 		return err

model/model_test.go

@@ -340,3 +340,28 @@ func TestRequest(t *testing.T) {
 		t.Errorf("Unexpected non nil data on insecure file read: %q", string(bs))
 	}
 }
+
+func TestSuppression(t *testing.T) {
+	var testdata = []struct {
+		lastChange time.Time
+		hold       int
+		result     bool
+	}{
+		{time.Unix(0, 0), 0, false},                    // First change
+		{time.Now().Add(-1 * time.Second), 0, true},    // Changed once one second ago, suppress
+		{time.Now().Add(-119 * time.Second), 0, true},  // Changed once 119 seconds ago, suppress
+		{time.Now().Add(-121 * time.Second), 0, false}, // Changed once 121 seconds ago, permit
+
+		{time.Now().Add(-179 * time.Second), 1, true},  // Suppressed once 179 seconds ago, suppress again
+		{time.Now().Add(-181 * time.Second), 1, false}, // Suppressed once 181 seconds ago, permit
+
+		{time.Now().Add(-599 * time.Second), 99, true},  // Suppressed lots of times, last allowed 599 seconds ago, suppress again
+		{time.Now().Add(-601 * time.Second), 99, false}, // Suppressed lots of times, last allowed 601 seconds ago, permit
+	}
+
+	for i, tc := range testdata {
+		if shouldSuppressChange(tc.lastChange, tc.hold) != tc.result {
+			t.Errorf("Incorrect result for test #%d: %v", i, tc)
+		}
+	}
+}

model/walk.go

@@ -9,6 +9,7 @@ import (
 	"path"
 	"path/filepath"
 	"strings"
+	"time"
 )
 
 const BlockSize = 128 * 1024
@@ -81,17 +82,38 @@ func (m *Model) genWalker(res *[]File, ign map[string][]string) filepath.WalkFun
 				// No change
 				*res = append(*res, hf)
 			} else {
+				m.Lock()
+				if m.shouldSuppressChange(rn) {
+					if m.trace["file"] {
+						log.Println("FILE: SUPPRESS:", rn, m.fileWasSuppressed[rn], time.Since(m.fileLastChanged[rn]))
+					}
+
+					if ok {
+						// Files that are ignored will be suppressed but don't actually exist in the local model
+						*res = append(*res, hf)
+					}
+					m.Unlock()
+					return nil
+				}
+				m.Unlock()
+
 				if m.trace["file"] {
 					log.Printf("FILE: Hash %q", p)
 				}
 				fd, err := os.Open(p)
 				if err != nil {
+					if m.trace["file"] {
+						log.Printf("FILE: %q: %v", p, err)
+					}
 					return nil
 				}
 				defer fd.Close()
 
 				blocks, err := Blocks(fd, BlockSize)
 				if err != nil {
+					if m.trace["file"] {
+						log.Printf("FILE: %q: %v", p, err)
+					}
 					return nil
 				}
 				f := File{