Update docs & translation

Revert "Don't show "Rescan" button for stopped folders"

.gitattributes

@@ -0,0 +1,9 @@
+# Text files use LF line endings in this repository
+*   text=auto
+
+# Except the dependencies, which we leave alone
+Godeps/**   -text=auto
+
+# Diffs on these files are meaningless
+gui.files.go    -diff
+*.svg           -diff

.gitignore

@@ -1,15 +1,16 @@
 syncthing
+!gui/syncthing
+!Godeps/_workspace/src/github.com/syncthing
 syncthing.exe
 *.tar.gz
 *.zip
 *.asc
-*.sublime*
-.idea/
 .jshintrc
 coverage.out
 files/pidx
 bin
 perfstats*.csv
 coverage.xml
-!gui/scripts/syncthing
-.DS_Store
+syncthing.sig
+RELEASE
+deb

AUTHORS

@@ -1,35 +1,72 @@
 # This is the official list of Syncthing authors for copyright purposes.
 
 Aaron Bieber <qbit@deftly.net>
+Adam Piggott	<aD@simplypeachy.co.uk> <simplypeachy@users.noreply.github.com>
 Alexander Graf <register-github@alex-graf.de>
 Andrew Dunham <andrew@du.nham.ca>
-Audrius Butkevicius <audrius.butkevicius@gmail.com>
+Antony Male <antony.male@gmail.com>
 Arthur Axel fREW Schmidt <frew@afoolishmanifesto.com> <frioux@gmail.com>
+Audrius Butkevicius <audrius.butkevicius@gmail.com>
+Bart De Vries <devriesb@gmail.com>
+Ben Curthoys <ben@bencurthoys.com>
 Ben Schulz <ueomkail@gmail.com> <uok@users.noreply.github.com>
 Ben Sidhom <bsidhom@gmail.com>
 Brandon Philips <brandon@ifup.org>
+Brendan Long <self@brendanlong.com>
+Brian R. Becker <brbecker@gmail.com>
 Caleb Callaway <enlightened.despot@gmail.com>
+Carsten Hagemann <moter8@gmail.com>
 Cathryne Linenweaver <cathryne.linenweaver@gmail.com> <Cathryne@users.noreply.github.com>
+Chris Howie <me@chrishowie.com>
 Chris Joel <chris@scriptolo.gy>
+Colin Kennedy <moshen.colin@gmail.com>
+Daniel Bergmann <dan.arne.bergmann@gmail.com> <brgmnn@users.noreply.github.com>
 Daniel Martí <mvdan@mvdan.cc>
+Denis A. <denisva@gmail.com>
 Dennis Wilson <dw@risu.io>
 Dominik Heidler <dominik@heidler.eu>
+Elias Jarlebring <jarlebring@gmail.com>
 Emil Hessman <emil@hessman.se>
+Erik Meitner <e.meitner@willystreet.coop>
+Federico Castagnini <federico.castagnini@gmail.com>
 Felix Ableitner <me@nutomic.com>
 Felix Unterpaintner <bigbear2nd@gmail.com>
+Francois-Xavier Gsell <fxgsell@gmail.com>
+Frank Isemann <frank@isemann.name>
 Gilli Sigurdsson <gilli@vx.is>
+Jacek Szafarkiewicz <szafar@linux.pl>
 Jakob Borg <jakob@nym.se>
+Jake Peterson <jake@acogdev.com>
 James Patterson <jamespatterson@operamail.com> <jpjp@users.noreply.github.com>
+Jaroslav Malec <dzardacz@gmail.com>
 Jens Diemer <github.com@jensdiemer.de> <git@jensdiemer.de>
 Jochen Voss <voss@seehuhn.de>
+Johan Vromans <jvromans@squirrel.nl>
+Karol Różycki <rozycki.karol@gmail.com>
+Ken'ichi Kamada <kamada@nanohz.org>
 Lode Hoste <zillode@zillode.be>
+Lord Landon Agahnim <lordlandon@gmail.com>
+Marc Laporte <marc@marclaporte.com> <marc@laporte.name>
+Marc Pujol <kilburn@la3.org>
 Marcin Dziadus <dziadus.marcin@gmail.com>
+Mateusz Naściszewski <matin1111@wp.pl>
+Matt Burke <mburke@amplify.com> <burkemw3@gmail.com>
+Michael Jephcote <rewt0r@gmx.com> <Rewt0r@users.noreply.github.com>
 Michael Tilli <pyfisch@gmail.com>
+Pascal Jungblut <github@pascalj.com> <mail@pascal-jungblut.com>
+Peter Hoeg <peter@speartail.com>
 Philippe Schommers <philippe@schommers.be>
 Phill Luby <phill.luby@newredo.com>
 Piotr Bejda <piotrb10@gmail.com>
 Ryan Sullivan <kayoticsully@gmail.com>
+Sergey Mishin <ralder@yandex.ru>
+Stefan Tatschner <stefan@sevenbyte.org> <rumpelsepp@sevenbyte.org>
+Stefan Kuntz <stefan.github@gmail.com> <Stefan.github@gmail.com>
+Tim Abell <tim@timwise.co.uk>
+Tobias Nygren <tnn@nygren.pp.se>
 Tomas Cerveny <kozec@kozec.com>
 Tully Robinson <tully@tojr.org>
+Tyler Brazier <tyler@tylerbrazier.com>
 Veeti Paananen <veeti.paananen@rojekti.fi>
 Vil Brekin <vilbrekin@gmail.com>
+Yannic A. <eipiminusone+github@gmail.com> <eipiminus1@users.noreply.github.com>

CONTRIBUTING.md

@@ -32,109 +32,21 @@ latest info on Transifex.
 ## Contributing Code
 
 Every contribution is welcome. If you want to contribute but are unsure
-where to start, any open issues are fair game! Be prepared for a
-[certain amount of review](https://discourse.syncthing.net/t/733); it's
-all in the name of quality. :) Following the points below will make this
-a smoother process.
+where to start, any open issues are fair game! See the [Contribution
+Guidelines](http://docs.syncthing.net/dev/contributing.html) for the full
+story on committing code.
 
-Individuals making significant and valuable contributions are given
-commit-access to the project. If you make a significant contribution and
-are not considered for commit-access, please contact any of the
-Syncthing core team members.
+## Contributing Documentation
 
-All nontrivial contributions should go through the pull request
-mechanism for internal review. Determining what is "nontrivial" is left
-at the discretion of the contributor.
-
-### Authorship
-
-All code authors are listed in the AUTHORS file. Commits must be made
-with the same name and email as listed in the AUTHORS file. To
-accomplish this, ensure that your git configuration is set correctly
-prior to making your first commit;
-
-    $ git config --global user.name "Jane Doe"
-    $ git config --global user.email janedoe@example.com
-
-You must be reachable on the given email address. If you do not wish to
-use your real name for whatever reason, using a nickname or pseudonym is
-perfectly acceptable.
-
-### Core Team
-
-The Syncthing core team currently consists of the following members;
-
- - Jakob Borg (@calmh)
- - Audrius Butkevicius (@AudriusButkevicius)
-
-## Coding Style
-
-- Follow the conventions laid out in [Effective Go](https://golang.org/doc/effective_go.html)
-  as much as makes sense.
-
-- All text files use Unix line endings.
-
-- Each commit should be `go fmt` clean.
-
-- The commit message subject should be a single short sentence
-  describing the change, starting with a capital letter.
-
-- Commits that resolve an existing issue must include the issue number
-  as `(fixes #123)` at the end of the commit message subject.
-
-- Imports are grouped per `goimports` standard; that is, standard
-  library first, then third party libraries after a blank line.
-
-- A contribution solving a single issue or introducing a single new
-  feature should probably be a single commit based on the current
-  `master` branch. You may be asked to "rebase" or "squash" your pull
-  request to make sure this is the case, especially if there have been
-  amendments during review.
+Updates to the [documentation site](http://docs.syncthing.net/) can be
+made as pull requests on the [documentation
+repository](https://github.com/syncthing/docs).
 
 ## Licensing
 
-All contributions are made under the same GPL license as the rest of the
-project, except documentation, user interface text and translation
+All contributions are made under the same MPLv2 license as the rest of
+the project, except documentation, user interface text and translation
 strings which are licensed under the Creative Commons Attribution 4.0
 International License. You retain the copyright to code you have
 written.
 
-When accepting your first contribution, the maintainer of the project
-will ensure that you are added to the AUTHORS file. You are welcome to
-add yourself as a separate commit in your first pull request.
-
-## Building
-
-[See the documentation](http://discourse.syncthing.net/t/44) on how to
-get started with a build environment.
-
-## Branches
-
- - `master` is the main branch containing good code that will end up in
-   the next release. You should base your work on it. It won't ever be
-   rebased or force-pushed to.
-
- - `vx.y` branches exist to make patch releases on otherwise obsolete
-   minor releases. Should only contain fixes cherry picked from master.
-   Don't base any work on them.
-
- - Other branches are probably topic branches and may be subject to
-   rebasing. Don't base any work on them unless you specifically know
-   otherwise.
-
-## Tags
-
-All releases are tagged semver style as `vx.y.z`. Release tags are
-signed by GPG key BCE524C7.
-
-## Tests
-
-Yes please!
-
-## Documentation
-
-[Over here!](http://discourse.syncthing.net/category/documentation)
-
-## License
-
-GPLv3

Godeps/Godeps.json

@@ -1,41 +1,46 @@
 {
 	"ImportPath": "github.com/syncthing/syncthing",
-	"GoVersion": "go1.4rc1",
+	"GoVersion": "go1.5.1",
 	"Packages": [
 		"./cmd/..."
 	],
 	"Deps": [
-		{
-			"ImportPath": "github.com/AudriusButkevicius/lfu-go",
-			"Rev": "164bcecceb92fd6037f4d18a8d97b495ec6ef669"
-		},
 		{
 			"ImportPath": "github.com/bkaradzic/go-lz4",
-			"Rev": "93a831dcee242be64a9cc9803dda84af25932de7"
+			"Rev": "74ddf82598bc4745b965729e9c6a463bedd33049"
 		},
 		{
-			"ImportPath": "github.com/calmh/logger",
-			"Rev": "f50d32b313bec2933a3e1049f7416a29f3413d29"
+			"ImportPath": "github.com/calmh/du",
+			"Rev": "3c0690cca16228b97741327b1b6781397afbdb24"
 		},
 		{
-			"ImportPath": "github.com/calmh/osext",
-			"Rev": "9bf61584e5f1f172e8766ddc9022d9c401faaa5e"
+			"ImportPath": "github.com/calmh/luhn",
+			"Rev": "0c8388ff95fa92d4094011e5a04fc99dea3d1632"
 		},
 		{
 			"ImportPath": "github.com/calmh/xdr",
-			"Rev": "45c46b7db7ff83b8b9ee09bbd95f36ab50043ece"
+			"Rev": "47c0042d09a827b81ee62497f99e5e0c7f0bd31c"
+		},
+		{
+			"ImportPath": "github.com/golang/snappy",
+			"Rev": "723cc1e459b8eea2dea4583200fd60757d40097a"
 		},
 		{
 			"ImportPath": "github.com/juju/ratelimit",
-			"Rev": "f9f36d11773655c0485207f0ad30dc2655f69d56"
+			"Rev": "772f5c38e468398c4511514f4f6aa9a4185bc0a0"
+		},
+		{
+			"ImportPath": "github.com/kardianos/osext",
+			"Rev": "6e7f843663477789fac7c02def0d0909e969b4e5"
 		},
 		{
 			"ImportPath": "github.com/syndtr/goleveldb/leveldb",
-			"Rev": "97e257099d2ab9578151ba85e2641e2cd14d3ca8"
+			"Rev": "1a9d62f03ea92815b46fcaab357cfd4df264b1a0"
 		},
 		{
-			"ImportPath": "github.com/syndtr/gosnappy/snappy",
-			"Rev": "ce8acff4829e0c2458a67ead32390ac0a381c862"
+			"ImportPath": "github.com/thejerf/suture",
+			"Comment": "v1.0.1",
+			"Rev": "99c1f2d613756768fc4299acd9dc621e11ed3fd7"
 		},
 		{
 			"ImportPath": "github.com/vitrun/qart/coding",
@@ -51,23 +56,31 @@
 		},
 		{
 			"ImportPath": "golang.org/x/crypto/bcrypt",
-			"Comment": "null-236",
-			"Rev": "69e2a90ed92d03812364aeb947b7068dc42e561e"
+			"Rev": "81bf7719a6b7ce9b665598222362b50122dfc13b"
 		},
 		{
 			"ImportPath": "golang.org/x/crypto/blowfish",
-			"Comment": "null-236",
-			"Rev": "69e2a90ed92d03812364aeb947b7068dc42e561e"
+			"Rev": "81bf7719a6b7ce9b665598222362b50122dfc13b"
+		},
+		{
+			"ImportPath": "golang.org/x/net/internal/iana",
+			"Rev": "4b709d93778b93d2f34943e3142c71578d83ad31"
+		},
+		{
+			"ImportPath": "golang.org/x/net/ipv6",
+			"Rev": "4b709d93778b93d2f34943e3142c71578d83ad31"
+		},
+		{
+			"ImportPath": "golang.org/x/net/proxy",
+			"Rev": "4b709d93778b93d2f34943e3142c71578d83ad31"
 		},
 		{
 			"ImportPath": "golang.org/x/text/transform",
-			"Comment": "null-112",
-			"Rev": "2f707e0ad64637ca1318279be7201f5ed19c4050"
+			"Rev": "723492b65e225eafcba054e76ba18bb9c5ac1ea2"
 		},
 		{
 			"ImportPath": "golang.org/x/text/unicode/norm",
-			"Comment": "null-112",
-			"Rev": "2f707e0ad64637ca1318279be7201f5ed19c4050"
+			"Rev": "723492b65e225eafcba054e76ba18bb9c5ac1ea2"
 		}
 	]
 }

Godeps/_workspace/src/github.com/AudriusButkevicius/lfu-go/LICENSE

@@ -1,19 +0,0 @@
-Copyright (C) 2012 Dave Grijalva
-
-Permission is hereby granted, free of charge, to any person obtaining a 
-copy of this software and associated documentation files (the "Software"), 
-to deal in the Software without restriction, including without limitation 
-the rights to use, copy, modify, merge, publish, distribute, sublicense, 
-and/or sell copies of the Software, and to permit persons to whom the 
-Software is furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included 
-in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 
-OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 
-THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 
-THE SOFTWARE.

Godeps/_workspace/src/github.com/AudriusButkevicius/lfu-go/README.md

@@ -1,19 +0,0 @@
-A simple LFU cache for golang.  Based on the paper [An O(1) algorithm for implementing the LFU cache eviction scheme](http://dhruvbird.com/lfu.pdf).
-
-Usage:
-
-```go
-import "github.com/dgrijalva/lfu-go"
-
-// Make a new thing
-c := lfu.New()
-
-// Set some values
-c.Set("myKey", myValue)
-
-// Retrieve some values
-myValue = c.Get("myKey")
-
-// Evict some values
-c.Evict(1)
-```

Godeps/_workspace/src/github.com/AudriusButkevicius/lfu-go/lfu.go

@@ -1,156 +0,0 @@
-package lfu
-
-import (
-	"container/list"
-	"sync"
-)
-
-type Eviction struct {
-	Key   string
-	Value interface{}
-}
-
-type Cache struct {
-	// If len > UpperBound, cache will automatically evict
-	// down to LowerBound.  If either value is 0, this behavior
-	// is disabled.
-	UpperBound      int
-	LowerBound      int
-	values          map[string]*cacheEntry
-	freqs           *list.List
-	len             int
-	lock            *sync.Mutex
-	EvictionChannel chan<- Eviction
-}
-
-type cacheEntry struct {
-	key      string
-	value    interface{}
-	freqNode *list.Element
-}
-
-type listEntry struct {
-	entries map[*cacheEntry]byte
-	freq    int
-}
-
-func New() *Cache {
-	c := new(Cache)
-	c.values = make(map[string]*cacheEntry)
-	c.freqs = list.New()
-	c.lock = new(sync.Mutex)
-	return c
-}
-
-func (c *Cache) Get(key string) interface{} {
-	c.lock.Lock()
-	defer c.lock.Unlock()
-	if e, ok := c.values[key]; ok {
-		c.increment(e)
-		return e.value
-	}
-	return nil
-}
-
-func (c *Cache) Set(key string, value interface{}) {
-	c.lock.Lock()
-	defer c.lock.Unlock()
-	if e, ok := c.values[key]; ok {
-		// value already exists for key.  overwrite
-		e.value = value
-		c.increment(e)
-	} else {
-		// value doesn't exist.  insert
-		e := new(cacheEntry)
-		e.key = key
-		e.value = value
-		c.values[key] = e
-		c.increment(e)
-		c.len++
-		// bounds mgmt
-		if c.UpperBound > 0 && c.LowerBound > 0 {
-			if c.len > c.UpperBound {
-				c.evict(c.len - c.LowerBound)
-			}
-		}
-	}
-}
-
-func (c *Cache) Len() int {
-	c.lock.Lock()
-	defer c.lock.Unlock()
-	return c.len
-}
-
-func (c *Cache) Evict(count int) int {
-	c.lock.Lock()
-	defer c.lock.Unlock()
-	return c.evict(count)
-}
-
-func (c *Cache) evict(count int) int {
-	// No lock here so it can be called
-	// from within the lock (during Set)
-	var evicted int
-	for i := 0; i < count; {
-		if place := c.freqs.Front(); place != nil {
-			for entry, _ := range place.Value.(*listEntry).entries {
-				if i < count {
-					if c.EvictionChannel != nil {
-						c.EvictionChannel <- Eviction{
-							Key:   entry.key,
-							Value: entry.value,
-						}
-					}
-					delete(c.values, entry.key)
-					c.remEntry(place, entry)
-					evicted++
-					c.len--
-					i++
-				}
-			}
-		}
-	}
-	return evicted
-}
-
-func (c *Cache) increment(e *cacheEntry) {
-	currentPlace := e.freqNode
-	var nextFreq int
-	var nextPlace *list.Element
-	if currentPlace == nil {
-		// new entry
-		nextFreq = 1
-		nextPlace = c.freqs.Front()
-	} else {
-		// move up
-		nextFreq = currentPlace.Value.(*listEntry).freq + 1
-		nextPlace = currentPlace.Next()
-	}
-
-	if nextPlace == nil || nextPlace.Value.(*listEntry).freq != nextFreq {
-		// create a new list entry
-		li := new(listEntry)
-		li.freq = nextFreq
-		li.entries = make(map[*cacheEntry]byte)
-		if currentPlace != nil {
-			nextPlace = c.freqs.InsertAfter(li, currentPlace)
-		} else {
-			nextPlace = c.freqs.PushFront(li)
-		}
-	}
-	e.freqNode = nextPlace
-	nextPlace.Value.(*listEntry).entries[e] = 1
-	if currentPlace != nil {
-		// remove from current position
-		c.remEntry(currentPlace, e)
-	}
-}
-
-func (c *Cache) remEntry(place *list.Element, entry *cacheEntry) {
-	entries := place.Value.(*listEntry).entries
-	delete(entries, entry)
-	if len(entries) == 0 {
-		c.freqs.Remove(place)
-	}
-}

Godeps/_workspace/src/github.com/AudriusButkevicius/lfu-go/lfu_test.go

@@ -1,68 +0,0 @@
-package lfu
-
-import (
-	"fmt"
-	"testing"
-)
-
-func TestLFU(t *testing.T) {
-	c := New()
-	c.Set("a", "a")
-	if v := c.Get("a"); v != "a" {
-		t.Errorf("Value was not saved: %v != 'a'", v)
-	}
-	if l := c.Len(); l != 1 {
-		t.Errorf("Length was not updated: %v != 1", l)
-	}
-
-	c.Set("b", "b")
-	if v := c.Get("b"); v != "b" {
-		t.Errorf("Value was not saved: %v != 'b'", v)
-	}
-	if l := c.Len(); l != 2 {
-		t.Errorf("Length was not updated: %v != 2", l)
-	}
-
-	c.Get("a")
-	evicted := c.Evict(1)
-	if v := c.Get("a"); v != "a" {
-		t.Errorf("Value was improperly evicted: %v != 'a'", v)
-	}
-	if v := c.Get("b"); v != nil {
-		t.Errorf("Value was not evicted: %v", v)
-	}
-	if l := c.Len(); l != 1 {
-		t.Errorf("Length was not updated: %v != 1", l)
-	}
-	if evicted != 1 {
-		t.Errorf("Number of evicted items is wrong: %v != 1", evicted)
-	}
-}
-
-func TestBoundsMgmt(t *testing.T) {
-	c := New()
-	c.UpperBound = 10
-	c.LowerBound = 5
-
-	for i := 0; i < 100; i++ {
-		c.Set(fmt.Sprintf("%v", i), i)
-	}
-	if c.Len() > 10 {
-		t.Errorf("Bounds management failed to evict properly: %v", c.Len())
-	}
-}
-
-func TestEviction(t *testing.T) {
-	ch := make(chan Eviction, 1)
-
-	c := New()
-	c.EvictionChannel = ch
-	c.Set("a", "b")
-	c.Evict(1)
-
-	ev := <-ch
-
-	if ev.Key != "a" || ev.Value.(string) != "b" {
-		t.Error("Incorrect item")
-	}
-}

Godeps/_workspace/src/github.com/bkaradzic/go-lz4/.travis.yml

@@ -4,4 +4,6 @@ go:
   - 1.1
   - 1.2
   - 1.3
+  - 1.4
+  - 1.5
   - tip

Godeps/_workspace/src/github.com/bkaradzic/go-lz4/README.md

@@ -4,7 +4,7 @@ go-lz4
 go-lz4 is port of LZ4 lossless compression algorithm to Go. The original C code
 is located at:
 
-https://code.google.com/p/lz4/
+https://github.com/Cyan4973/lz4
 
 Status
 ------

Godeps/_workspace/src/github.com/bkaradzic/go-lz4/fuzz.go

@@ -0,0 +1,23 @@
+// +build gofuzz
+
+package lz4
+
+import "encoding/binary"
+
+func Fuzz(data []byte) int {
+
+	if len(data) < 4 {
+		return 0
+	}
+
+	ln := binary.LittleEndian.Uint32(data)
+	if ln > (1 << 21) {
+		return 0
+	}
+
+	if _, err := Decode(nil, data); err != nil {
+		return 0
+	}
+
+	return 1
+}

Godeps/_workspace/src/github.com/bkaradzic/go-lz4/reader.go

@@ -1,194 +1,199 @@
-/*
- * Copyright 2011-2012 Branimir Karadzic. All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without modification,
- * are permitted provided that the following conditions are met:
- *
- *    1. Redistributions of source code must retain the above copyright notice, this
- *       list of conditions and the following disclaimer.
- *
- *    2. Redistributions in binary form must reproduce the above copyright notice,
- *       this list of conditions and the following disclaimer in the documentation
- *       and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDER ``AS IS'' AND ANY EXPRESS OR
- * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
- * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
- * SHALL COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
- * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
- * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
- * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
- * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
- * THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-package lz4
-
-import (
-	"encoding/binary"
-	"errors"
-	"io"
-)
-
-var (
-	// ErrCorrupt indicates the input was corrupt
-	ErrCorrupt = errors.New("corrupt input")
-)
-
-const (
-	mlBits  = 4
-	mlMask  = (1 << mlBits) - 1
-	runBits = 8 - mlBits
-	runMask = (1 << runBits) - 1
-)
-
-type decoder struct {
-	src  []byte
-	dst  []byte
-	spos uint32
-	dpos uint32
-	ref  uint32
-}
-
-func (d *decoder) readByte() (uint8, error) {
-	if int(d.spos) == len(d.src) {
-		return 0, io.EOF
-	}
-	b := d.src[d.spos]
-	d.spos++
-	return b, nil
-}
-
-func (d *decoder) getLen() (uint32, error) {
-
-	length := uint32(0)
-	ln, err := d.readByte()
-	if err != nil {
-		return 0, ErrCorrupt
-	}
-	for ln == 255 {
-		length += 255
-		ln, err = d.readByte()
-		if err != nil {
-			return 0, ErrCorrupt
-		}
-	}
-	length += uint32(ln)
-
-	return length, nil
-}
-
-func (d *decoder) cp(length, decr uint32) {
-
-	if int(d.ref+length) < int(d.dpos) {
-		copy(d.dst[d.dpos:], d.dst[d.ref:d.ref+length])
-	} else {
-		for ii := uint32(0); ii < length; ii++ {
-			d.dst[d.dpos+ii] = d.dst[d.ref+ii]
-		}
-	}
-	d.dpos += length
-	d.ref += length - decr
-}
-
-func (d *decoder) finish(err error) error {
-	if err == io.EOF {
-		return nil
-	}
-
-	return err
-}
-
-// Decode returns the decoded form of src.  The returned slice may be a
-// subslice of dst if it was large enough to hold the entire decoded block.
-func Decode(dst, src []byte) ([]byte, error) {
-
-	if len(src) < 4 {
-		return nil, ErrCorrupt
-	}
-
-	uncompressedLen := binary.LittleEndian.Uint32(src)
-
-	if uncompressedLen == 0 {
-		return nil, nil
-	}
-
-	if uncompressedLen > MaxInputSize {
-		return nil, ErrTooLarge
-	}
-
-	if dst == nil || len(dst) < int(uncompressedLen) {
-		dst = make([]byte, uncompressedLen)
-	}
-
-	d := decoder{src: src, dst: dst[:uncompressedLen], spos: 4}
-
-	decr := []uint32{0, 3, 2, 3}
-
-	for {
-		code, err := d.readByte()
-		if err != nil {
-			return d.dst, d.finish(err)
-		}
-
-		length := uint32(code >> mlBits)
-		if length == runMask {
-			ln, err := d.getLen()
-			if err != nil {
-				return nil, ErrCorrupt
-			}
-			length += ln
-		}
-
-		if int(d.spos+length) > len(d.src) {
-			return nil, ErrCorrupt
-		}
-
-		for ii := uint32(0); ii < length; ii++ {
-			d.dst[d.dpos+ii] = d.src[d.spos+ii]
-		}
-
-		d.spos += length
-		d.dpos += length
-
-		if int(d.spos) == len(d.src) {
-			return d.dst, nil
-		}
-
-		if int(d.spos+2) >= len(d.src) {
-			return nil, ErrCorrupt
-		}
-
-		back := uint32(d.src[d.spos]) | uint32(d.src[d.spos+1])<<8
-
-		if back > d.dpos {
-			return nil, ErrCorrupt
-		}
-
-		d.spos += 2
-		d.ref = d.dpos - back
-
-		length = uint32(code & mlMask)
-		if length == mlMask {
-			ln, err := d.getLen()
-			if err != nil {
-				return nil, ErrCorrupt
-			}
-			length += ln
-		}
-
-		literal := d.dpos - d.ref
-		if literal < 4 {
-			d.cp(4, decr[literal])
-		} else {
-			length += 4
-		}
-
-		if d.dpos+length > uncompressedLen {
-			return nil, ErrCorrupt
-		}
-
-		d.cp(length, 0)
-	}
-}
+/*
+ * Copyright 2011-2012 Branimir Karadzic. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without modification,
+ * are permitted provided that the following conditions are met:
+ *
+ *    1. Redistributions of source code must retain the above copyright notice, this
+ *       list of conditions and the following disclaimer.
+ *
+ *    2. Redistributions in binary form must reproduce the above copyright notice,
+ *       this list of conditions and the following disclaimer in the documentation
+ *       and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDER ``AS IS'' AND ANY EXPRESS OR
+ * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
+ * SHALL COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+ * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+ * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package lz4
+
+import (
+	"encoding/binary"
+	"errors"
+	"io"
+)
+
+var (
+	// ErrCorrupt indicates the input was corrupt
+	ErrCorrupt = errors.New("corrupt input")
+)
+
+const (
+	mlBits  = 4
+	mlMask  = (1 << mlBits) - 1
+	runBits = 8 - mlBits
+	runMask = (1 << runBits) - 1
+)
+
+type decoder struct {
+	src  []byte
+	dst  []byte
+	spos uint32
+	dpos uint32
+	ref  uint32
+}
+
+func (d *decoder) readByte() (uint8, error) {
+	if int(d.spos) == len(d.src) {
+		return 0, io.EOF
+	}
+	b := d.src[d.spos]
+	d.spos++
+	return b, nil
+}
+
+func (d *decoder) getLen() (uint32, error) {
+
+	length := uint32(0)
+	ln, err := d.readByte()
+	if err != nil {
+		return 0, ErrCorrupt
+	}
+	for ln == 255 {
+		length += 255
+		ln, err = d.readByte()
+		if err != nil {
+			return 0, ErrCorrupt
+		}
+	}
+	length += uint32(ln)
+
+	return length, nil
+}
+
+func (d *decoder) cp(length, decr uint32) {
+
+	if int(d.ref+length) < int(d.dpos) {
+		copy(d.dst[d.dpos:], d.dst[d.ref:d.ref+length])
+	} else {
+		for ii := uint32(0); ii < length; ii++ {
+			d.dst[d.dpos+ii] = d.dst[d.ref+ii]
+		}
+	}
+	d.dpos += length
+	d.ref += length - decr
+}
+
+func (d *decoder) finish(err error) error {
+	if err == io.EOF {
+		return nil
+	}
+
+	return err
+}
+
+// Decode returns the decoded form of src.  The returned slice may be a
+// subslice of dst if it was large enough to hold the entire decoded block.
+func Decode(dst, src []byte) ([]byte, error) {
+
+	if len(src) < 4 {
+		return nil, ErrCorrupt
+	}
+
+	uncompressedLen := binary.LittleEndian.Uint32(src)
+
+	if uncompressedLen == 0 {
+		return nil, nil
+	}
+
+	if uncompressedLen > MaxInputSize {
+		return nil, ErrTooLarge
+	}
+
+	if dst == nil || len(dst) < int(uncompressedLen) {
+		dst = make([]byte, uncompressedLen)
+	}
+
+	d := decoder{src: src, dst: dst[:uncompressedLen], spos: 4}
+
+	decr := []uint32{0, 3, 2, 3}
+
+	for {
+		code, err := d.readByte()
+		if err != nil {
+			return d.dst, d.finish(err)
+		}
+
+		length := uint32(code >> mlBits)
+		if length == runMask {
+			ln, err := d.getLen()
+			if err != nil {
+				return nil, ErrCorrupt
+			}
+			length += ln
+		}
+
+		if int(d.spos+length) > len(d.src) || int(d.dpos+length) > len(d.dst) {
+			return nil, ErrCorrupt
+		}
+
+		for ii := uint32(0); ii < length; ii++ {
+			d.dst[d.dpos+ii] = d.src[d.spos+ii]
+		}
+
+		d.spos += length
+		d.dpos += length
+
+		if int(d.spos) == len(d.src) {
+			return d.dst, nil
+		}
+
+		if int(d.spos+2) >= len(d.src) {
+			return nil, ErrCorrupt
+		}
+
+		back := uint32(d.src[d.spos]) | uint32(d.src[d.spos+1])<<8
+
+		if back > d.dpos {
+			return nil, ErrCorrupt
+		}
+
+		d.spos += 2
+		d.ref = d.dpos - back
+
+		length = uint32(code & mlMask)
+		if length == mlMask {
+			ln, err := d.getLen()
+			if err != nil {
+				return nil, ErrCorrupt
+			}
+			length += ln
+		}
+
+		literal := d.dpos - d.ref
+
+		if literal < 4 {
+			if int(d.dpos+4) > len(d.dst) {
+				return nil, ErrCorrupt
+			}
+
+			d.cp(4, decr[literal])
+		} else {
+			length += 4
+		}
+
+		if d.dpos+length > uncompressedLen {
+			return nil, ErrCorrupt
+		}
+
+		d.cp(length, 0)
+	}
+}

Godeps/_workspace/src/github.com/bkaradzic/go-lz4/writer.go

@@ -1,188 +1,190 @@
-/*
- * Copyright 2011-2012 Branimir Karadzic. All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without modification,
- * are permitted provided that the following conditions are met:
- *
- *    1. Redistributions of source code must retain the above copyright notice, this
- *       list of conditions and the following disclaimer.
- *
- *    2. Redistributions in binary form must reproduce the above copyright notice,
- *       this list of conditions and the following disclaimer in the documentation
- *       and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDER ``AS IS'' AND ANY EXPRESS OR
- * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
- * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
- * SHALL COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
- * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
- * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
- * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
- * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
- * THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-package lz4
-
-import "encoding/binary"
-import "errors"
-
-const (
-	minMatch              = 4
-	hashLog               = 17
-	hashTableSize         = 1 << hashLog
-	hashShift             = (minMatch * 8) - hashLog
-	incompressible uint32 = 128
-	uninitHash            = 0x88888888
-
-	// MaxInputSize is the largest buffer than can be compressed in a single block
-	MaxInputSize = 0x7E000000
-)
-
-var (
-	// ErrTooLarge indicates the input buffer was too large
-	ErrTooLarge = errors.New("input too large")
-)
-
-type encoder struct {
-	src       []byte
-	dst       []byte
-	hashTable []uint32
-	pos       uint32
-	anchor    uint32
-	dpos      uint32
-}
-
-// CompressBound returns the maximum length of a lz4 block, given it's uncompressed length
-func CompressBound(isize int) int {
-	if isize > MaxInputSize {
-		return 0
-	}
-	return isize + ((isize) / 255) + 16 + 4
-}
-
-func (e *encoder) writeLiterals(length, mlLen, pos uint32) {
-
-	ln := length
-
-	var code byte
-	if ln > runMask-1 {
-		code = runMask
-	} else {
-		code = byte(ln)
-	}
-
-	if mlLen > mlMask-1 {
-		e.dst[e.dpos] = (code << mlBits) + byte(mlMask)
-	} else {
-		e.dst[e.dpos] = (code << mlBits) + byte(mlLen)
-	}
-	e.dpos++
-
-	if code == runMask {
-		ln -= runMask
-		for ; ln > 254; ln -= 255 {
-			e.dst[e.dpos] = 255
-			e.dpos++
-		}
-
-		e.dst[e.dpos] = byte(ln)
-		e.dpos++
-	}
-
-	for ii := uint32(0); ii < length; ii++ {
-		e.dst[e.dpos+ii] = e.src[pos+ii]
-	}
-
-	e.dpos += length
-}
-
-// Encode returns the encoded form of src.  The returned array may be a
-// sub-slice of dst if it was large enough to hold the entire output.
-func Encode(dst, src []byte) ([]byte, error) {
-
-	if len(src) >= MaxInputSize {
-		return nil, ErrTooLarge
-	}
-
-	if n := CompressBound(len(src)); len(dst) < n {
-		dst = make([]byte, n)
-	}
-
-	e := encoder{src: src, dst: dst, hashTable: make([]uint32, hashTableSize)}
-
-	binary.LittleEndian.PutUint32(dst, uint32(len(src)))
-	e.dpos = 4
-
-	var (
-		step  uint32 = 1
-		limit        = incompressible
-	)
-
-	for {
-		if int(e.pos)+12 >= len(e.src) {
-			e.writeLiterals(uint32(len(e.src))-e.anchor, 0, e.anchor)
-			return e.dst[:e.dpos], nil
-		}
-
-		sequence := uint32(e.src[e.pos+3])<<24 | uint32(e.src[e.pos+2])<<16 | uint32(e.src[e.pos+1])<<8 | uint32(e.src[e.pos+0])
-
-		hash := (sequence * 2654435761) >> hashShift
-		ref := e.hashTable[hash] + uninitHash
-		e.hashTable[hash] = e.pos - uninitHash
-
-		if ((e.pos-ref)>>16) != 0 || uint32(e.src[ref+3])<<24|uint32(e.src[ref+2])<<16|uint32(e.src[ref+1])<<8|uint32(e.src[ref+0]) != sequence {
-			if e.pos-e.anchor > limit {
-				limit <<= 1
-				step += 1 + (step >> 2)
-			}
-			e.pos += step
-			continue
-		}
-
-		if step > 1 {
-			e.hashTable[hash] = ref - uninitHash
-			e.pos -= step - 1
-			step = 1
-			continue
-		}
-		limit = incompressible
-
-		ln := e.pos - e.anchor
-		back := e.pos - ref
-
-		anchor := e.anchor
-
-		e.pos += minMatch
-		ref += minMatch
-		e.anchor = e.pos
-
-		for int(e.pos) < len(e.src)-5 && e.src[e.pos] == e.src[ref] {
-			e.pos++
-			ref++
-		}
-
-		mlLen := e.pos - e.anchor
-
-		e.writeLiterals(ln, mlLen, anchor)
-		e.dst[e.dpos] = uint8(back)
-		e.dst[e.dpos+1] = uint8(back >> 8)
-		e.dpos += 2
-
-		if mlLen > mlMask-1 {
-			mlLen -= mlMask
-			for mlLen > 254 {
-				mlLen -= 255
-
-				e.dst[e.dpos] = 255
-				e.dpos++
-			}
-
-			e.dst[e.dpos] = byte(mlLen)
-			e.dpos++
-		}
-
-		e.anchor = e.pos
-	}
-}
+/*
+ * Copyright 2011-2012 Branimir Karadzic. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without modification,
+ * are permitted provided that the following conditions are met:
+ *
+ *    1. Redistributions of source code must retain the above copyright notice, this
+ *       list of conditions and the following disclaimer.
+ *
+ *    2. Redistributions in binary form must reproduce the above copyright notice,
+ *       this list of conditions and the following disclaimer in the documentation
+ *       and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDER ``AS IS'' AND ANY EXPRESS OR
+ * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
+ * SHALL COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+ * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+ * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package lz4
+
+import (
+	"encoding/binary"
+	"errors"
+)
+
+const (
+	minMatch              = 4
+	hashLog               = 17
+	hashTableSize         = 1 << hashLog
+	hashShift             = (minMatch * 8) - hashLog
+	incompressible uint32 = 128
+	uninitHash            = 0x88888888
+
+	// MaxInputSize is the largest buffer than can be compressed in a single block
+	MaxInputSize = 0x7E000000
+)
+
+var (
+	// ErrTooLarge indicates the input buffer was too large
+	ErrTooLarge = errors.New("input too large")
+)
+
+type encoder struct {
+	src       []byte
+	dst       []byte
+	hashTable []uint32
+	pos       uint32
+	anchor    uint32
+	dpos      uint32
+}
+
+// CompressBound returns the maximum length of a lz4 block, given it's uncompressed length
+func CompressBound(isize int) int {
+	if isize > MaxInputSize {
+		return 0
+	}
+	return isize + ((isize) / 255) + 16 + 4
+}
+
+func (e *encoder) writeLiterals(length, mlLen, pos uint32) {
+
+	ln := length
+
+	var code byte
+	if ln > runMask-1 {
+		code = runMask
+	} else {
+		code = byte(ln)
+	}
+
+	if mlLen > mlMask-1 {
+		e.dst[e.dpos] = (code << mlBits) + byte(mlMask)
+	} else {
+		e.dst[e.dpos] = (code << mlBits) + byte(mlLen)
+	}
+	e.dpos++
+
+	if code == runMask {
+		ln -= runMask
+		for ; ln > 254; ln -= 255 {
+			e.dst[e.dpos] = 255
+			e.dpos++
+		}
+
+		e.dst[e.dpos] = byte(ln)
+		e.dpos++
+	}
+
+	for ii := uint32(0); ii < length; ii++ {
+		e.dst[e.dpos+ii] = e.src[pos+ii]
+	}
+
+	e.dpos += length
+}
+
+// Encode returns the encoded form of src.  The returned array may be a
+// sub-slice of dst if it was large enough to hold the entire output.
+func Encode(dst, src []byte) ([]byte, error) {
+
+	if len(src) >= MaxInputSize {
+		return nil, ErrTooLarge
+	}
+
+	if n := CompressBound(len(src)); len(dst) < n {
+		dst = make([]byte, n)
+	}
+
+	e := encoder{src: src, dst: dst, hashTable: make([]uint32, hashTableSize)}
+
+	binary.LittleEndian.PutUint32(dst, uint32(len(src)))
+	e.dpos = 4
+
+	var (
+		step  uint32 = 1
+		limit        = incompressible
+	)
+
+	for {
+		if int(e.pos)+12 >= len(e.src) {
+			e.writeLiterals(uint32(len(e.src))-e.anchor, 0, e.anchor)
+			return e.dst[:e.dpos], nil
+		}
+
+		sequence := uint32(e.src[e.pos+3])<<24 | uint32(e.src[e.pos+2])<<16 | uint32(e.src[e.pos+1])<<8 | uint32(e.src[e.pos+0])
+
+		hash := (sequence * 2654435761) >> hashShift
+		ref := e.hashTable[hash] + uninitHash
+		e.hashTable[hash] = e.pos - uninitHash
+
+		if ((e.pos-ref)>>16) != 0 || uint32(e.src[ref+3])<<24|uint32(e.src[ref+2])<<16|uint32(e.src[ref+1])<<8|uint32(e.src[ref+0]) != sequence {
+			if e.pos-e.anchor > limit {
+				limit <<= 1
+				step += 1 + (step >> 2)
+			}
+			e.pos += step
+			continue
+		}
+
+		if step > 1 {
+			e.hashTable[hash] = ref - uninitHash
+			e.pos -= step - 1
+			step = 1
+			continue
+		}
+		limit = incompressible
+
+		ln := e.pos - e.anchor
+		back := e.pos - ref
+
+		anchor := e.anchor
+
+		e.pos += minMatch
+		ref += minMatch
+		e.anchor = e.pos
+
+		for int(e.pos) < len(e.src)-5 && e.src[e.pos] == e.src[ref] {
+			e.pos++
+			ref++
+		}
+
+		mlLen := e.pos - e.anchor
+
+		e.writeLiterals(ln, mlLen, anchor)
+		e.dst[e.dpos] = uint8(back)
+		e.dst[e.dpos+1] = uint8(back >> 8)
+		e.dpos += 2
+
+		if mlLen > mlMask-1 {
+			mlLen -= mlMask
+			for mlLen > 254 {
+				mlLen -= 255
+
+				e.dst[e.dpos] = 255
+				e.dpos++
+			}
+
+			e.dst[e.dpos] = byte(mlLen)
+			e.dpos++
+		}
+
+		e.anchor = e.pos
+	}
+}

Godeps/_workspace/src/github.com/calmh/du/LICENSE

@@ -0,0 +1,24 @@
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any
+means.
+
+In jurisdictions that recognize copyright laws, the author or authors
+of this software dedicate any and all copyright interest in the
+software to the public domain. We make this dedication for the benefit
+of the public at large and to the detriment of our heirs and
+successors. We intend this dedication to be an overt act of
+relinquishment in perpetuity of all present and future rights to this
+software under copyright law.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+For more information, please refer to <http://unlicense.org>

Godeps/_workspace/src/github.com/calmh/du/README.md

@@ -0,0 +1,14 @@
+du
+==
+
+Get total and available disk space on a given volume.
+
+Documentation
+-------------
+
+http://godoc.org/github.com/calmh/du
+
+License
+-------
+
+Public Domain

Godeps/_workspace/src/github.com/calmh/du/cmd/du/main.go

@@ -0,0 +1,21 @@
+package main
+
+import (
+	"fmt"
+	"log"
+	"os"
+
+	"github.com/calmh/du"
+)
+
+var KB = int64(1024)
+
+func main() {
+	usage, err := du.Get(os.Args[1])
+	if err != nil {
+		log.Fatal(err)
+	}
+	fmt.Println("Free:", usage.FreeBytes/(KB*KB), "MiB")
+	fmt.Println("Available:", usage.AvailBytes/(KB*KB), "MiB")
+	fmt.Println("Size:", usage.TotalBytes/(KB*KB), "MiB")
+}

Godeps/_workspace/src/github.com/calmh/du/diskusage.go

@@ -0,0 +1,8 @@
+package du
+
+// Usage holds information about total and available storage on a volume.
+type Usage struct {
+	TotalBytes int64 // Size of volume
+	FreeBytes  int64 // Unused size
+	AvailBytes int64 // Available to a non-privileged user
+}

Godeps/_workspace/src/github.com/calmh/du/diskusage_posix.go

@@ -0,0 +1,24 @@
+// +build !windows,!netbsd,!openbsd,!solaris
+
+package du
+
+import (
+	"path/filepath"
+	"syscall"
+)
+
+// Get returns the Usage of a given path, or an error if usage data is
+// unavailable.
+func Get(path string) (Usage, error) {
+	var stat syscall.Statfs_t
+	err := syscall.Statfs(filepath.Clean(path), &stat)
+	if err != nil {
+		return Usage{}, err
+	}
+	u := Usage{
+		FreeBytes:  int64(stat.Bfree) * int64(stat.Bsize),
+		TotalBytes: int64(stat.Blocks) * int64(stat.Bsize),
+		AvailBytes: int64(stat.Bavail) * int64(stat.Bsize),
+	}
+	return u, nil
+}

Godeps/_workspace/src/github.com/calmh/du/diskusage_unsupported.go

@@ -0,0 +1,13 @@
+// +build netbsd openbsd solaris
+
+package du
+
+import "errors"
+
+var ErrUnsupported = errors.New("unsupported platform")
+
+// Get returns the Usage of a given path, or an error if usage data is
+// unavailable.
+func Get(path string) (Usage, error) {
+	return Usage{}, ErrUnsupported
+}

Godeps/_workspace/src/github.com/calmh/du/diskusage_windows.go

@@ -0,0 +1,27 @@
+package du
+
+import (
+	"syscall"
+	"unsafe"
+)
+
+// Get returns the Usage of a given path, or an error if usage data is
+// unavailable.
+func Get(path string) (Usage, error) {
+	h := syscall.MustLoadDLL("kernel32.dll")
+	c := h.MustFindProc("GetDiskFreeSpaceExW")
+
+	var u Usage
+
+	ret, _, err := c.Call(
+		uintptr(unsafe.Pointer(syscall.StringToUTF16Ptr(path))),
+		uintptr(unsafe.Pointer(&u.FreeBytes)),
+		uintptr(unsafe.Pointer(&u.TotalBytes)),
+		uintptr(unsafe.Pointer(&u.AvailBytes)))
+
+	if ret == 0 {
+		return u, err
+	}
+
+	return u, nil
+}

Godeps/_workspace/src/github.com/calmh/logger/README.md

@@ -1,15 +0,0 @@
-logger
-======
-
-A small wrapper around `log` to provide log levels.
-
-Documentation
--------------
-
-http://godoc.org/github.com/calmh/logger
-
-License
--------
-
-MIT
-

Godeps/_workspace/src/github.com/calmh/logger/logger.go

@@ -1,160 +0,0 @@
-// Copyright (C) 2014 Jakob Borg. All rights reserved. Use of this source code
-// is governed by an MIT-style license that can be found in the LICENSE file.
-
-// Package logger implements a standardized logger with callback functionality
-package logger
-
-import (
-	"fmt"
-	"log"
-	"os"
-	"strings"
-	"sync"
-)
-
-type LogLevel int
-
-const (
-	LevelDebug LogLevel = iota
-	LevelInfo
-	LevelOK
-	LevelWarn
-	LevelFatal
-	NumLevels
-)
-
-// A MessageHandler is called with the log level and message text.
-type MessageHandler func(l LogLevel, msg string)
-
-type Logger struct {
-	logger   *log.Logger
-	handlers [NumLevels][]MessageHandler
-	mut      sync.Mutex
-}
-
-// The default logger logs to standard output with a time prefix.
-var DefaultLogger = New()
-
-func New() *Logger {
-	return &Logger{
-		logger: log.New(os.Stdout, "", log.Ltime),
-	}
-}
-
-// AddHandler registers a new MessageHandler to receive messages with the
-// specified log level or above.
-func (l *Logger) AddHandler(level LogLevel, h MessageHandler) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	l.handlers[level] = append(l.handlers[level], h)
-}
-
-// See log.SetFlags
-func (l *Logger) SetFlags(flag int) {
-	l.logger.SetFlags(flag)
-}
-
-// See log.SetPrefix
-func (l *Logger) SetPrefix(prefix string) {
-	l.logger.SetPrefix(prefix)
-}
-
-func (l *Logger) callHandlers(level LogLevel, s string) {
-	for _, h := range l.handlers[level] {
-		h(level, strings.TrimSpace(s))
-	}
-}
-
-// Debugln logs a line with a DEBUG prefix.
-func (l *Logger) Debugln(vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintln(vals...)
-	l.logger.Output(2, "DEBUG: "+s)
-	l.callHandlers(LevelDebug, s)
-}
-
-// Debugf logs a formatted line with a DEBUG prefix.
-func (l *Logger) Debugf(format string, vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintf(format, vals...)
-	l.logger.Output(2, "DEBUG: "+s)
-	l.callHandlers(LevelDebug, s)
-}
-
-// Infoln logs a line with an INFO prefix.
-func (l *Logger) Infoln(vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintln(vals...)
-	l.logger.Output(2, "INFO: "+s)
-	l.callHandlers(LevelInfo, s)
-}
-
-// Infof logs a formatted line with an INFO prefix.
-func (l *Logger) Infof(format string, vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintf(format, vals...)
-	l.logger.Output(2, "INFO: "+s)
-	l.callHandlers(LevelInfo, s)
-}
-
-// Okln logs a line with an OK prefix.
-func (l *Logger) Okln(vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintln(vals...)
-	l.logger.Output(2, "OK: "+s)
-	l.callHandlers(LevelOK, s)
-}
-
-// Okf logs a formatted line with an OK prefix.
-func (l *Logger) Okf(format string, vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintf(format, vals...)
-	l.logger.Output(2, "OK: "+s)
-	l.callHandlers(LevelOK, s)
-}
-
-// Warnln logs a formatted line with a WARNING prefix.
-func (l *Logger) Warnln(vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintln(vals...)
-	l.logger.Output(2, "WARNING: "+s)
-	l.callHandlers(LevelWarn, s)
-}
-
-// Warnf logs a formatted line with a WARNING prefix.
-func (l *Logger) Warnf(format string, vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintf(format, vals...)
-	l.logger.Output(2, "WARNING: "+s)
-	l.callHandlers(LevelWarn, s)
-}
-
-// Fatalln logs a line with a FATAL prefix and exits the process with exit
-// code 1.
-func (l *Logger) Fatalln(vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintln(vals...)
-	l.logger.Output(2, "FATAL: "+s)
-	l.callHandlers(LevelFatal, s)
-	os.Exit(1)
-}
-
-// Fatalf logs a formatted line with a FATAL prefix and exits the process with
-// exit code 1.
-func (l *Logger) Fatalf(format string, vals ...interface{}) {
-	l.mut.Lock()
-	defer l.mut.Unlock()
-	s := fmt.Sprintf(format, vals...)
-	l.logger.Output(2, "FATAL: "+s)
-	l.callHandlers(LevelFatal, s)
-	os.Exit(1)
-}

Godeps/_workspace/src/github.com/calmh/logger/logger_test.go

@@ -1,58 +0,0 @@
-// Copyright (C) 2014 Jakob Borg. All rights reserved. Use of this source code
-// is governed by an MIT-style license that can be found in the LICENSE file.
-
-package logger
-
-import (
-	"strings"
-	"testing"
-)
-
-func TestAPI(t *testing.T) {
-	l := New()
-	l.SetFlags(0)
-	l.SetPrefix("testing")
-
-	debug := 0
-	l.AddHandler(LevelDebug, checkFunc(t, LevelDebug, "test 0", &debug))
-	info := 0
-	l.AddHandler(LevelInfo, checkFunc(t, LevelInfo, "test 1", &info))
-	warn := 0
-	l.AddHandler(LevelWarn, checkFunc(t, LevelWarn, "test 2", &warn))
-	ok := 0
-	l.AddHandler(LevelOK, checkFunc(t, LevelOK, "test 3", &ok))
-
-	l.Debugf("test %d", 0)
-	l.Debugln("test", 0)
-	l.Infof("test %d", 1)
-	l.Infoln("test", 1)
-	l.Warnf("test %d", 2)
-	l.Warnln("test", 2)
-	l.Okf("test %d", 3)
-	l.Okln("test", 3)
-
-	if debug != 2 {
-		t.Errorf("Debug handler called %d != 2 times", debug)
-	}
-	if info != 2 {
-		t.Errorf("Info handler called %d != 2 times", info)
-	}
-	if warn != 2 {
-		t.Errorf("Warn handler called %d != 2 times", warn)
-	}
-	if ok != 2 {
-		t.Errorf("Ok handler called %d != 2 times", ok)
-	}
-}
-
-func checkFunc(t *testing.T, expectl LogLevel, expectmsg string, counter *int) func(LogLevel, string) {
-	return func(l LogLevel, msg string) {
-		*counter++
-		if l != expectl {
-			t.Errorf("Incorrect message level %d != %d", l, expectl)
-		}
-		if !strings.HasSuffix(msg, expectmsg) {
-			t.Errorf("%q does not end with %q", msg, expectmsg)
-		}
-	}
-}

Godeps/_workspace/src/github.com/calmh/luhn/LICENSE

@@ -0,0 +1,19 @@
+Copyright (C) 2014 Jakob Borg
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+- The above copyright notice and this permission notice shall be included in
+  all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Godeps/_workspace/src/github.com/calmh/luhn/luhn.go

@@ -1,17 +1,4 @@
-// Copyright (C) 2014 The Syncthing Authors.
-//
-// This program is free software: you can redistribute it and/or modify it
-// under the terms of the GNU General Public License as published by the Free
-// Software Foundation, either version 3 of the License, or (at your option)
-// any later version.
-//
-// This program is distributed in the hope that it will be useful, but WITHOUT
-// ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
-// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
-// more details.
-//
-// You should have received a copy of the GNU General Public License along
-// with this program. If not, see <http://www.gnu.org/licenses/>.
+// Copyright (C) 2014 Jakob Borg
 
 // Package luhn generates and validates Luhn mod N check digits.
 package luhn

Godeps/_workspace/src/github.com/calmh/luhn/luhn_test.go

@@ -1,24 +1,11 @@
-// Copyright (C) 2014 The Syncthing Authors.
-//
-// This program is free software: you can redistribute it and/or modify it
-// under the terms of the GNU General Public License as published by the Free
-// Software Foundation, either version 3 of the License, or (at your option)
-// any later version.
-//
-// This program is distributed in the hope that it will be useful, but WITHOUT
-// ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
-// FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
-// more details.
-//
-// You should have received a copy of the GNU General Public License along
-// with this program. If not, see <http://www.gnu.org/licenses/>.
+// Copyright (C) 2014 Jakob Borg
 
 package luhn_test
 
 import (
 	"testing"
 
-	"github.com/syncthing/syncthing/internal/luhn"
+	"github.com/calmh/luhn"
 )
 
 func TestGenerate(t *testing.T) {

Godeps/_workspace/src/github.com/calmh/osext/LICENSE

@@ -1,20 +0,0 @@
-Copyright (c) 2012 Daniel Theophanes
-
-This software is provided 'as-is', without any express or implied
-warranty. In no event will the authors be held liable for any damages
-arising from the use of this software.
-
-Permission is granted to anyone to use this software for any purpose,
-including commercial applications, and to alter it and redistribute it
-freely, subject to the following restrictions:
-
-   1. The origin of this software must not be misrepresented; you must not
-   claim that you wrote the original software. If you use this software
-   in a product, an acknowledgment in the product documentation would be
-   appreciated but is not required.
-
-   2. Altered source versions must be plainly marked as such, and must not be
-   misrepresented as being the original software.
-
-   3. This notice may not be removed or altered from any source
-   distribution.

Godeps/_workspace/src/github.com/calmh/osext/osext_test.go

@@ -1,79 +0,0 @@
-// Copyright 2012 The Go Authors. All rights reserved.
-// Use of this source code is governed by a BSD-style
-// license that can be found in the LICENSE file.
-
-// +build darwin linux freebsd netbsd windows
-
-package osext
-
-import (
-	"fmt"
-	"os"
-	oexec "os/exec"
-	"path/filepath"
-	"runtime"
-	"testing"
-)
-
-const execPath_EnvVar = "OSTEST_OUTPUT_EXECPATH"
-
-func TestExecPath(t *testing.T) {
-	ep, err := Executable()
-	if err != nil {
-		t.Fatalf("ExecPath failed: %v", err)
-	}
-	// we want fn to be of the form "dir/prog"
-	dir := filepath.Dir(filepath.Dir(ep))
-	fn, err := filepath.Rel(dir, ep)
-	if err != nil {
-		t.Fatalf("filepath.Rel: %v", err)
-	}
-	cmd := &oexec.Cmd{}
-	// make child start with a relative program path
-	cmd.Dir = dir
-	cmd.Path = fn
-	// forge argv[0] for child, so that we can verify we could correctly
-	// get real path of the executable without influenced by argv[0].
-	cmd.Args = []string{"-", "-test.run=XXXX"}
-	cmd.Env = []string{fmt.Sprintf("%s=1", execPath_EnvVar)}
-	out, err := cmd.CombinedOutput()
-	if err != nil {
-		t.Fatalf("exec(self) failed: %v", err)
-	}
-	outs := string(out)
-	if !filepath.IsAbs(outs) {
-		t.Fatalf("Child returned %q, want an absolute path", out)
-	}
-	if !sameFile(outs, ep) {
-		t.Fatalf("Child returned %q, not the same file as %q", out, ep)
-	}
-}
-
-func sameFile(fn1, fn2 string) bool {
-	fi1, err := os.Stat(fn1)
-	if err != nil {
-		return false
-	}
-	fi2, err := os.Stat(fn2)
-	if err != nil {
-		return false
-	}
-	return os.SameFile(fi1, fi2)
-}
-
-func init() {
-	if e := os.Getenv(execPath_EnvVar); e != "" {
-		// first chdir to another path
-		dir := "/"
-		if runtime.GOOS == "windows" {
-			dir = filepath.VolumeName(".")
-		}
-		os.Chdir(dir)
-		if ep, err := Executable(); err != nil {
-			fmt.Fprint(os.Stderr, "ERROR: ", err)
-		} else {
-			fmt.Fprint(os.Stderr, ep)
-		}
-		os.Exit(0)
-	}
-}

Godeps/_workspace/src/github.com/calmh/xdr/.travis.yml

@@ -4,7 +4,7 @@ go:
 
 install:
 - export PATH=$PATH:$HOME/gopath/bin
-- go get code.google.com/p/go.tools/cmd/cover
+- go get golang.org/x/tools/cover
 - go get github.com/mattn/goveralls
 
 script:

Godeps/_workspace/src/github.com/calmh/xdr/README.md

@@ -1,7 +1,7 @@
 xdr
 ===
 
-[![Build Status](https://img.shields.io/travis/calmh/xdr.svg?style=flat)](https://travis-ci.org/calmh/xdr)
+[![Build Status](https://img.shields.io/circleci/project/calmh/xdr.svg?style=flat-square)](https://circleci.com/gh/calmh/xdr)
 [![Coverage Status](https://img.shields.io/coveralls/calmh/xdr.svg?style=flat)](https://coveralls.io/r/calmh/xdr?branch=master)
 [![API Documentation](http://img.shields.io/badge/api-Godoc-blue.svg?style=flat)](http://godoc.org/github.com/calmh/xdr)
 [![MIT License](http://img.shields.io/badge/license-MIT-blue.svg?style=flat)](http://opensource.org/licenses/MIT)

Godeps/_workspace/src/github.com/calmh/xdr/bench_test.go

@@ -67,7 +67,7 @@ func BenchmarkThisEncode(b *testing.B) {
 func BenchmarkThisEncoder(b *testing.B) {
 	w := xdr.NewWriter(ioutil.Discard)
 	for i := 0; i < b.N; i++ {
-		_, err := s.encodeXDR(w)
+		_, err := s.EncodeXDRInto(w)
 		if err != nil {
 			b.Fatal(err)
 		}
@@ -108,7 +108,7 @@ func BenchmarkThisDecoder(b *testing.B) {
 	r := xdr.NewReader(rr)
 	var t XDRBenchStruct
 	for i := 0; i < b.N; i++ {
-		err := t.decodeXDR(r)
+		err := t.DecodeXDRFrom(r)
 		if err != nil {
 			b.Fatal(err)
 		}

Godeps/_workspace/src/github.com/calmh/xdr/bench_xdr_test.go

@@ -26,7 +26,9 @@ XDRBenchStruct Structure:
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |            0x0000             |              I3               |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-|                             uint8                             |
+/                                                               /
+\                        uint8 Structure                        \
+/                                                               /
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                         Length of Bs0                         |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
@@ -69,7 +71,7 @@ struct XDRBenchStruct {
 
 func (o XDRBenchStruct) EncodeXDR(w io.Writer) (int, error) {
 	var xw = xdr.NewWriter(w)
-	return o.encodeXDR(xw)
+	return o.EncodeXDRInto(xw)
 }
 
 func (o XDRBenchStruct) MarshalXDR() ([]byte, error) {
@@ -87,11 +89,11 @@ func (o XDRBenchStruct) MustMarshalXDR() []byte {
 func (o XDRBenchStruct) AppendXDR(bs []byte) ([]byte, error) {
 	var aw = xdr.AppendWriter(bs)
 	var xw = xdr.NewWriter(&aw)
-	_, err := o.encodeXDR(xw)
+	_, err := o.EncodeXDRInto(xw)
 	return []byte(aw), err
 }
 
-func (o XDRBenchStruct) encodeXDR(xw *xdr.Writer) (int, error) {
+func (o XDRBenchStruct) EncodeXDRInto(xw *xdr.Writer) (int, error) {
 	xw.WriteUint64(o.I1)
 	xw.WriteUint32(o.I2)
 	xw.WriteUint16(o.I3)
@@ -111,16 +113,16 @@ func (o XDRBenchStruct) encodeXDR(xw *xdr.Writer) (int, error) {
 
 func (o *XDRBenchStruct) DecodeXDR(r io.Reader) error {
 	xr := xdr.NewReader(r)
-	return o.decodeXDR(xr)
+	return o.DecodeXDRFrom(xr)
 }
 
 func (o *XDRBenchStruct) UnmarshalXDR(bs []byte) error {
 	var br = bytes.NewReader(bs)
 	var xr = xdr.NewReader(br)
-	return o.decodeXDR(xr)
+	return o.DecodeXDRFrom(xr)
 }
 
-func (o *XDRBenchStruct) decodeXDR(xr *xdr.Reader) error {
+func (o *XDRBenchStruct) DecodeXDRFrom(xr *xdr.Reader) error {
 	o.I1 = xr.ReadUint64()
 	o.I2 = xr.ReadUint32()
 	o.I3 = xr.ReadUint16()
@@ -155,7 +157,7 @@ struct repeatReader {
 
 func (o repeatReader) EncodeXDR(w io.Writer) (int, error) {
 	var xw = xdr.NewWriter(w)
-	return o.encodeXDR(xw)
+	return o.EncodeXDRInto(xw)
 }
 
 func (o repeatReader) MarshalXDR() ([]byte, error) {
@@ -173,27 +175,27 @@ func (o repeatReader) MustMarshalXDR() []byte {
 func (o repeatReader) AppendXDR(bs []byte) ([]byte, error) {
 	var aw = xdr.AppendWriter(bs)
 	var xw = xdr.NewWriter(&aw)
-	_, err := o.encodeXDR(xw)
+	_, err := o.EncodeXDRInto(xw)
 	return []byte(aw), err
 }
 
-func (o repeatReader) encodeXDR(xw *xdr.Writer) (int, error) {
+func (o repeatReader) EncodeXDRInto(xw *xdr.Writer) (int, error) {
 	xw.WriteBytes(o.data)
 	return xw.Tot(), xw.Error()
 }
 
 func (o *repeatReader) DecodeXDR(r io.Reader) error {
 	xr := xdr.NewReader(r)
-	return o.decodeXDR(xr)
+	return o.DecodeXDRFrom(xr)
 }
 
 func (o *repeatReader) UnmarshalXDR(bs []byte) error {
 	var br = bytes.NewReader(bs)
 	var xr = xdr.NewReader(br)
-	return o.decodeXDR(xr)
+	return o.DecodeXDRFrom(xr)
 }
 
-func (o *repeatReader) decodeXDR(xr *xdr.Reader) error {
+func (o *repeatReader) DecodeXDRFrom(xr *xdr.Reader) error {
 	o.data = xr.ReadBytes()
 	return xr.Error()
 }

Godeps/_workspace/src/github.com/calmh/xdr/cmd/genxdr/main.go

@@ -28,6 +28,7 @@ type fieldInfo struct {
 	Encoder   string // the encoder name, i.e. "Uint64" for Read/WriteUint64
 	Convert   string // what to convert to when encoding, i.e. "uint64"
 	Max       int    // max size for slices and strings
+	Submax    int    // max size for strings inside slices
 }
 
 type structInfo struct {
@@ -52,7 +53,7 @@ import (
 var encodeTpl = template.Must(template.New("encoder").Parse(`
 func (o {{.TypeName}}) EncodeXDR(w io.Writer) (int, error) {
 	var xw = xdr.NewWriter(w)
-	return o.encodeXDR(xw)
+	return o.EncodeXDRInto(xw)
 }//+n
 
 func (o {{.TypeName}}) MarshalXDR() ([]byte, error) {
@@ -70,11 +71,11 @@ func (o {{.TypeName}}) MustMarshalXDR() []byte {
 func (o {{.TypeName}}) AppendXDR(bs []byte) ([]byte, error) {
 	var aw = xdr.AppendWriter(bs)
 	var xw = xdr.NewWriter(&aw)
-	_, err := o.encodeXDR(xw)
+	_, err := o.EncodeXDRInto(xw)
 	return []byte(aw), err
 }//+n
 
-func (o {{.TypeName}}) encodeXDR(xw *xdr.Writer) (int, error) {
+func (o {{.TypeName}}) EncodeXDRInto(xw *xdr.Writer) (int, error) {
 	{{range $fieldInfo := .Fields}}
 		{{if not $fieldInfo.IsSlice}}
 			{{if ne $fieldInfo.Convert ""}}
@@ -87,7 +88,7 @@ func (o {{.TypeName}}) encodeXDR(xw *xdr.Writer) (int, error) {
 				{{end}}
 				xw.Write{{$fieldInfo.Encoder}}(o.{{$fieldInfo.Name}})
 			{{else}}
-				_, err := o.{{$fieldInfo.Name}}.encodeXDR(xw)
+				_, err := o.{{$fieldInfo.Name}}.EncodeXDRInto(xw)
 				if err != nil {
 					return xw.Tot(), err
 				}
@@ -105,7 +106,7 @@ func (o {{.TypeName}}) encodeXDR(xw *xdr.Writer) (int, error) {
 			{{else if $fieldInfo.IsBasic}}
 				xw.Write{{$fieldInfo.Encoder}}(o.{{$fieldInfo.Name}}[i])
 			{{else}}
-				_, err := o.{{$fieldInfo.Name}}[i].encodeXDR(xw)
+				_, err := o.{{$fieldInfo.Name}}[i].EncodeXDRInto(xw)
 				if err != nil {
 					return xw.Tot(), err
 				}
@@ -118,16 +119,16 @@ func (o {{.TypeName}}) encodeXDR(xw *xdr.Writer) (int, error) {
 
 func (o *{{.TypeName}}) DecodeXDR(r io.Reader) error {
 	xr := xdr.NewReader(r)
-	return o.decodeXDR(xr)
+	return o.DecodeXDRFrom(xr)
 }//+n
 
 func (o *{{.TypeName}}) UnmarshalXDR(bs []byte) error {
 	var br = bytes.NewReader(bs)
 	var xr = xdr.NewReader(br)
-	return o.decodeXDR(xr)
+	return o.DecodeXDRFrom(xr)
 }//+n
 
-func (o *{{.TypeName}}) decodeXDR(xr *xdr.Reader) error {
+func (o *{{.TypeName}}) DecodeXDRFrom(xr *xdr.Reader) error {
 	{{range $fieldInfo := .Fields}}
 		{{if not $fieldInfo.IsSlice}}
 			{{if ne $fieldInfo.Convert ""}}
@@ -139,10 +140,13 @@ func (o *{{.TypeName}}) decodeXDR(xr *xdr.Reader) error {
 					o.{{$fieldInfo.Name}} = xr.Read{{$fieldInfo.Encoder}}()
 				{{end}}
 			{{else}}
-				(&o.{{$fieldInfo.Name}}).decodeXDR(xr)
+				(&o.{{$fieldInfo.Name}}).DecodeXDRFrom(xr)
 			{{end}}
 		{{else}}
 			_{{$fieldInfo.Name}}Size := int(xr.ReadUint32())
+			if _{{$fieldInfo.Name}}Size < 0 {
+				return xdr.ElementSizeExceeded("{{$fieldInfo.Name}}", _{{$fieldInfo.Name}}Size, {{$fieldInfo.Max}})
+			}
 			{{if ge $fieldInfo.Max 1}}
 				if _{{$fieldInfo.Name}}Size > {{$fieldInfo.Max}} {
 					return xdr.ElementSizeExceeded("{{$fieldInfo.Name}}", _{{$fieldInfo.Name}}Size, {{$fieldInfo.Max}})
@@ -153,9 +157,13 @@ func (o *{{.TypeName}}) decodeXDR(xr *xdr.Reader) error {
 				{{if ne $fieldInfo.Convert ""}}
 					o.{{$fieldInfo.Name}}[i] = {{$fieldInfo.FieldType}}(xr.Read{{$fieldInfo.Encoder}}())
 				{{else if $fieldInfo.IsBasic}}
-					o.{{$fieldInfo.Name}}[i] = xr.Read{{$fieldInfo.Encoder}}()
+					{{if ge $fieldInfo.Submax 1}}
+						o.{{$fieldInfo.Name}}[i] = xr.Read{{$fieldInfo.Encoder}}Max({{$fieldInfo.Submax}})
+					{{else}}
+						o.{{$fieldInfo.Name}}[i] = xr.Read{{$fieldInfo.Encoder}}()
+					{{end}}
 				{{else}}
-					(&o.{{$fieldInfo.Name}}[i]).decodeXDR(xr)
+					(&o.{{$fieldInfo.Name}}[i]).DecodeXDRFrom(xr)
 				{{end}}
 			}
 		{{end}}
@@ -163,7 +171,7 @@ func (o *{{.TypeName}}) decodeXDR(xr *xdr.Reader) error {
 	return xr.Error()
 }`))
 
-var maxRe = regexp.MustCompile(`\Wmax:(\d+)`)
+var maxRe = regexp.MustCompile(`(?:\Wmax:)(\d+)(?:\s*,\s*(\d+))?`)
 
 type typeSet struct {
 	Type    string
@@ -195,11 +203,15 @@ func handleStruct(t *ast.StructType) []fieldInfo {
 		}
 
 		fn := sf.Names[0].Name
-		var max = 0
+		var max1, max2 int
 		if sf.Comment != nil {
 			c := sf.Comment.List[0].Text
-			if m := maxRe.FindStringSubmatch(c); m != nil {
-				max, _ = strconv.Atoi(m[1])
+			m := maxRe.FindStringSubmatch(c)
+			if len(m) >= 2 {
+				max1, _ = strconv.Atoi(m[1])
+			}
+			if len(m) >= 3 {
+				max2, _ = strconv.Atoi(m[2])
 			}
 			if strings.Contains(c, "noencode") {
 				continue
@@ -217,14 +229,16 @@ func handleStruct(t *ast.StructType) []fieldInfo {
 					FieldType: tn,
 					Encoder:   enc.Encoder,
 					Convert:   enc.Type,
-					Max:       max,
+					Max:       max1,
+					Submax:    max2,
 				}
 			} else {
 				f = fieldInfo{
 					Name:      fn,
 					IsBasic:   false,
 					FieldType: tn,
-					Max:       max,
+					Max:       max1,
+					Submax:    max2,
 				}
 			}
 
@@ -242,7 +256,8 @@ func handleStruct(t *ast.StructType) []fieldInfo {
 					FieldType: tn,
 					Encoder:   enc.Encoder,
 					Convert:   enc.Type,
-					Max:       max,
+					Max:       max1,
+					Submax:    max2,
 				}
 			} else if enc, ok := xdrEncoders[tn]; ok {
 				f = fieldInfo{
@@ -252,17 +267,26 @@ func handleStruct(t *ast.StructType) []fieldInfo {
 					FieldType: tn,
 					Encoder:   enc.Encoder,
 					Convert:   enc.Type,
-					Max:       max,
+					Max:       max1,
+					Submax:    max2,
 				}
 			} else {
 				f = fieldInfo{
 					Name:      fn,
-					IsBasic:   false,
 					IsSlice:   true,
 					FieldType: tn,
-					Max:       max,
+					Max:       max1,
+					Submax:    max2,
 				}
 			}
+
+		case *ast.SelectorExpr:
+			f = fieldInfo{
+				Name:      fn,
+				FieldType: ft.Sel.Name,
+				Max:       max1,
+				Submax:    max2,
+			}
 		}
 
 		fs = append(fs, f)
@@ -310,10 +334,9 @@ func generateDiagram(output io.Writer, s structInfo) {
 
 	for _, f := range fs {
 		tn := f.FieldType
-		sl := f.IsSlice
 		name := uncamelize(f.Name)
 
-		if sl {
+		if f.IsSlice {
 			fmt.Fprintf(output, "| %s |\n", center("Number of "+name, 61))
 			fmt.Fprintln(output, line)
 		}
@@ -321,10 +344,10 @@ func generateDiagram(output io.Writer, s structInfo) {
 		case "bool":
 			fmt.Fprintf(output, "| %s |V|\n", center(name+" (V=0 or 1)", 59))
 			fmt.Fprintln(output, line)
-		case "uint16":
+		case "int16", "uint16":
 			fmt.Fprintf(output, "| %s | %s |\n", center("0x0000", 29), center(name, 29))
 			fmt.Fprintln(output, line)
-		case "uint32":
+		case "int32", "uint32":
 			fmt.Fprintf(output, "| %s |\n", center(name, 61))
 			fmt.Fprintln(output, line)
 		case "int64", "uint64":
@@ -340,13 +363,16 @@ func generateDiagram(output io.Writer, s structInfo) {
 			fmt.Fprintf(output, "/ %61s /\n", "")
 			fmt.Fprintln(output, line)
 		default:
-			if sl {
+			if f.IsSlice {
 				tn = "Zero or more " + tn + " Structures"
 				fmt.Fprintf(output, "/ %s /\n", center("", 61))
 				fmt.Fprintf(output, "\\ %s \\\n", center(tn, 61))
 				fmt.Fprintf(output, "/ %s /\n", center("", 61))
 			} else {
-				fmt.Fprintf(output, "| %s |\n", center(tn, 61))
+				tn = tn + " Structure"
+				fmt.Fprintf(output, "/ %s /\n", center("", 61))
+				fmt.Fprintf(output, "\\ %s \\\n", center(tn, 61))
+				fmt.Fprintf(output, "/ %s /\n", center("", 61))
 			}
 			fmt.Fprintln(output, line)
 		}
@@ -374,6 +400,8 @@ func generateXdr(output io.Writer, s structInfo) {
 		}
 
 		switch tn {
+		case "int16", "int32":
+			fmt.Fprintf(output, "\tint %s%s;\n", fn, suf)
 		case "uint16", "uint32":
 			fmt.Fprintf(output, "\tunsigned int %s%s;\n", fn, suf)
 		case "int64":

Godeps/_workspace/src/github.com/calmh/xdr/encdec_test.go

@@ -32,11 +32,11 @@ type TestStruct struct {
 
 type Opaque [32]byte
 
-func (u *Opaque) encodeXDR(w *xdr.Writer) (int, error) {
+func (u *Opaque) EncodeXDRInto(w *xdr.Writer) (int, error) {
 	return w.WriteRaw(u[:])
 }
 
-func (u *Opaque) decodeXDR(r *xdr.Reader) (int, error) {
+func (u *Opaque) DecodeXDRFrom(r *xdr.Reader) (int, error) {
 	return r.ReadRaw(u[:])
 }
 

Godeps/_workspace/src/github.com/calmh/xdr/encdec_xdr_test.go

@@ -18,17 +18,23 @@ TestStruct Structure:
  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-|                              int                              |
+/                                                               /
+\                         int Structure                         \
+/                                                               /
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-|                             int8                              |
+/                                                               /
+\                        int8 Structure                         \
+/                                                               /
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-|                             uint8                             |
+/                                                               /
+\                        uint8 Structure                        \
+/                                                               /
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-|                             int16                             |
+|            0x0000             |              I16              |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |            0x0000             |             UI16              |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-|                             int32                             |
+|                              I32                              |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                             UI32                              |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
@@ -52,7 +58,9 @@ TestStruct Structure:
 \                      S (variable length)                      \
 /                                                               /
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-|                            Opaque                             |
+/                                                               /
+\                       Opaque Structure                        \
+/                                                               /
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                         Number of SS                          |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
@@ -68,9 +76,9 @@ struct TestStruct {
 	int I;
 	int8 I8;
 	uint8 UI8;
-	int16 I16;
+	int I16;
 	unsigned int UI16;
-	int32 I32;
+	int I32;
 	unsigned int UI32;
 	hyper I64;
 	unsigned hyper UI64;
@@ -84,7 +92,7 @@ struct TestStruct {
 
 func (o TestStruct) EncodeXDR(w io.Writer) (int, error) {
 	var xw = xdr.NewWriter(w)
-	return o.encodeXDR(xw)
+	return o.EncodeXDRInto(xw)
 }
 
 func (o TestStruct) MarshalXDR() ([]byte, error) {
@@ -102,11 +110,11 @@ func (o TestStruct) MustMarshalXDR() []byte {
 func (o TestStruct) AppendXDR(bs []byte) ([]byte, error) {
 	var aw = xdr.AppendWriter(bs)
 	var xw = xdr.NewWriter(&aw)
-	_, err := o.encodeXDR(xw)
+	_, err := o.EncodeXDRInto(xw)
 	return []byte(aw), err
 }
 
-func (o TestStruct) encodeXDR(xw *xdr.Writer) (int, error) {
+func (o TestStruct) EncodeXDRInto(xw *xdr.Writer) (int, error) {
 	xw.WriteUint64(uint64(o.I))
 	xw.WriteUint8(uint8(o.I8))
 	xw.WriteUint8(o.UI8)
@@ -124,7 +132,7 @@ func (o TestStruct) encodeXDR(xw *xdr.Writer) (int, error) {
 		return xw.Tot(), xdr.ElementSizeExceeded("S", l, 1024)
 	}
 	xw.WriteString(o.S)
-	_, err := o.C.encodeXDR(xw)
+	_, err := o.C.EncodeXDRInto(xw)
 	if err != nil {
 		return xw.Tot(), err
 	}
@@ -140,16 +148,16 @@ func (o TestStruct) encodeXDR(xw *xdr.Writer) (int, error) {
 
 func (o *TestStruct) DecodeXDR(r io.Reader) error {
 	xr := xdr.NewReader(r)
-	return o.decodeXDR(xr)
+	return o.DecodeXDRFrom(xr)
 }
 
 func (o *TestStruct) UnmarshalXDR(bs []byte) error {
 	var br = bytes.NewReader(bs)
 	var xr = xdr.NewReader(br)
-	return o.decodeXDR(xr)
+	return o.DecodeXDRFrom(xr)
 }
 
-func (o *TestStruct) decodeXDR(xr *xdr.Reader) error {
+func (o *TestStruct) DecodeXDRFrom(xr *xdr.Reader) error {
 	o.I = int(xr.ReadUint64())
 	o.I8 = int8(xr.ReadUint8())
 	o.UI8 = xr.ReadUint8()
@@ -161,8 +169,11 @@ func (o *TestStruct) decodeXDR(xr *xdr.Reader) error {
 	o.UI64 = xr.ReadUint64()
 	o.BS = xr.ReadBytesMax(1024)
 	o.S = xr.ReadStringMax(1024)
-	(&o.C).decodeXDR(xr)
+	(&o.C).DecodeXDRFrom(xr)
 	_SSSize := int(xr.ReadUint32())
+	if _SSSize < 0 {
+		return xdr.ElementSizeExceeded("SS", _SSSize, 1024)
+	}
 	if _SSSize > 1024 {
 		return xdr.ElementSizeExceeded("SS", _SSSize, 1024)
 	}

Godeps/_workspace/src/github.com/calmh/xdr/reader.go

@@ -68,7 +68,8 @@ func (r *Reader) ReadBytesMaxInto(max int, dst []byte) []byte {
 	if r.err != nil {
 		return nil
 	}
-	if max > 0 && l > max {
+	if l < 0 || max > 0 && l > max {
+		// l may be negative on 32 bit builds
 		r.err = ElementSizeExceeded("bytes field", l, max)
 		return nil
 	}
@@ -154,6 +155,10 @@ func (e XDRError) Error() string {
 	return "xdr " + e.op + ": " + e.err.Error()
 }
 
+func (e XDRError) IsEOF() bool {
+	return e.err == io.EOF
+}
+
 func (r *Reader) Error() error {
 	if r.err == nil {
 		return nil

Godeps/_workspace/src/github.com/golang/snappy/AUTHORS

@@ -0,0 +1,14 @@
+# This is the official list of Snappy-Go authors for copyright purposes.
+# This file is distinct from the CONTRIBUTORS files.
+# See the latter for an explanation.
+
+# Names should be added to this file as
+#	Name or Organization <email address>
+# The email address is not required for organizations.
+
+# Please keep the list sorted.
+
+Damian Gryski <dgryski@gmail.com>
+Google Inc.
+Jan Mercl <0xjnml@gmail.com>
+Sebastien Binet <seb.binet@gmail.com>

Godeps/_workspace/src/github.com/golang/snappy/CONTRIBUTORS

@@ -0,0 +1,36 @@
+# This is the official list of people who can contribute
+# (and typically have contributed) code to the Snappy-Go repository.
+# The AUTHORS file lists the copyright holders; this file
+# lists people.  For example, Google employees are listed here
+# but not in AUTHORS, because Google holds the copyright.
+#
+# The submission process automatically checks to make sure
+# that people submitting code are listed in this file (by email address).
+#
+# Names should be added to this file only after verifying that
+# the individual or the individual's organization has agreed to
+# the appropriate Contributor License Agreement, found here:
+#
+#     http://code.google.com/legal/individual-cla-v1.0.html
+#     http://code.google.com/legal/corporate-cla-v1.0.html
+#
+# The agreement for individuals can be filled out on the web.
+#
+# When adding J Random Contributor's name to this file,
+# either J's name or J's organization's name should be
+# added to the AUTHORS file, depending on whether the
+# individual or corporate CLA was used.
+
+# Names should be added to this file like so:
+#     Name <email address>
+
+# Please keep the list sorted.
+
+Damian Gryski <dgryski@gmail.com>
+Jan Mercl <0xjnml@gmail.com>
+Kai Backman <kaib@golang.org>
+Marc-Antoine Ruel <maruel@chromium.org>
+Nigel Tao <nigeltao@golang.org>
+Rob Pike <r@golang.org>
+Russ Cox <rsc@golang.org>
+Sebastien Binet <seb.binet@gmail.com>

Godeps/_workspace/src/github.com/golang/snappy/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2011 The Snappy-Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Godeps/_workspace/src/github.com/golang/snappy/README

@@ -0,0 +1,7 @@
+The Snappy compression format in the Go programming language.
+
+To download and install from source:
+$ go get github.com/golang/snappy
+
+Unless otherwise noted, the Snappy-Go source files are distributed
+under the BSD-style license found in the LICENSE file.

Godeps/_workspace/src/github.com/golang/snappy/decode.go

@@ -0,0 +1,294 @@
+// Copyright 2011 The Snappy-Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package snappy
+
+import (
+	"encoding/binary"
+	"errors"
+	"io"
+)
+
+var (
+	// ErrCorrupt reports that the input is invalid.
+	ErrCorrupt = errors.New("snappy: corrupt input")
+	// ErrTooLarge reports that the uncompressed length is too large.
+	ErrTooLarge = errors.New("snappy: decoded block is too large")
+	// ErrUnsupported reports that the input isn't supported.
+	ErrUnsupported = errors.New("snappy: unsupported input")
+)
+
+// DecodedLen returns the length of the decoded block.
+func DecodedLen(src []byte) (int, error) {
+	v, _, err := decodedLen(src)
+	return v, err
+}
+
+// decodedLen returns the length of the decoded block and the number of bytes
+// that the length header occupied.
+func decodedLen(src []byte) (blockLen, headerLen int, err error) {
+	v, n := binary.Uvarint(src)
+	if n <= 0 || v > 0xffffffff {
+		return 0, 0, ErrCorrupt
+	}
+
+	const wordSize = 32 << (^uint(0) >> 32 & 1)
+	if wordSize == 32 && v > 0x7fffffff {
+		return 0, 0, ErrTooLarge
+	}
+	return int(v), n, nil
+}
+
+// Decode returns the decoded form of src. The returned slice may be a sub-
+// slice of dst if dst was large enough to hold the entire decoded block.
+// Otherwise, a newly allocated slice will be returned.
+// It is valid to pass a nil dst.
+func Decode(dst, src []byte) ([]byte, error) {
+	dLen, s, err := decodedLen(src)
+	if err != nil {
+		return nil, err
+	}
+	if len(dst) < dLen {
+		dst = make([]byte, dLen)
+	}
+
+	var d, offset, length int
+	for s < len(src) {
+		switch src[s] & 0x03 {
+		case tagLiteral:
+			x := uint(src[s] >> 2)
+			switch {
+			case x < 60:
+				s++
+			case x == 60:
+				s += 2
+				if s > len(src) {
+					return nil, ErrCorrupt
+				}
+				x = uint(src[s-1])
+			case x == 61:
+				s += 3
+				if s > len(src) {
+					return nil, ErrCorrupt
+				}
+				x = uint(src[s-2]) | uint(src[s-1])<<8
+			case x == 62:
+				s += 4
+				if s > len(src) {
+					return nil, ErrCorrupt
+				}
+				x = uint(src[s-3]) | uint(src[s-2])<<8 | uint(src[s-1])<<16
+			case x == 63:
+				s += 5
+				if s > len(src) {
+					return nil, ErrCorrupt
+				}
+				x = uint(src[s-4]) | uint(src[s-3])<<8 | uint(src[s-2])<<16 | uint(src[s-1])<<24
+			}
+			length = int(x + 1)
+			if length <= 0 {
+				return nil, errors.New("snappy: unsupported literal length")
+			}
+			if length > len(dst)-d || length > len(src)-s {
+				return nil, ErrCorrupt
+			}
+			copy(dst[d:], src[s:s+length])
+			d += length
+			s += length
+			continue
+
+		case tagCopy1:
+			s += 2
+			if s > len(src) {
+				return nil, ErrCorrupt
+			}
+			length = 4 + int(src[s-2])>>2&0x7
+			offset = int(src[s-2])&0xe0<<3 | int(src[s-1])
+
+		case tagCopy2:
+			s += 3
+			if s > len(src) {
+				return nil, ErrCorrupt
+			}
+			length = 1 + int(src[s-3])>>2
+			offset = int(src[s-2]) | int(src[s-1])<<8
+
+		case tagCopy4:
+			return nil, errors.New("snappy: unsupported COPY_4 tag")
+		}
+
+		end := d + length
+		if offset > d || end > len(dst) {
+			return nil, ErrCorrupt
+		}
+		for ; d < end; d++ {
+			dst[d] = dst[d-offset]
+		}
+	}
+	if d != dLen {
+		return nil, ErrCorrupt
+	}
+	return dst[:d], nil
+}
+
+// NewReader returns a new Reader that decompresses from r, using the framing
+// format described at
+// https://github.com/google/snappy/blob/master/framing_format.txt
+func NewReader(r io.Reader) *Reader {
+	return &Reader{
+		r:       r,
+		decoded: make([]byte, maxUncompressedChunkLen),
+		buf:     make([]byte, MaxEncodedLen(maxUncompressedChunkLen)+checksumSize),
+	}
+}
+
+// Reader is an io.Reader than can read Snappy-compressed bytes.
+type Reader struct {
+	r       io.Reader
+	err     error
+	decoded []byte
+	buf     []byte
+	// decoded[i:j] contains decoded bytes that have not yet been passed on.
+	i, j       int
+	readHeader bool
+}
+
+// Reset discards any buffered data, resets all state, and switches the Snappy
+// reader to read from r. This permits reusing a Reader rather than allocating
+// a new one.
+func (r *Reader) Reset(reader io.Reader) {
+	r.r = reader
+	r.err = nil
+	r.i = 0
+	r.j = 0
+	r.readHeader = false
+}
+
+func (r *Reader) readFull(p []byte) (ok bool) {
+	if _, r.err = io.ReadFull(r.r, p); r.err != nil {
+		if r.err == io.ErrUnexpectedEOF {
+			r.err = ErrCorrupt
+		}
+		return false
+	}
+	return true
+}
+
+// Read satisfies the io.Reader interface.
+func (r *Reader) Read(p []byte) (int, error) {
+	if r.err != nil {
+		return 0, r.err
+	}
+	for {
+		if r.i < r.j {
+			n := copy(p, r.decoded[r.i:r.j])
+			r.i += n
+			return n, nil
+		}
+		if !r.readFull(r.buf[:4]) {
+			return 0, r.err
+		}
+		chunkType := r.buf[0]
+		if !r.readHeader {
+			if chunkType != chunkTypeStreamIdentifier {
+				r.err = ErrCorrupt
+				return 0, r.err
+			}
+			r.readHeader = true
+		}
+		chunkLen := int(r.buf[1]) | int(r.buf[2])<<8 | int(r.buf[3])<<16
+		if chunkLen > len(r.buf) {
+			r.err = ErrUnsupported
+			return 0, r.err
+		}
+
+		// The chunk types are specified at
+		// https://github.com/google/snappy/blob/master/framing_format.txt
+		switch chunkType {
+		case chunkTypeCompressedData:
+			// Section 4.2. Compressed data (chunk type 0x00).
+			if chunkLen < checksumSize {
+				r.err = ErrCorrupt
+				return 0, r.err
+			}
+			buf := r.buf[:chunkLen]
+			if !r.readFull(buf) {
+				return 0, r.err
+			}
+			checksum := uint32(buf[0]) | uint32(buf[1])<<8 | uint32(buf[2])<<16 | uint32(buf[3])<<24
+			buf = buf[checksumSize:]
+
+			n, err := DecodedLen(buf)
+			if err != nil {
+				r.err = err
+				return 0, r.err
+			}
+			if n > len(r.decoded) {
+				r.err = ErrCorrupt
+				return 0, r.err
+			}
+			if _, err := Decode(r.decoded, buf); err != nil {
+				r.err = err
+				return 0, r.err
+			}
+			if crc(r.decoded[:n]) != checksum {
+				r.err = ErrCorrupt
+				return 0, r.err
+			}
+			r.i, r.j = 0, n
+			continue
+
+		case chunkTypeUncompressedData:
+			// Section 4.3. Uncompressed data (chunk type 0x01).
+			if chunkLen < checksumSize {
+				r.err = ErrCorrupt
+				return 0, r.err
+			}
+			buf := r.buf[:checksumSize]
+			if !r.readFull(buf) {
+				return 0, r.err
+			}
+			checksum := uint32(buf[0]) | uint32(buf[1])<<8 | uint32(buf[2])<<16 | uint32(buf[3])<<24
+			// Read directly into r.decoded instead of via r.buf.
+			n := chunkLen - checksumSize
+			if !r.readFull(r.decoded[:n]) {
+				return 0, r.err
+			}
+			if crc(r.decoded[:n]) != checksum {
+				r.err = ErrCorrupt
+				return 0, r.err
+			}
+			r.i, r.j = 0, n
+			continue
+
+		case chunkTypeStreamIdentifier:
+			// Section 4.1. Stream identifier (chunk type 0xff).
+			if chunkLen != len(magicBody) {
+				r.err = ErrCorrupt
+				return 0, r.err
+			}
+			if !r.readFull(r.buf[:len(magicBody)]) {
+				return 0, r.err
+			}
+			for i := 0; i < len(magicBody); i++ {
+				if r.buf[i] != magicBody[i] {
+					r.err = ErrCorrupt
+					return 0, r.err
+				}
+			}
+			continue
+		}
+
+		if chunkType <= 0x7f {
+			// Section 4.5. Reserved unskippable chunks (chunk types 0x02-0x7f).
+			r.err = ErrUnsupported
+			return 0, r.err
+		}
+		// Section 4.4 Padding (chunk type 0xfe).
+		// Section 4.6. Reserved skippable chunks (chunk types 0x80-0xfd).
+		if !r.readFull(r.buf[:chunkLen]) {
+			return 0, r.err
+		}
+	}
+}

Godeps/_workspace/src/github.com/golang/snappy/encode.go

@@ -6,6 +6,7 @@ package snappy
 
 import (
 	"encoding/binary"
+	"io"
 )
 
 // We limit how far copy back-references can go, the same as the C++ code.
@@ -78,7 +79,7 @@ func emitCopy(dst []byte, offset, length int) int {
 // slice of dst if dst was large enough to hold the entire encoded block.
 // Otherwise, a newly allocated slice will be returned.
 // It is valid to pass a nil dst.
-func Encode(dst, src []byte) ([]byte, error) {
+func Encode(dst, src []byte) []byte {
 	if n := MaxEncodedLen(len(src)); len(dst) < n {
 		dst = make([]byte, n)
 	}
@@ -91,7 +92,7 @@ func Encode(dst, src []byte) ([]byte, error) {
 		if len(src) != 0 {
 			d += emitLiteral(dst[d:], src)
 		}
-		return dst[:d], nil
+		return dst[:d]
 	}
 
 	// Initialize the hash table. Its size ranges from 1<<8 to 1<<14 inclusive.
@@ -144,7 +145,7 @@ func Encode(dst, src []byte) ([]byte, error) {
 	if lit != len(src) {
 		d += emitLiteral(dst[d:], src[lit:])
 	}
-	return dst[:d], nil
+	return dst[:d]
 }
 
 // MaxEncodedLen returns the maximum length of a snappy block, given its
@@ -172,3 +173,82 @@ func MaxEncodedLen(srcLen int) int {
 	// This last factor dominates the blowup, so the final estimate is:
 	return 32 + srcLen + srcLen/6
 }
+
+// NewWriter returns a new Writer that compresses to w, using the framing
+// format described at
+// https://github.com/google/snappy/blob/master/framing_format.txt
+func NewWriter(w io.Writer) *Writer {
+	return &Writer{
+		w:   w,
+		enc: make([]byte, MaxEncodedLen(maxUncompressedChunkLen)),
+	}
+}
+
+// Writer is an io.Writer than can write Snappy-compressed bytes.
+type Writer struct {
+	w           io.Writer
+	err         error
+	enc         []byte
+	buf         [checksumSize + chunkHeaderSize]byte
+	wroteHeader bool
+}
+
+// Reset discards the writer's state and switches the Snappy writer to write to
+// w. This permits reusing a Writer rather than allocating a new one.
+func (w *Writer) Reset(writer io.Writer) {
+	w.w = writer
+	w.err = nil
+	w.wroteHeader = false
+}
+
+// Write satisfies the io.Writer interface.
+func (w *Writer) Write(p []byte) (n int, errRet error) {
+	if w.err != nil {
+		return 0, w.err
+	}
+	if !w.wroteHeader {
+		copy(w.enc, magicChunk)
+		if _, err := w.w.Write(w.enc[:len(magicChunk)]); err != nil {
+			w.err = err
+			return n, err
+		}
+		w.wroteHeader = true
+	}
+	for len(p) > 0 {
+		var uncompressed []byte
+		if len(p) > maxUncompressedChunkLen {
+			uncompressed, p = p[:maxUncompressedChunkLen], p[maxUncompressedChunkLen:]
+		} else {
+			uncompressed, p = p, nil
+		}
+		checksum := crc(uncompressed)
+
+		// Compress the buffer, discarding the result if the improvement
+		// isn't at least 12.5%.
+		chunkType := uint8(chunkTypeCompressedData)
+		chunkBody := Encode(w.enc, uncompressed)
+		if len(chunkBody) >= len(uncompressed)-len(uncompressed)/8 {
+			chunkType, chunkBody = chunkTypeUncompressedData, uncompressed
+		}
+
+		chunkLen := 4 + len(chunkBody)
+		w.buf[0] = chunkType
+		w.buf[1] = uint8(chunkLen >> 0)
+		w.buf[2] = uint8(chunkLen >> 8)
+		w.buf[3] = uint8(chunkLen >> 16)
+		w.buf[4] = uint8(checksum >> 0)
+		w.buf[5] = uint8(checksum >> 8)
+		w.buf[6] = uint8(checksum >> 16)
+		w.buf[7] = uint8(checksum >> 24)
+		if _, err := w.w.Write(w.buf[:]); err != nil {
+			w.err = err
+			return n, err
+		}
+		if _, err := w.w.Write(chunkBody); err != nil {
+			w.err = err
+			return n, err
+		}
+		n += len(uncompressed)
+	}
+	return n, nil
+}

Godeps/_workspace/src/github.com/golang/snappy/snappy.go

@@ -5,9 +5,13 @@
 // Package snappy implements the snappy block-based compression format.
 // It aims for very high speeds and reasonable compression.
 //
-// The C++ snappy implementation is at http://code.google.com/p/snappy/
+// The C++ snappy implementation is at https://github.com/google/snappy
 package snappy
 
+import (
+	"hash/crc32"
+)
+
 /*
 Each encoded block begins with the varint-encoded length of the decoded data,
 followed by a sequence of chunks. Chunks begin and end on byte boundaries. The
@@ -36,3 +40,29 @@ const (
 	tagCopy2   = 0x02
 	tagCopy4   = 0x03
 )
+
+const (
+	checksumSize    = 4
+	chunkHeaderSize = 4
+	magicChunk      = "\xff\x06\x00\x00" + magicBody
+	magicBody       = "sNaPpY"
+	// https://github.com/google/snappy/blob/master/framing_format.txt says
+	// that "the uncompressed data in a chunk must be no longer than 65536 bytes".
+	maxUncompressedChunkLen = 65536
+)
+
+const (
+	chunkTypeCompressedData   = 0x00
+	chunkTypeUncompressedData = 0x01
+	chunkTypePadding          = 0xfe
+	chunkTypeStreamIdentifier = 0xff
+)
+
+var crcTable = crc32.MakeTable(crc32.Castagnoli)
+
+// crc implements the checksum specified in section 3 of
+// https://github.com/google/snappy/blob/master/framing_format.txt
+func crc(b []byte) uint32 {
+	c := crc32.Update(0, crcTable, b)
+	return uint32(c>>15|c<<17) + 0xa282ead8
+}

Godeps/_workspace/src/github.com/golang/snappy/snappy_test.go

@@ -18,14 +18,13 @@ import (
 	"testing"
 )
 
-var download = flag.Bool("download", false, "If true, download any missing files before running benchmarks")
+var (
+	download = flag.Bool("download", false, "If true, download any missing files before running benchmarks")
+	testdata = flag.String("testdata", "testdata", "Directory containing the test data")
+)
 
 func roundtrip(b, ebuf, dbuf []byte) error {
-	e, err := Encode(ebuf, b)
-	if err != nil {
-		return fmt.Errorf("encoding error: %v", err)
-	}
-	d, err := Decode(dbuf, e)
+	d, err := Decode(dbuf, Encode(ebuf, b))
 	if err != nil {
 		return fmt.Errorf("decoding error: %v", err)
 	}
@@ -55,11 +54,11 @@ func TestSmallCopy(t *testing.T) {
 }
 
 func TestSmallRand(t *testing.T) {
-	rand.Seed(27354294)
+	rng := rand.New(rand.NewSource(27354294))
 	for n := 1; n < 20000; n += 23 {
 		b := make([]byte, n)
-		for i, _ := range b {
-			b[i] = uint8(rand.Uint32())
+		for i := range b {
+			b[i] = uint8(rng.Uint32())
 		}
 		if err := roundtrip(b, nil, nil); err != nil {
 			t.Fatal(err)
@@ -70,7 +69,7 @@ func TestSmallRand(t *testing.T) {
 func TestSmallRegular(t *testing.T) {
 	for n := 1; n < 20000; n += 23 {
 		b := make([]byte, n)
-		for i, _ := range b {
+		for i := range b {
 			b[i] = uint8(i%10 + 'a')
 		}
 		if err := roundtrip(b, nil, nil); err != nil {
@@ -79,11 +78,142 @@ func TestSmallRegular(t *testing.T) {
 	}
 }
 
-func benchDecode(b *testing.B, src []byte) {
-	encoded, err := Encode(nil, src)
-	if err != nil {
-		b.Fatal(err)
+func TestInvalidVarint(t *testing.T) {
+	data := []byte("\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00")
+	if _, err := DecodedLen(data); err != ErrCorrupt {
+		t.Errorf("DecodedLen: got %v, want ErrCorrupt", err)
 	}
+	if _, err := Decode(nil, data); err != ErrCorrupt {
+		t.Errorf("Decode: got %v, want ErrCorrupt", err)
+	}
+
+	// The encoded varint overflows 32 bits
+	data = []byte("\xff\xff\xff\xff\xff\x00")
+
+	if _, err := DecodedLen(data); err != ErrCorrupt {
+		t.Errorf("DecodedLen: got %v, want ErrCorrupt", err)
+	}
+	if _, err := Decode(nil, data); err != ErrCorrupt {
+		t.Errorf("Decode: got %v, want ErrCorrupt", err)
+	}
+}
+
+func cmp(a, b []byte) error {
+	if len(a) != len(b) {
+		return fmt.Errorf("got %d bytes, want %d", len(a), len(b))
+	}
+	for i := range a {
+		if a[i] != b[i] {
+			return fmt.Errorf("byte #%d: got 0x%02x, want 0x%02x", i, a[i], b[i])
+		}
+	}
+	return nil
+}
+
+func TestFramingFormat(t *testing.T) {
+	// src is comprised of alternating 1e5-sized sequences of random
+	// (incompressible) bytes and repeated (compressible) bytes. 1e5 was chosen
+	// because it is larger than maxUncompressedChunkLen (64k).
+	src := make([]byte, 1e6)
+	rng := rand.New(rand.NewSource(1))
+	for i := 0; i < 10; i++ {
+		if i%2 == 0 {
+			for j := 0; j < 1e5; j++ {
+				src[1e5*i+j] = uint8(rng.Intn(256))
+			}
+		} else {
+			for j := 0; j < 1e5; j++ {
+				src[1e5*i+j] = uint8(i)
+			}
+		}
+	}
+
+	buf := new(bytes.Buffer)
+	if _, err := NewWriter(buf).Write(src); err != nil {
+		t.Fatalf("Write: encoding: %v", err)
+	}
+	dst, err := ioutil.ReadAll(NewReader(buf))
+	if err != nil {
+		t.Fatalf("ReadAll: decoding: %v", err)
+	}
+	if err := cmp(dst, src); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestReaderReset(t *testing.T) {
+	gold := bytes.Repeat([]byte("All that is gold does not glitter,\n"), 10000)
+	buf := new(bytes.Buffer)
+	if _, err := NewWriter(buf).Write(gold); err != nil {
+		t.Fatalf("Write: %v", err)
+	}
+	encoded, invalid, partial := buf.String(), "invalid", "partial"
+	r := NewReader(nil)
+	for i, s := range []string{encoded, invalid, partial, encoded, partial, invalid, encoded, encoded} {
+		if s == partial {
+			r.Reset(strings.NewReader(encoded))
+			if _, err := r.Read(make([]byte, 101)); err != nil {
+				t.Errorf("#%d: %v", i, err)
+				continue
+			}
+			continue
+		}
+		r.Reset(strings.NewReader(s))
+		got, err := ioutil.ReadAll(r)
+		switch s {
+		case encoded:
+			if err != nil {
+				t.Errorf("#%d: %v", i, err)
+				continue
+			}
+			if err := cmp(got, gold); err != nil {
+				t.Errorf("#%d: %v", i, err)
+				continue
+			}
+		case invalid:
+			if err == nil {
+				t.Errorf("#%d: got nil error, want non-nil", i)
+				continue
+			}
+		}
+	}
+}
+
+func TestWriterReset(t *testing.T) {
+	gold := bytes.Repeat([]byte("Not all those who wander are lost;\n"), 10000)
+	var gots, wants [][]byte
+	const n = 20
+	w, failed := NewWriter(nil), false
+	for i := 0; i <= n; i++ {
+		buf := new(bytes.Buffer)
+		w.Reset(buf)
+		want := gold[:len(gold)*i/n]
+		if _, err := w.Write(want); err != nil {
+			t.Errorf("#%d: Write: %v", i, err)
+			failed = true
+			continue
+		}
+		got, err := ioutil.ReadAll(NewReader(buf))
+		if err != nil {
+			t.Errorf("#%d: ReadAll: %v", i, err)
+			failed = true
+			continue
+		}
+		gots = append(gots, got)
+		wants = append(wants, want)
+	}
+	if failed {
+		return
+	}
+	for i := range gots {
+		if err := cmp(gots[i], wants[i]); err != nil {
+			t.Errorf("#%d: %v", i, err)
+		}
+	}
+}
+
+func benchDecode(b *testing.B, src []byte) {
+	encoded := Encode(nil, src)
 	// Bandwidth is in amount of uncompressed data.
 	b.SetBytes(int64(len(src)))
 	b.ResetTimer()
@@ -102,10 +232,10 @@ func benchEncode(b *testing.B, src []byte) {
 	}
 }
 
-func readFile(b *testing.B, filename string) []byte {
+func readFile(b testing.TB, filename string) []byte {
 	src, err := ioutil.ReadFile(filename)
 	if err != nil {
-		b.Fatalf("failed reading %s: %s", filename, err)
+		b.Skipf("skipping benchmark: %v", err)
 	}
 	if len(src) == 0 {
 		b.Fatalf("%s has zero length", filename)
@@ -144,7 +274,7 @@ func BenchmarkWordsEncode1e5(b *testing.B) { benchWords(b, 1e5, false) }
 func BenchmarkWordsEncode1e6(b *testing.B) { benchWords(b, 1e6, false) }
 
 // testFiles' values are copied directly from
-// https://code.google.com/p/snappy/source/browse/trunk/snappy_unittest.cc.
+// https://raw.githubusercontent.com/google/snappy/master/snappy_unittest.cc
 // The label field is unused in snappy-go.
 var testFiles = []struct {
 	label    string
@@ -152,29 +282,36 @@ var testFiles = []struct {
 }{
 	{"html", "html"},
 	{"urls", "urls.10K"},
-	{"jpg", "house.jpg"},
-	{"pdf", "mapreduce-osdi-1.pdf"},
+	{"jpg", "fireworks.jpeg"},
+	{"jpg_200", "fireworks.jpeg"},
+	{"pdf", "paper-100k.pdf"},
 	{"html4", "html_x_4"},
-	{"cp", "cp.html"},
-	{"c", "fields.c"},
-	{"lsp", "grammar.lsp"},
-	{"xls", "kennedy.xls"},
 	{"txt1", "alice29.txt"},
 	{"txt2", "asyoulik.txt"},
 	{"txt3", "lcet10.txt"},
 	{"txt4", "plrabn12.txt"},
-	{"bin", "ptt5"},
-	{"sum", "sum"},
-	{"man", "xargs.1"},
 	{"pb", "geo.protodata"},
 	{"gaviota", "kppkn.gtb"},
 }
 
 // The test data files are present at this canonical URL.
-const baseURL = "https://snappy.googlecode.com/svn/trunk/testdata/"
+const baseURL = "https://raw.githubusercontent.com/google/snappy/master/testdata/"
+
+func downloadTestdata(b *testing.B, basename string) (errRet error) {
+	filename := filepath.Join(*testdata, basename)
+	if stat, err := os.Stat(filename); err == nil && stat.Size() != 0 {
+		return nil
+	}
+
+	if !*download {
+		b.Skipf("test data not found; skipping benchmark without the -download flag")
+	}
+	// Download the official snappy C++ implementation reference test data
+	// files for benchmarking.
+	if err := os.Mkdir(*testdata, 0777); err != nil && !os.IsExist(err) {
+		return fmt.Errorf("failed to create testdata: %s", err)
+	}
 
-func downloadTestdata(basename string) (errRet error) {
-	filename := filepath.Join("testdata", basename)
 	f, err := os.Create(filename)
 	if err != nil {
 		return fmt.Errorf("failed to create %s: %s", filename, err)
@@ -185,36 +322,27 @@ func downloadTestdata(basename string) (errRet error) {
 			os.Remove(filename)
 		}
 	}()
-	resp, err := http.Get(baseURL + basename)
+	url := baseURL + basename
+	resp, err := http.Get(url)
 	if err != nil {
-		return fmt.Errorf("failed to download %s: %s", baseURL+basename, err)
+		return fmt.Errorf("failed to download %s: %s", url, err)
 	}
 	defer resp.Body.Close()
+	if s := resp.StatusCode; s != http.StatusOK {
+		return fmt.Errorf("downloading %s: HTTP status code %d (%s)", url, s, http.StatusText(s))
+	}
 	_, err = io.Copy(f, resp.Body)
 	if err != nil {
-		return fmt.Errorf("failed to write %s: %s", filename, err)
+		return fmt.Errorf("failed to download %s to %s: %s", url, filename, err)
 	}
 	return nil
 }
 
 func benchFile(b *testing.B, n int, decode bool) {
-	filename := filepath.Join("testdata", testFiles[n].filename)
-	if stat, err := os.Stat(filename); err != nil || stat.Size() == 0 {
-		if !*download {
-			b.Fatal("test data not found; skipping benchmark without the -download flag")
-		}
-		// Download the official snappy C++ implementation reference test data
-		// files for benchmarking.
-		if err := os.Mkdir("testdata", 0777); err != nil && !os.IsExist(err) {
-			b.Fatalf("failed to create testdata: %s", err)
-		}
-		for _, tf := range testFiles {
-			if err := downloadTestdata(tf.filename); err != nil {
-				b.Fatalf("failed to download testdata: %s", err)
-			}
-		}
+	if err := downloadTestdata(b, testFiles[n].filename); err != nil {
+		b.Fatalf("failed to download testdata: %s", err)
 	}
-	data := readFile(b, filename)
+	data := readFile(b, filepath.Join(*testdata, testFiles[n].filename))
 	if decode {
 		benchDecode(b, data)
 	} else {
@@ -235,12 +363,6 @@ func Benchmark_UFlat8(b *testing.B)  { benchFile(b, 8, true) }
 func Benchmark_UFlat9(b *testing.B)  { benchFile(b, 9, true) }
 func Benchmark_UFlat10(b *testing.B) { benchFile(b, 10, true) }
 func Benchmark_UFlat11(b *testing.B) { benchFile(b, 11, true) }
-func Benchmark_UFlat12(b *testing.B) { benchFile(b, 12, true) }
-func Benchmark_UFlat13(b *testing.B) { benchFile(b, 13, true) }
-func Benchmark_UFlat14(b *testing.B) { benchFile(b, 14, true) }
-func Benchmark_UFlat15(b *testing.B) { benchFile(b, 15, true) }
-func Benchmark_UFlat16(b *testing.B) { benchFile(b, 16, true) }
-func Benchmark_UFlat17(b *testing.B) { benchFile(b, 17, true) }
 func Benchmark_ZFlat0(b *testing.B)  { benchFile(b, 0, false) }
 func Benchmark_ZFlat1(b *testing.B)  { benchFile(b, 1, false) }
 func Benchmark_ZFlat2(b *testing.B)  { benchFile(b, 2, false) }
@@ -253,9 +375,3 @@ func Benchmark_ZFlat8(b *testing.B)  { benchFile(b, 8, false) }
 func Benchmark_ZFlat9(b *testing.B)  { benchFile(b, 9, false) }
 func Benchmark_ZFlat10(b *testing.B) { benchFile(b, 10, false) }
 func Benchmark_ZFlat11(b *testing.B) { benchFile(b, 11, false) }
-func Benchmark_ZFlat12(b *testing.B) { benchFile(b, 12, false) }
-func Benchmark_ZFlat13(b *testing.B) { benchFile(b, 13, false) }
-func Benchmark_ZFlat14(b *testing.B) { benchFile(b, 14, false) }
-func Benchmark_ZFlat15(b *testing.B) { benchFile(b, 15, false) }
-func Benchmark_ZFlat16(b *testing.B) { benchFile(b, 16, false) }
-func Benchmark_ZFlat17(b *testing.B) { benchFile(b, 17, false) }

Godeps/_workspace/src/github.com/juju/ratelimit/LICENSE

@@ -1,3 +1,9 @@
+All files in this repository are licensed as follows. If you contribute
+to this repository, it is assumed that you license your contribution
+under the same license unless you state otherwise.
+
+All files Copyright (C) 2015 Canonical Ltd. unless otherwise specified in the file.
+
 This software is licensed under the LGPLv3, included below.
 
 As a special exception to the GNU Lesser General Public License version 3

Godeps/_workspace/src/github.com/juju/ratelimit/README.md

@@ -20,7 +20,7 @@ token in the bucket represents one byte.
 ```go
 func Writer(w io.Writer, bucket *Bucket) io.Writer
 ```
-Writer returns a reader that is rate limited by the given token bucket. Each
+Writer returns a writer that is rate limited by the given token bucket. Each
 token in the bucket represents one byte.
 
 #### type Bucket

Godeps/_workspace/src/github.com/juju/ratelimit/ratelimit.go

@@ -2,7 +2,8 @@
 // Licensed under the LGPLv3 with static-linking exception.
 // See LICENCE file for details.
 
-// The ratelimit package provides an efficient token bucket implementation.
+// The ratelimit package provides an efficient token bucket implementation
+// that can be used to limit the rate of arbitrary things.
 // See http://en.wikipedia.org/wiki/Token_bucket.
 package ratelimit
 
@@ -10,6 +11,7 @@ import (
 	"strconv"
 	"sync"
 	"time"
+	"math"
 )
 
 // Bucket represents a token bucket that fills at a predetermined rate.
@@ -55,7 +57,7 @@ func NewBucketWithRate(rate float64, capacity int64) *Bucket {
 			continue
 		}
 		tb := NewBucketWithQuantum(fillInterval, capacity, quantum)
-		if diff := abs(tb.Rate() - rate); diff/rate <= rateMargin {
+		if diff := math.Abs(tb.Rate() - rate); diff/rate <= rateMargin {
 			return tb
 		}
 	}
@@ -217,10 +219,3 @@ func (tb *Bucket) adjust(now time.Time) (currentTick int64) {
 	tb.availTick = currentTick
 	return
 }
-
-func abs(f float64) float64 {
-	if f < 0 {
-		return -f
-	}
-	return f
-}

Godeps/_workspace/src/github.com/kardianos/osext/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2012 The Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Godeps/_workspace/src/github.com/kardianos/osext/README.md

@@ -0,0 +1,16 @@
+### Extensions to the "os" package.
+
+## Find the current Executable and ExecutableFolder.
+
+There is sometimes utility in finding the current executable file
+that is running. This can be used for upgrading the current executable
+or finding resources located relative to the executable file. Both
+working directory and the os.Args[0] value are arbitrary and cannot
+be relied on; os.Args[0] can be "faked".
+
+Multi-platform and supports:
+ * Linux
+ * OS X
+ * Windows
+ * Plan 9
+ * BSDs.

Godeps/_workspace/src/github.com/kardianos/osext/osext.go

@@ -16,17 +16,12 @@ func Executable() (string, error) {
 }
 
 // Returns same path as Executable, returns just the folder
-// path. Excludes the executable name.
+// path. Excludes the executable name and any trailing slash.
 func ExecutableFolder() (string, error) {
 	p, err := Executable()
 	if err != nil {
 		return "", err
 	}
-	folder, _ := filepath.Split(p)
-	return folder, nil
-}
 
-// Depricated. Same as Executable().
-func GetExePath() (exePath string, err error) {
-	return Executable()
+	return filepath.Dir(p), nil
 }

Godeps/_workspace/src/github.com/kardianos/osext/osext_plan9.go

@@ -5,16 +5,16 @@
 package osext
 
 import (
-    "syscall"
-    "os"
-    "strconv"
+	"os"
+	"strconv"
+	"syscall"
 )
 
 func executable() (string, error) {
-    f, err := os.Open("/proc/" + strconv.Itoa(os.Getpid()) + "/text")
-    if err != nil {
-        return "", err
-    }
-    defer f.Close()
-    return syscall.Fd2path(int(f.Fd()))
+	f, err := os.Open("/proc/" + strconv.Itoa(os.Getpid()) + "/text")
+	if err != nil {
+		return "", err
+	}
+	defer f.Close()
+	return syscall.Fd2path(int(f.Fd()))
 }

Godeps/_workspace/src/github.com/kardianos/osext/osext_procfs.go

@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-// +build linux netbsd openbsd solaris
+// +build linux netbsd openbsd solaris dragonfly
 
 package osext
 
@@ -11,15 +11,23 @@ import (
 	"fmt"
 	"os"
 	"runtime"
+	"strings"
 )
 
 func executable() (string, error) {
 	switch runtime.GOOS {
 	case "linux":
-		return os.Readlink("/proc/self/exe")
+		const deletedTag = " (deleted)"
+		execpath, err := os.Readlink("/proc/self/exe")
+		if err != nil {
+			return execpath, err
+		}
+		execpath = strings.TrimSuffix(execpath, deletedTag)
+		execpath = strings.TrimPrefix(execpath, deletedTag)
+		return execpath, nil
 	case "netbsd":
 		return os.Readlink("/proc/curproc/exe")
-	case "openbsd":
+	case "openbsd", "dragonfly":
 		return os.Readlink("/proc/curproc/file")
 	case "solaris":
 		return os.Readlink(fmt.Sprintf("/proc/%d/path/a.out", os.Getpid()))

Godeps/_workspace/src/github.com/kardianos/osext/osext_test.go

@@ -0,0 +1,203 @@
+// Copyright 2012 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// +build darwin linux freebsd netbsd windows
+
+package osext
+
+import (
+	"bytes"
+	"fmt"
+	"io"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"testing"
+)
+
+const (
+	executableEnvVar = "OSTEST_OUTPUT_EXECUTABLE"
+
+	executableEnvValueMatch  = "match"
+	executableEnvValueDelete = "delete"
+)
+
+func TestPrintExecutable(t *testing.T) {
+	ef, err := Executable()
+	if err != nil {
+		t.Fatalf("Executable failed: %v", err)
+	}
+	t.Log("Executable:", ef)
+}
+func TestPrintExecutableFolder(t *testing.T) {
+	ef, err := ExecutableFolder()
+	if err != nil {
+		t.Fatalf("ExecutableFolder failed: %v", err)
+	}
+	t.Log("Executable Folder:", ef)
+}
+func TestExecutableFolder(t *testing.T) {
+	ef, err := ExecutableFolder()
+	if err != nil {
+		t.Fatalf("ExecutableFolder failed: %v", err)
+	}
+	if ef[len(ef)-1] == filepath.Separator {
+		t.Fatal("ExecutableFolder ends with a trailing slash.")
+	}
+}
+func TestExecutableMatch(t *testing.T) {
+	ep, err := Executable()
+	if err != nil {
+		t.Fatalf("Executable failed: %v", err)
+	}
+
+	// fullpath to be of the form "dir/prog".
+	dir := filepath.Dir(filepath.Dir(ep))
+	fullpath, err := filepath.Rel(dir, ep)
+	if err != nil {
+		t.Fatalf("filepath.Rel: %v", err)
+	}
+	// Make child start with a relative program path.
+	// Alter argv[0] for child to verify getting real path without argv[0].
+	cmd := &exec.Cmd{
+		Dir:  dir,
+		Path: fullpath,
+		Env:  []string{fmt.Sprintf("%s=%s", executableEnvVar, executableEnvValueMatch)},
+	}
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		t.Fatalf("exec(self) failed: %v", err)
+	}
+	outs := string(out)
+	if !filepath.IsAbs(outs) {
+		t.Fatalf("Child returned %q, want an absolute path", out)
+	}
+	if !sameFile(outs, ep) {
+		t.Fatalf("Child returned %q, not the same file as %q", out, ep)
+	}
+}
+
+func TestExecutableDelete(t *testing.T) {
+	if runtime.GOOS != "linux" {
+		t.Skip()
+	}
+	fpath, err := Executable()
+	if err != nil {
+		t.Fatalf("Executable failed: %v", err)
+	}
+
+	r, w := io.Pipe()
+	stderrBuff := &bytes.Buffer{}
+	stdoutBuff := &bytes.Buffer{}
+	cmd := &exec.Cmd{
+		Path:   fpath,
+		Env:    []string{fmt.Sprintf("%s=%s", executableEnvVar, executableEnvValueDelete)},
+		Stdin:  r,
+		Stderr: stderrBuff,
+		Stdout: stdoutBuff,
+	}
+	err = cmd.Start()
+	if err != nil {
+		t.Fatalf("exec(self) start failed: %v", err)
+	}
+
+	tempPath := fpath + "_copy"
+	_ = os.Remove(tempPath)
+
+	err = copyFile(tempPath, fpath)
+	if err != nil {
+		t.Fatalf("copy file failed: %v", err)
+	}
+	err = os.Remove(fpath)
+	if err != nil {
+		t.Fatalf("remove running test file failed: %v", err)
+	}
+	err = os.Rename(tempPath, fpath)
+	if err != nil {
+		t.Fatalf("rename copy to previous name failed: %v", err)
+	}
+
+	w.Write([]byte{0})
+	w.Close()
+
+	err = cmd.Wait()
+	if err != nil {
+		t.Fatalf("exec wait failed: %v", err)
+	}
+
+	childPath := stderrBuff.String()
+	if !filepath.IsAbs(childPath) {
+		t.Fatalf("Child returned %q, want an absolute path", childPath)
+	}
+	if !sameFile(childPath, fpath) {
+		t.Fatalf("Child returned %q, not the same file as %q", childPath, fpath)
+	}
+}
+
+func sameFile(fn1, fn2 string) bool {
+	fi1, err := os.Stat(fn1)
+	if err != nil {
+		return false
+	}
+	fi2, err := os.Stat(fn2)
+	if err != nil {
+		return false
+	}
+	return os.SameFile(fi1, fi2)
+}
+func copyFile(dest, src string) error {
+	df, err := os.Create(dest)
+	if err != nil {
+		return err
+	}
+	defer df.Close()
+
+	sf, err := os.Open(src)
+	if err != nil {
+		return err
+	}
+	defer sf.Close()
+
+	_, err = io.Copy(df, sf)
+	return err
+}
+
+func TestMain(m *testing.M) {
+	env := os.Getenv(executableEnvVar)
+	switch env {
+	case "":
+		os.Exit(m.Run())
+	case executableEnvValueMatch:
+		// First chdir to another path.
+		dir := "/"
+		if runtime.GOOS == "windows" {
+			dir = filepath.VolumeName(".")
+		}
+		os.Chdir(dir)
+		if ep, err := Executable(); err != nil {
+			fmt.Fprint(os.Stderr, "ERROR: ", err)
+		} else {
+			fmt.Fprint(os.Stderr, ep)
+		}
+	case executableEnvValueDelete:
+		bb := make([]byte, 1)
+		var err error
+		n, err := os.Stdin.Read(bb)
+		if err != nil {
+			fmt.Fprint(os.Stderr, "ERROR: ", err)
+			os.Exit(2)
+		}
+		if n != 1 {
+			fmt.Fprint(os.Stderr, "ERROR: n != 1, n == ", n)
+			os.Exit(2)
+		}
+		if ep, err := Executable(); err != nil {
+			fmt.Fprint(os.Stderr, "ERROR: ", err)
+		} else {
+			fmt.Fprint(os.Stderr, ep)
+		}
+	}
+	os.Exit(0)
+}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/bench2_test.go

@@ -4,7 +4,7 @@
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
-// +build go1.3
+// +build !go1.2
 
 package leveldb
 

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/cache/bench2_test.go

@@ -0,0 +1,30 @@
+// Copyright (c) 2012, Suryandaru Triandana <syndtr@gmail.com>
+// All rights reserved.
+//
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+// +build !go1.2
+
+package cache
+
+import (
+	"math/rand"
+	"testing"
+)
+
+func BenchmarkLRUCache(b *testing.B) {
+	c := NewCache(NewLRU(10000))
+
+	b.SetParallelism(10)
+	b.RunParallel(func(pb *testing.PB) {
+		r := rand.New(rand.NewSource(time.Now().UnixNano()))
+
+		for pb.Next() {
+			key := uint64(r.Intn(1000000))
+			c.Get(0, key, func() (int, Value) {
+				return 1, key
+			}).Release()
+		}
+	})
+}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/cache/cache.go

@@ -8,152 +8,669 @@
 package cache
 
 import (
+	"sync"
 	"sync/atomic"
+	"unsafe"
+
+	"github.com/syndtr/goleveldb/leveldb/util"
 )
 
-// SetFunc is the function that will be called by Namespace.Get to create
-// a cache object, if charge is less than one than the cache object will
-// not be registered to cache tree, if value is nil then the cache object
-// will not be created.
-type SetFunc func() (charge int, value interface{})
-
-// DelFin is the function that will be called as the result of a delete operation.
-// Exist == true is indication that the object is exist, and pending == true is
-// indication of deletion already happen but haven't done yet (wait for all handles
-// to be released). And exist == false means the object doesn't exist.
-type DelFin func(exist, pending bool)
-
-// PurgeFin is the function that will be called as the result of a purge operation.
-type PurgeFin func(ns, key uint64)
-
-// Cache is a cache tree. A cache instance must be goroutine-safe.
-type Cache interface {
-	// SetCapacity sets cache tree capacity.
-	SetCapacity(capacity int)
-
-	// Capacity returns cache tree capacity.
+// Cacher provides interface to implements a caching functionality.
+// An implementation must be goroutine-safe.
+type Cacher interface {
+	// Capacity returns cache capacity.
 	Capacity() int
 
-	// Used returns used cache tree capacity.
-	Used() int
+	// SetCapacity sets cache capacity.
+	SetCapacity(capacity int)
 
-	// Size returns entire alive cache objects size.
-	Size() int
+	// Promote promotes the 'cache node'.
+	Promote(n *Node)
 
-	// NumObjects returns number of alive objects.
-	NumObjects() int
+	// Ban evicts the 'cache node' and prevent subsequent 'promote'.
+	Ban(n *Node)
 
-	// GetNamespace gets cache namespace with the given id.
-	// GetNamespace is never return nil.
-	GetNamespace(id uint64) Namespace
+	// Evict evicts the 'cache node'.
+	Evict(n *Node)
 
-	// PurgeNamespace purges cache namespace with the given id from this cache tree.
-	// Also read Namespace.Purge.
-	PurgeNamespace(id uint64, fin PurgeFin)
+	// EvictNS evicts 'cache node' with the given namespace.
+	EvictNS(ns uint64)
 
-	// ZapNamespace detaches cache namespace with the given id from this cache tree.
-	// Also read Namespace.Zap.
-	ZapNamespace(id uint64)
+	// EvictAll evicts all 'cache node'.
+	EvictAll()
 
-	// Purge purges all cache namespace from this cache tree.
-	// This is behave the same as calling Namespace.Purge method on all cache namespace.
-	Purge(fin PurgeFin)
-
-	// Zap detaches all cache namespace from this cache tree.
-	// This is behave the same as calling Namespace.Zap method on all cache namespace.
-	Zap()
+	// Close closes the 'cache tree'
+	Close() error
 }
 
-// Namespace is a cache namespace. A namespace instance must be goroutine-safe.
-type Namespace interface {
-	// Get gets cache object with the given key.
-	// If cache object is not found and setf is not nil, Get will atomically creates
-	// the cache object by calling setf. Otherwise Get will returns nil.
-	//
-	// The returned cache handle should be released after use by calling Release
-	// method.
-	Get(key uint64, setf SetFunc) Handle
+// Value is a 'cacheable object'. It may implements util.Releaser, if
+// so the the Release method will be called once object is released.
+type Value interface{}
 
-	// Delete removes cache object with the given key from cache tree.
-	// A deleted cache object will be released as soon as all of its handles have
-	// been released.
-	// Delete only happen once, subsequent delete will consider cache object doesn't
-	// exist, even if the cache object ins't released yet.
-	//
-	// If not nil, fin will be called if the cache object doesn't exist or when
-	// finally be released.
-	//
-	// Delete returns true if such cache object exist and never been deleted.
-	Delete(key uint64, fin DelFin) bool
-
-	// Purge removes all cache objects within this namespace from cache tree.
-	// This is the same as doing delete on all cache objects.
-	//
-	// If not nil, fin will be called on all cache objects when its finally be
-	// released.
-	Purge(fin PurgeFin)
-
-	// Zap detaches namespace from cache tree and release all its cache objects.
-	// A zapped namespace can never be filled again.
-	// Calling Get on zapped namespace will always return nil.
-	Zap()
+type CacheGetter struct {
+	Cache *Cache
+	NS    uint64
 }
 
-// Handle is a cache handle.
-type Handle interface {
-	// Release releases this cache handle. This method can be safely called mutiple
-	// times.
-	Release()
-
-	// Value returns value of this cache handle.
-	// Value will returns nil after this cache handle have be released.
-	Value() interface{}
+func (g *CacheGetter) Get(key uint64, setFunc func() (size int, value Value)) *Handle {
+	return g.Cache.Get(g.NS, key, setFunc)
 }
 
+// The hash tables implementation is based on:
+// "Dynamic-Sized Nonblocking Hash Tables", by Yujie Liu, Kunlong Zhang, and Michael Spear. ACM Symposium on Principles of Distributed Computing, Jul 2014.
+
 const (
-	DelNotExist = iota
-	DelExist
-	DelPendig
+	mInitialSize           = 1 << 4
+	mOverflowThreshold     = 1 << 5
+	mOverflowGrowThreshold = 1 << 7
 )
 
-// Namespace state.
-type nsState int
-
-const (
-	nsEffective nsState = iota
-	nsZapped
-)
-
-// Node state.
-type nodeState int
-
-const (
-	nodeZero nodeState = iota
-	nodeEffective
-	nodeEvicted
-	nodeDeleted
-)
-
-// Fake handle.
-type fakeHandle struct {
-	value interface{}
-	fin   func()
-	once  uint32
+type mBucket struct {
+	mu     sync.Mutex
+	node   []*Node
+	frozen bool
 }
 
-func (h *fakeHandle) Value() interface{} {
-	if atomic.LoadUint32(&h.once) == 0 {
-		return h.value
+func (b *mBucket) freeze() []*Node {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	if !b.frozen {
+		b.frozen = true
+	}
+	return b.node
+}
+
+func (b *mBucket) get(r *Cache, h *mNode, hash uint32, ns, key uint64, noset bool) (done, added bool, n *Node) {
+	b.mu.Lock()
+
+	if b.frozen {
+		b.mu.Unlock()
+		return
+	}
+
+	// Scan the node.
+	for _, n := range b.node {
+		if n.hash == hash && n.ns == ns && n.key == key {
+			atomic.AddInt32(&n.ref, 1)
+			b.mu.Unlock()
+			return true, false, n
+		}
+	}
+
+	// Get only.
+	if noset {
+		b.mu.Unlock()
+		return true, false, nil
+	}
+
+	// Create node.
+	n = &Node{
+		r:    r,
+		hash: hash,
+		ns:   ns,
+		key:  key,
+		ref:  1,
+	}
+	// Add node to bucket.
+	b.node = append(b.node, n)
+	bLen := len(b.node)
+	b.mu.Unlock()
+
+	// Update counter.
+	grow := atomic.AddInt32(&r.nodes, 1) >= h.growThreshold
+	if bLen > mOverflowThreshold {
+		grow = grow || atomic.AddInt32(&h.overflow, 1) >= mOverflowGrowThreshold
+	}
+
+	// Grow.
+	if grow && atomic.CompareAndSwapInt32(&h.resizeInProgess, 0, 1) {
+		nhLen := len(h.buckets) << 1
+		nh := &mNode{
+			buckets:         make([]unsafe.Pointer, nhLen),
+			mask:            uint32(nhLen) - 1,
+			pred:            unsafe.Pointer(h),
+			growThreshold:   int32(nhLen * mOverflowThreshold),
+			shrinkThreshold: int32(nhLen >> 1),
+		}
+		ok := atomic.CompareAndSwapPointer(&r.mHead, unsafe.Pointer(h), unsafe.Pointer(nh))
+		if !ok {
+			panic("BUG: failed swapping head")
+		}
+		go nh.initBuckets()
+	}
+
+	return true, true, n
+}
+
+func (b *mBucket) delete(r *Cache, h *mNode, hash uint32, ns, key uint64) (done, deleted bool) {
+	b.mu.Lock()
+
+	if b.frozen {
+		b.mu.Unlock()
+		return
+	}
+
+	// Scan the node.
+	var (
+		n    *Node
+		bLen int
+	)
+	for i := range b.node {
+		n = b.node[i]
+		if n.ns == ns && n.key == key {
+			if atomic.LoadInt32(&n.ref) == 0 {
+				deleted = true
+
+				// Call releaser.
+				if n.value != nil {
+					if r, ok := n.value.(util.Releaser); ok {
+						r.Release()
+					}
+					n.value = nil
+				}
+
+				// Remove node from bucket.
+				b.node = append(b.node[:i], b.node[i+1:]...)
+				bLen = len(b.node)
+			}
+			break
+		}
+	}
+	b.mu.Unlock()
+
+	if deleted {
+		// Call OnDel.
+		for _, f := range n.onDel {
+			f()
+		}
+
+		// Update counter.
+		atomic.AddInt32(&r.size, int32(n.size)*-1)
+		shrink := atomic.AddInt32(&r.nodes, -1) < h.shrinkThreshold
+		if bLen >= mOverflowThreshold {
+			atomic.AddInt32(&h.overflow, -1)
+		}
+
+		// Shrink.
+		if shrink && len(h.buckets) > mInitialSize && atomic.CompareAndSwapInt32(&h.resizeInProgess, 0, 1) {
+			nhLen := len(h.buckets) >> 1
+			nh := &mNode{
+				buckets:         make([]unsafe.Pointer, nhLen),
+				mask:            uint32(nhLen) - 1,
+				pred:            unsafe.Pointer(h),
+				growThreshold:   int32(nhLen * mOverflowThreshold),
+				shrinkThreshold: int32(nhLen >> 1),
+			}
+			ok := atomic.CompareAndSwapPointer(&r.mHead, unsafe.Pointer(h), unsafe.Pointer(nh))
+			if !ok {
+				panic("BUG: failed swapping head")
+			}
+			go nh.initBuckets()
+		}
+	}
+
+	return true, deleted
+}
+
+type mNode struct {
+	buckets         []unsafe.Pointer // []*mBucket
+	mask            uint32
+	pred            unsafe.Pointer // *mNode
+	resizeInProgess int32
+
+	overflow        int32
+	growThreshold   int32
+	shrinkThreshold int32
+}
+
+func (n *mNode) initBucket(i uint32) *mBucket {
+	if b := (*mBucket)(atomic.LoadPointer(&n.buckets[i])); b != nil {
+		return b
+	}
+
+	p := (*mNode)(atomic.LoadPointer(&n.pred))
+	if p != nil {
+		var node []*Node
+		if n.mask > p.mask {
+			// Grow.
+			pb := (*mBucket)(atomic.LoadPointer(&p.buckets[i&p.mask]))
+			if pb == nil {
+				pb = p.initBucket(i & p.mask)
+			}
+			m := pb.freeze()
+			// Split nodes.
+			for _, x := range m {
+				if x.hash&n.mask == i {
+					node = append(node, x)
+				}
+			}
+		} else {
+			// Shrink.
+			pb0 := (*mBucket)(atomic.LoadPointer(&p.buckets[i]))
+			if pb0 == nil {
+				pb0 = p.initBucket(i)
+			}
+			pb1 := (*mBucket)(atomic.LoadPointer(&p.buckets[i+uint32(len(n.buckets))]))
+			if pb1 == nil {
+				pb1 = p.initBucket(i + uint32(len(n.buckets)))
+			}
+			m0 := pb0.freeze()
+			m1 := pb1.freeze()
+			// Merge nodes.
+			node = make([]*Node, 0, len(m0)+len(m1))
+			node = append(node, m0...)
+			node = append(node, m1...)
+		}
+		b := &mBucket{node: node}
+		if atomic.CompareAndSwapPointer(&n.buckets[i], nil, unsafe.Pointer(b)) {
+			if len(node) > mOverflowThreshold {
+				atomic.AddInt32(&n.overflow, int32(len(node)-mOverflowThreshold))
+			}
+			return b
+		}
+	}
+
+	return (*mBucket)(atomic.LoadPointer(&n.buckets[i]))
+}
+
+func (n *mNode) initBuckets() {
+	for i := range n.buckets {
+		n.initBucket(uint32(i))
+	}
+	atomic.StorePointer(&n.pred, nil)
+}
+
+// Cache is a 'cache map'.
+type Cache struct {
+	mu     sync.RWMutex
+	mHead  unsafe.Pointer // *mNode
+	nodes  int32
+	size   int32
+	cacher Cacher
+	closed bool
+}
+
+// NewCache creates a new 'cache map'. The cacher is optional and
+// may be nil.
+func NewCache(cacher Cacher) *Cache {
+	h := &mNode{
+		buckets:         make([]unsafe.Pointer, mInitialSize),
+		mask:            mInitialSize - 1,
+		growThreshold:   int32(mInitialSize * mOverflowThreshold),
+		shrinkThreshold: 0,
+	}
+	for i := range h.buckets {
+		h.buckets[i] = unsafe.Pointer(&mBucket{})
+	}
+	r := &Cache{
+		mHead:  unsafe.Pointer(h),
+		cacher: cacher,
+	}
+	return r
+}
+
+func (r *Cache) getBucket(hash uint32) (*mNode, *mBucket) {
+	h := (*mNode)(atomic.LoadPointer(&r.mHead))
+	i := hash & h.mask
+	b := (*mBucket)(atomic.LoadPointer(&h.buckets[i]))
+	if b == nil {
+		b = h.initBucket(i)
+	}
+	return h, b
+}
+
+func (r *Cache) delete(n *Node) bool {
+	for {
+		h, b := r.getBucket(n.hash)
+		done, deleted := b.delete(r, h, n.hash, n.ns, n.key)
+		if done {
+			return deleted
+		}
+	}
+	return false
+}
+
+// Nodes returns number of 'cache node' in the map.
+func (r *Cache) Nodes() int {
+	return int(atomic.LoadInt32(&r.nodes))
+}
+
+// Size returns sums of 'cache node' size in the map.
+func (r *Cache) Size() int {
+	return int(atomic.LoadInt32(&r.size))
+}
+
+// Capacity returns cache capacity.
+func (r *Cache) Capacity() int {
+	if r.cacher == nil {
+		return 0
+	}
+	return r.cacher.Capacity()
+}
+
+// SetCapacity sets cache capacity.
+func (r *Cache) SetCapacity(capacity int) {
+	if r.cacher != nil {
+		r.cacher.SetCapacity(capacity)
+	}
+}
+
+// Get gets 'cache node' with the given namespace and key.
+// If cache node is not found and setFunc is not nil, Get will atomically creates
+// the 'cache node' by calling setFunc. Otherwise Get will returns nil.
+//
+// The returned 'cache handle' should be released after use by calling Release
+// method.
+func (r *Cache) Get(ns, key uint64, setFunc func() (size int, value Value)) *Handle {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+	if r.closed {
+		return nil
+	}
+
+	hash := murmur32(ns, key, 0xf00)
+	for {
+		h, b := r.getBucket(hash)
+		done, _, n := b.get(r, h, hash, ns, key, setFunc == nil)
+		if done {
+			if n != nil {
+				n.mu.Lock()
+				if n.value == nil {
+					if setFunc == nil {
+						n.mu.Unlock()
+						n.unref()
+						return nil
+					}
+
+					n.size, n.value = setFunc()
+					if n.value == nil {
+						n.size = 0
+						n.mu.Unlock()
+						n.unref()
+						return nil
+					}
+					atomic.AddInt32(&r.size, int32(n.size))
+				}
+				n.mu.Unlock()
+				if r.cacher != nil {
+					r.cacher.Promote(n)
+				}
+				return &Handle{unsafe.Pointer(n)}
+			}
+
+			break
+		}
 	}
 	return nil
 }
 
-func (h *fakeHandle) Release() {
-	if !atomic.CompareAndSwapUint32(&h.once, 0, 1) {
+// Delete removes and ban 'cache node' with the given namespace and key.
+// A banned 'cache node' will never inserted into the 'cache tree'. Ban
+// only attributed to the particular 'cache node', so when a 'cache node'
+// is recreated it will not be banned.
+//
+// If onDel is not nil, then it will be executed if such 'cache node'
+// doesn't exist or once the 'cache node' is released.
+//
+// Delete return true is such 'cache node' exist.
+func (r *Cache) Delete(ns, key uint64, onDel func()) bool {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+	if r.closed {
+		return false
+	}
+
+	hash := murmur32(ns, key, 0xf00)
+	for {
+		h, b := r.getBucket(hash)
+		done, _, n := b.get(r, h, hash, ns, key, true)
+		if done {
+			if n != nil {
+				if onDel != nil {
+					n.mu.Lock()
+					n.onDel = append(n.onDel, onDel)
+					n.mu.Unlock()
+				}
+				if r.cacher != nil {
+					r.cacher.Ban(n)
+				}
+				n.unref()
+				return true
+			}
+
+			break
+		}
+	}
+
+	if onDel != nil {
+		onDel()
+	}
+
+	return false
+}
+
+// Evict evicts 'cache node' with the given namespace and key. This will
+// simply call Cacher.Evict.
+//
+// Evict return true is such 'cache node' exist.
+func (r *Cache) Evict(ns, key uint64) bool {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+	if r.closed {
+		return false
+	}
+
+	hash := murmur32(ns, key, 0xf00)
+	for {
+		h, b := r.getBucket(hash)
+		done, _, n := b.get(r, h, hash, ns, key, true)
+		if done {
+			if n != nil {
+				if r.cacher != nil {
+					r.cacher.Evict(n)
+				}
+				n.unref()
+				return true
+			}
+
+			break
+		}
+	}
+
+	return false
+}
+
+// EvictNS evicts 'cache node' with the given namespace. This will
+// simply call Cacher.EvictNS.
+func (r *Cache) EvictNS(ns uint64) {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+	if r.closed {
 		return
 	}
-	if h.fin != nil {
-		h.fin()
-		h.fin = nil
+
+	if r.cacher != nil {
+		r.cacher.EvictNS(ns)
 	}
 }
+
+// EvictAll evicts all 'cache node'. This will simply call Cacher.EvictAll.
+func (r *Cache) EvictAll() {
+	r.mu.RLock()
+	defer r.mu.RUnlock()
+	if r.closed {
+		return
+	}
+
+	if r.cacher != nil {
+		r.cacher.EvictAll()
+	}
+}
+
+// Close closes the 'cache map' and releases all 'cache node'.
+func (r *Cache) Close() error {
+	r.mu.Lock()
+	if !r.closed {
+		r.closed = true
+
+		if r.cacher != nil {
+			if err := r.cacher.Close(); err != nil {
+				return err
+			}
+		}
+
+		h := (*mNode)(r.mHead)
+		h.initBuckets()
+
+		for i := range h.buckets {
+			b := (*mBucket)(h.buckets[i])
+			for _, n := range b.node {
+				// Call releaser.
+				if n.value != nil {
+					if r, ok := n.value.(util.Releaser); ok {
+						r.Release()
+					}
+					n.value = nil
+				}
+
+				// Call OnDel.
+				for _, f := range n.onDel {
+					f()
+				}
+			}
+		}
+	}
+	r.mu.Unlock()
+	return nil
+}
+
+// Node is a 'cache node'.
+type Node struct {
+	r *Cache
+
+	hash    uint32
+	ns, key uint64
+
+	mu    sync.Mutex
+	size  int
+	value Value
+
+	ref   int32
+	onDel []func()
+
+	CacheData unsafe.Pointer
+}
+
+// NS returns this 'cache node' namespace.
+func (n *Node) NS() uint64 {
+	return n.ns
+}
+
+// Key returns this 'cache node' key.
+func (n *Node) Key() uint64 {
+	return n.key
+}
+
+// Size returns this 'cache node' size.
+func (n *Node) Size() int {
+	return n.size
+}
+
+// Value returns this 'cache node' value.
+func (n *Node) Value() Value {
+	return n.value
+}
+
+// Ref returns this 'cache node' ref counter.
+func (n *Node) Ref() int32 {
+	return atomic.LoadInt32(&n.ref)
+}
+
+// GetHandle returns an handle for this 'cache node'.
+func (n *Node) GetHandle() *Handle {
+	if atomic.AddInt32(&n.ref, 1) <= 1 {
+		panic("BUG: Node.GetHandle on zero ref")
+	}
+	return &Handle{unsafe.Pointer(n)}
+}
+
+func (n *Node) unref() {
+	if atomic.AddInt32(&n.ref, -1) == 0 {
+		n.r.delete(n)
+	}
+}
+
+func (n *Node) unrefLocked() {
+	if atomic.AddInt32(&n.ref, -1) == 0 {
+		n.r.mu.RLock()
+		if !n.r.closed {
+			n.r.delete(n)
+		}
+		n.r.mu.RUnlock()
+	}
+}
+
+type Handle struct {
+	n unsafe.Pointer // *Node
+}
+
+func (h *Handle) Value() Value {
+	n := (*Node)(atomic.LoadPointer(&h.n))
+	if n != nil {
+		return n.value
+	}
+	return nil
+}
+
+func (h *Handle) Release() {
+	nPtr := atomic.LoadPointer(&h.n)
+	if nPtr != nil && atomic.CompareAndSwapPointer(&h.n, nPtr, nil) {
+		n := (*Node)(nPtr)
+		n.unrefLocked()
+	}
+}
+
+func murmur32(ns, key uint64, seed uint32) uint32 {
+	const (
+		m = uint32(0x5bd1e995)
+		r = 24
+	)
+
+	k1 := uint32(ns >> 32)
+	k2 := uint32(ns)
+	k3 := uint32(key >> 32)
+	k4 := uint32(key)
+
+	k1 *= m
+	k1 ^= k1 >> r
+	k1 *= m
+
+	k2 *= m
+	k2 ^= k2 >> r
+	k2 *= m
+
+	k3 *= m
+	k3 ^= k3 >> r
+	k3 *= m
+
+	k4 *= m
+	k4 ^= k4 >> r
+	k4 *= m
+
+	h := seed
+
+	h *= m
+	h ^= k1
+	h *= m
+	h ^= k2
+	h *= m
+	h ^= k3
+	h *= m
+	h ^= k4
+
+	h ^= h >> 13
+	h *= m
+	h ^= h >> 15
+
+	return h
+}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/cache/cache_test.go

@@ -13,11 +13,26 @@ import (
 	"sync/atomic"
 	"testing"
 	"time"
+	"unsafe"
 )
 
+type int32o int32
+
+func (o *int32o) acquire() {
+	if atomic.AddInt32((*int32)(o), 1) != 1 {
+		panic("BUG: invalid ref")
+	}
+}
+
+func (o *int32o) Release() {
+	if atomic.AddInt32((*int32)(o), -1) != 0 {
+		panic("BUG: invalid ref")
+	}
+}
+
 type releaserFunc struct {
 	fn    func()
-	value interface{}
+	value Value
 }
 
 func (r releaserFunc) Release() {
@@ -26,8 +41,8 @@ func (r releaserFunc) Release() {
 	}
 }
 
-func set(ns Namespace, key uint64, value interface{}, charge int, relf func()) Handle {
-	return ns.Get(key, func() (int, interface{}) {
+func set(c *Cache, ns, key uint64, value Value, charge int, relf func()) *Handle {
+	return c.Get(ns, key, func() (int, Value) {
 		if relf != nil {
 			return charge, releaserFunc{relf, value}
 		} else {
@@ -36,7 +51,246 @@ func set(ns Namespace, key uint64, value interface{}, charge int, relf func()) H
 	})
 }
 
-func TestCache_HitMiss(t *testing.T) {
+func TestCacheMap(t *testing.T) {
+	runtime.GOMAXPROCS(runtime.NumCPU())
+
+	nsx := []struct {
+		nobjects, nhandles, concurrent, repeat int
+	}{
+		{10000, 400, 50, 3},
+		{100000, 1000, 100, 10},
+	}
+
+	var (
+		objects [][]int32o
+		handles [][]unsafe.Pointer
+	)
+
+	for _, x := range nsx {
+		objects = append(objects, make([]int32o, x.nobjects))
+		handles = append(handles, make([]unsafe.Pointer, x.nhandles))
+	}
+
+	c := NewCache(nil)
+
+	wg := new(sync.WaitGroup)
+	var done int32
+
+	for ns, x := range nsx {
+		for i := 0; i < x.concurrent; i++ {
+			wg.Add(1)
+			go func(ns, i, repeat int, objects []int32o, handles []unsafe.Pointer) {
+				defer wg.Done()
+				r := rand.New(rand.NewSource(time.Now().UnixNano()))
+
+				for j := len(objects) * repeat; j >= 0; j-- {
+					key := uint64(r.Intn(len(objects)))
+					h := c.Get(uint64(ns), key, func() (int, Value) {
+						o := &objects[key]
+						o.acquire()
+						return 1, o
+					})
+					if v := h.Value().(*int32o); v != &objects[key] {
+						t.Fatalf("#%d invalid value: want=%p got=%p", ns, &objects[key], v)
+					}
+					if objects[key] != 1 {
+						t.Fatalf("#%d invalid object %d: %d", ns, key, objects[key])
+					}
+					if !atomic.CompareAndSwapPointer(&handles[r.Intn(len(handles))], nil, unsafe.Pointer(h)) {
+						h.Release()
+					}
+				}
+			}(ns, i, x.repeat, objects[ns], handles[ns])
+		}
+
+		go func(handles []unsafe.Pointer) {
+			r := rand.New(rand.NewSource(time.Now().UnixNano()))
+
+			for atomic.LoadInt32(&done) == 0 {
+				i := r.Intn(len(handles))
+				h := (*Handle)(atomic.LoadPointer(&handles[i]))
+				if h != nil && atomic.CompareAndSwapPointer(&handles[i], unsafe.Pointer(h), nil) {
+					h.Release()
+				}
+				time.Sleep(time.Millisecond)
+			}
+		}(handles[ns])
+	}
+
+	go func() {
+		handles := make([]*Handle, 100000)
+		for atomic.LoadInt32(&done) == 0 {
+			for i := range handles {
+				handles[i] = c.Get(999999999, uint64(i), func() (int, Value) {
+					return 1, 1
+				})
+			}
+			for _, h := range handles {
+				h.Release()
+			}
+		}
+	}()
+
+	wg.Wait()
+
+	atomic.StoreInt32(&done, 1)
+
+	for _, handles0 := range handles {
+		for i := range handles0 {
+			h := (*Handle)(atomic.LoadPointer(&handles0[i]))
+			if h != nil && atomic.CompareAndSwapPointer(&handles0[i], unsafe.Pointer(h), nil) {
+				h.Release()
+			}
+		}
+	}
+
+	for ns, objects0 := range objects {
+		for i, o := range objects0 {
+			if o != 0 {
+				t.Fatalf("invalid object #%d.%d: ref=%d", ns, i, o)
+			}
+		}
+	}
+}
+
+func TestCacheMap_NodesAndSize(t *testing.T) {
+	c := NewCache(nil)
+	if c.Nodes() != 0 {
+		t.Errorf("invalid nodes counter: want=%d got=%d", 0, c.Nodes())
+	}
+	if c.Size() != 0 {
+		t.Errorf("invalid size counter: want=%d got=%d", 0, c.Size())
+	}
+	set(c, 0, 1, 1, 1, nil)
+	set(c, 0, 2, 2, 2, nil)
+	set(c, 1, 1, 3, 3, nil)
+	set(c, 2, 1, 4, 1, nil)
+	if c.Nodes() != 4 {
+		t.Errorf("invalid nodes counter: want=%d got=%d", 4, c.Nodes())
+	}
+	if c.Size() != 7 {
+		t.Errorf("invalid size counter: want=%d got=%d", 4, c.Size())
+	}
+}
+
+func TestLRUCache_Capacity(t *testing.T) {
+	c := NewCache(NewLRU(10))
+	if c.Capacity() != 10 {
+		t.Errorf("invalid capacity: want=%d got=%d", 10, c.Capacity())
+	}
+	set(c, 0, 1, 1, 1, nil).Release()
+	set(c, 0, 2, 2, 2, nil).Release()
+	set(c, 1, 1, 3, 3, nil).Release()
+	set(c, 2, 1, 4, 1, nil).Release()
+	set(c, 2, 2, 5, 1, nil).Release()
+	set(c, 2, 3, 6, 1, nil).Release()
+	set(c, 2, 4, 7, 1, nil).Release()
+	set(c, 2, 5, 8, 1, nil).Release()
+	if c.Nodes() != 7 {
+		t.Errorf("invalid nodes counter: want=%d got=%d", 7, c.Nodes())
+	}
+	if c.Size() != 10 {
+		t.Errorf("invalid size counter: want=%d got=%d", 10, c.Size())
+	}
+	c.SetCapacity(9)
+	if c.Capacity() != 9 {
+		t.Errorf("invalid capacity: want=%d got=%d", 9, c.Capacity())
+	}
+	if c.Nodes() != 6 {
+		t.Errorf("invalid nodes counter: want=%d got=%d", 6, c.Nodes())
+	}
+	if c.Size() != 8 {
+		t.Errorf("invalid size counter: want=%d got=%d", 8, c.Size())
+	}
+}
+
+func TestCacheMap_NilValue(t *testing.T) {
+	c := NewCache(NewLRU(10))
+	h := c.Get(0, 0, func() (size int, value Value) {
+		return 1, nil
+	})
+	if h != nil {
+		t.Error("cache handle is non-nil")
+	}
+	if c.Nodes() != 0 {
+		t.Errorf("invalid nodes counter: want=%d got=%d", 0, c.Nodes())
+	}
+	if c.Size() != 0 {
+		t.Errorf("invalid size counter: want=%d got=%d", 0, c.Size())
+	}
+}
+
+func TestLRUCache_GetLatency(t *testing.T) {
+	runtime.GOMAXPROCS(runtime.NumCPU())
+
+	const (
+		concurrentSet = 30
+		concurrentGet = 3
+		duration      = 3 * time.Second
+		delay         = 3 * time.Millisecond
+		maxkey        = 100000
+	)
+
+	var (
+		set, getHit, getAll        int32
+		getMaxLatency, getDuration int64
+	)
+
+	c := NewCache(NewLRU(5000))
+	wg := &sync.WaitGroup{}
+	until := time.Now().Add(duration)
+	for i := 0; i < concurrentSet; i++ {
+		wg.Add(1)
+		go func(i int) {
+			defer wg.Done()
+			r := rand.New(rand.NewSource(time.Now().UnixNano()))
+			for time.Now().Before(until) {
+				c.Get(0, uint64(r.Intn(maxkey)), func() (int, Value) {
+					time.Sleep(delay)
+					atomic.AddInt32(&set, 1)
+					return 1, 1
+				}).Release()
+			}
+		}(i)
+	}
+	for i := 0; i < concurrentGet; i++ {
+		wg.Add(1)
+		go func(i int) {
+			defer wg.Done()
+			r := rand.New(rand.NewSource(time.Now().UnixNano()))
+			for {
+				mark := time.Now()
+				if mark.Before(until) {
+					h := c.Get(0, uint64(r.Intn(maxkey)), nil)
+					latency := int64(time.Now().Sub(mark))
+					m := atomic.LoadInt64(&getMaxLatency)
+					if latency > m {
+						atomic.CompareAndSwapInt64(&getMaxLatency, m, latency)
+					}
+					atomic.AddInt64(&getDuration, latency)
+					if h != nil {
+						atomic.AddInt32(&getHit, 1)
+						h.Release()
+					}
+					atomic.AddInt32(&getAll, 1)
+				} else {
+					break
+				}
+			}
+		}(i)
+	}
+
+	wg.Wait()
+	getAvglatency := time.Duration(getDuration) / time.Duration(getAll)
+	t.Logf("set=%d getHit=%d getAll=%d getMaxLatency=%v getAvgLatency=%v",
+		set, getHit, getAll, time.Duration(getMaxLatency), getAvglatency)
+
+	if getAvglatency > delay/3 {
+		t.Errorf("get avg latency > %v: got=%v", delay/3, getAvglatency)
+	}
+}
+
+func TestLRUCache_HitMiss(t *testing.T) {
 	cases := []struct {
 		key   uint64
 		value string
@@ -54,14 +308,13 @@ func TestCache_HitMiss(t *testing.T) {
 	}
 
 	setfin := 0
-	c := NewLRUCache(1000)
-	ns := c.GetNamespace(0)
+	c := NewCache(NewLRU(1000))
 	for i, x := range cases {
-		set(ns, x.key, x.value, len(x.value), func() {
+		set(c, 0, x.key, x.value, len(x.value), func() {
 			setfin++
 		}).Release()
 		for j, y := range cases {
-			h := ns.Get(y.key, nil)
+			h := c.Get(0, y.key, nil)
 			if j <= i {
 				// should hit
 				if h == nil {
@@ -85,7 +338,7 @@ func TestCache_HitMiss(t *testing.T) {
 
 	for i, x := range cases {
 		finalizerOk := false
-		ns.Delete(x.key, func(exist, pending bool) {
+		c.Delete(0, x.key, func() {
 			finalizerOk = true
 		})
 
@@ -94,7 +347,7 @@ func TestCache_HitMiss(t *testing.T) {
 		}
 
 		for j, y := range cases {
-			h := ns.Get(y.key, nil)
+			h := c.Get(0, y.key, nil)
 			if j > i {
 				// should hit
 				if h == nil {
@@ -122,20 +375,19 @@ func TestCache_HitMiss(t *testing.T) {
 }
 
 func TestLRUCache_Eviction(t *testing.T) {
-	c := NewLRUCache(12)
-	ns := c.GetNamespace(0)
-	o1 := set(ns, 1, 1, 1, nil)
-	set(ns, 2, 2, 1, nil).Release()
-	set(ns, 3, 3, 1, nil).Release()
-	set(ns, 4, 4, 1, nil).Release()
-	set(ns, 5, 5, 1, nil).Release()
-	if h := ns.Get(2, nil); h != nil { // 1,3,4,5,2
+	c := NewCache(NewLRU(12))
+	o1 := set(c, 0, 1, 1, 1, nil)
+	set(c, 0, 2, 2, 1, nil).Release()
+	set(c, 0, 3, 3, 1, nil).Release()
+	set(c, 0, 4, 4, 1, nil).Release()
+	set(c, 0, 5, 5, 1, nil).Release()
+	if h := c.Get(0, 2, nil); h != nil { // 1,3,4,5,2
 		h.Release()
 	}
-	set(ns, 9, 9, 10, nil).Release() // 5,2,9
+	set(c, 0, 9, 9, 10, nil).Release() // 5,2,9
 
 	for _, key := range []uint64{9, 2, 5, 1} {
-		h := ns.Get(key, nil)
+		h := c.Get(0, key, nil)
 		if h == nil {
 			t.Errorf("miss for key '%d'", key)
 		} else {
@@ -147,7 +399,7 @@ func TestLRUCache_Eviction(t *testing.T) {
 	}
 	o1.Release()
 	for _, key := range []uint64{1, 2, 5} {
-		h := ns.Get(key, nil)
+		h := c.Get(0, key, nil)
 		if h == nil {
 			t.Errorf("miss for key '%d'", key)
 		} else {
@@ -158,7 +410,7 @@ func TestLRUCache_Eviction(t *testing.T) {
 		}
 	}
 	for _, key := range []uint64{3, 4, 9} {
-		h := ns.Get(key, nil)
+		h := c.Get(0, key, nil)
 		if h != nil {
 			t.Errorf("hit for key '%d'", key)
 			if x := h.Value().(int); x != int(key) {
@@ -169,487 +421,134 @@ func TestLRUCache_Eviction(t *testing.T) {
 	}
 }
 
-func TestLRUCache_SetGet(t *testing.T) {
-	c := NewLRUCache(13)
-	ns := c.GetNamespace(0)
-	for i := 0; i < 200; i++ {
-		n := uint64(rand.Intn(99999) % 20)
-		set(ns, n, n, 1, nil).Release()
-		if h := ns.Get(n, nil); h != nil {
-			if h.Value() == nil {
-				t.Errorf("key '%d' contains nil value", n)
+func TestLRUCache_Evict(t *testing.T) {
+	c := NewCache(NewLRU(6))
+	set(c, 0, 1, 1, 1, nil).Release()
+	set(c, 0, 2, 2, 1, nil).Release()
+	set(c, 1, 1, 4, 1, nil).Release()
+	set(c, 1, 2, 5, 1, nil).Release()
+	set(c, 2, 1, 6, 1, nil).Release()
+	set(c, 2, 2, 7, 1, nil).Release()
+
+	for ns := 0; ns < 3; ns++ {
+		for key := 1; key < 3; key++ {
+			if h := c.Get(uint64(ns), uint64(key), nil); h != nil {
+				h.Release()
 			} else {
-				if x := h.Value().(uint64); x != n {
-					t.Errorf("invalid value for key '%d' want '%d', got '%d'", n, n, x)
-				}
+				t.Errorf("Cache.Get on #%d.%d return nil", ns, key)
 			}
+		}
+	}
+
+	if ok := c.Evict(0, 1); !ok {
+		t.Error("first Cache.Evict on #0.1 return false")
+	}
+	if ok := c.Evict(0, 1); ok {
+		t.Error("second Cache.Evict on #0.1 return true")
+	}
+	if h := c.Get(0, 1, nil); h != nil {
+		t.Errorf("Cache.Get on #0.1 return non-nil: %v", h.Value())
+	}
+
+	c.EvictNS(1)
+	if h := c.Get(1, 1, nil); h != nil {
+		t.Errorf("Cache.Get on #1.1 return non-nil: %v", h.Value())
+	}
+	if h := c.Get(1, 2, nil); h != nil {
+		t.Errorf("Cache.Get on #1.2 return non-nil: %v", h.Value())
+	}
+
+	c.EvictAll()
+	for ns := 0; ns < 3; ns++ {
+		for key := 1; key < 3; key++ {
+			if h := c.Get(uint64(ns), uint64(key), nil); h != nil {
+				t.Errorf("Cache.Get on #%d.%d return non-nil: %v", ns, key, h.Value())
+			}
+		}
+	}
+}
+
+func TestLRUCache_Delete(t *testing.T) {
+	delFuncCalled := 0
+	delFunc := func() {
+		delFuncCalled++
+	}
+
+	c := NewCache(NewLRU(2))
+	set(c, 0, 1, 1, 1, nil).Release()
+	set(c, 0, 2, 2, 1, nil).Release()
+
+	if ok := c.Delete(0, 1, delFunc); !ok {
+		t.Error("Cache.Delete on #1 return false")
+	}
+	if h := c.Get(0, 1, nil); h != nil {
+		t.Errorf("Cache.Get on #1 return non-nil: %v", h.Value())
+	}
+	if ok := c.Delete(0, 1, delFunc); ok {
+		t.Error("Cache.Delete on #1 return true")
+	}
+
+	h2 := c.Get(0, 2, nil)
+	if h2 == nil {
+		t.Error("Cache.Get on #2 return nil")
+	}
+	if ok := c.Delete(0, 2, delFunc); !ok {
+		t.Error("(1) Cache.Delete on #2 return false")
+	}
+	if ok := c.Delete(0, 2, delFunc); !ok {
+		t.Error("(2) Cache.Delete on #2 return false")
+	}
+
+	set(c, 0, 3, 3, 1, nil).Release()
+	set(c, 0, 4, 4, 1, nil).Release()
+	c.Get(0, 2, nil).Release()
+
+	for key := 2; key <= 4; key++ {
+		if h := c.Get(0, uint64(key), nil); h != nil {
 			h.Release()
 		} else {
-			t.Errorf("key '%d' doesn't exist", n)
-		}
-	}
-}
-
-func TestLRUCache_Purge(t *testing.T) {
-	c := NewLRUCache(3)
-	ns1 := c.GetNamespace(0)
-	o1 := set(ns1, 1, 1, 1, nil)
-	o2 := set(ns1, 2, 2, 1, nil)
-	ns1.Purge(nil)
-	set(ns1, 3, 3, 1, nil).Release()
-	for _, key := range []uint64{1, 2, 3} {
-		h := ns1.Get(key, nil)
-		if h == nil {
-			t.Errorf("miss for key '%d'", key)
-		} else {
-			if x := h.Value().(int); x != int(key) {
-				t.Errorf("invalid value for key '%d' want '%d', got '%d'", key, key, x)
-			}
-			h.Release()
-		}
-	}
-	o1.Release()
-	o2.Release()
-	for _, key := range []uint64{1, 2} {
-		h := ns1.Get(key, nil)
-		if h != nil {
-			t.Errorf("hit for key '%d'", key)
-			if x := h.Value().(int); x != int(key) {
-				t.Errorf("invalid value for key '%d' want '%d', got '%d'", key, key, x)
-			}
-			h.Release()
-		}
-	}
-}
-
-type testingCacheObjectCounter struct {
-	created  uint
-	released uint
-}
-
-func (c *testingCacheObjectCounter) createOne() {
-	c.created++
-}
-
-func (c *testingCacheObjectCounter) releaseOne() {
-	c.released++
-}
-
-type testingCacheObject struct {
-	t   *testing.T
-	cnt *testingCacheObjectCounter
-
-	ns, key uint64
-
-	releaseCalled bool
-}
-
-func (x *testingCacheObject) Release() {
-	if !x.releaseCalled {
-		x.releaseCalled = true
-		x.cnt.releaseOne()
-	} else {
-		x.t.Errorf("duplicate setfin NS#%d KEY#%d", x.ns, x.key)
-	}
-}
-
-func TestLRUCache_ConcurrentSetGet(t *testing.T) {
-	runtime.GOMAXPROCS(runtime.NumCPU())
-
-	seed := time.Now().UnixNano()
-	t.Logf("seed=%d", seed)
-
-	const (
-		N = 2000000
-		M = 4000
-		C = 3
-	)
-
-	var set, get uint32
-
-	wg := &sync.WaitGroup{}
-	c := NewLRUCache(M / 4)
-	for ni := uint64(0); ni < C; ni++ {
-		r0 := rand.New(rand.NewSource(seed + int64(ni)))
-		r1 := rand.New(rand.NewSource(seed + int64(ni) + 1))
-		ns := c.GetNamespace(ni)
-
-		wg.Add(2)
-		go func(ns Namespace, r *rand.Rand) {
-			for i := 0; i < N; i++ {
-				x := uint64(r.Int63n(M))
-				o := ns.Get(x, func() (int, interface{}) {
-					atomic.AddUint32(&set, 1)
-					return 1, x
-				})
-				if v := o.Value().(uint64); v != x {
-					t.Errorf("#%d invalid value, got=%d", x, v)
-				}
-				o.Release()
-			}
-			wg.Done()
-		}(ns, r0)
-		go func(ns Namespace, r *rand.Rand) {
-			for i := 0; i < N; i++ {
-				x := uint64(r.Int63n(M))
-				o := ns.Get(x, nil)
-				if o != nil {
-					atomic.AddUint32(&get, 1)
-					if v := o.Value().(uint64); v != x {
-						t.Errorf("#%d invalid value, got=%d", x, v)
-					}
-					o.Release()
-				}
-			}
-			wg.Done()
-		}(ns, r1)
-	}
-
-	wg.Wait()
-
-	t.Logf("set=%d get=%d", set, get)
-}
-
-func TestLRUCache_Finalizer(t *testing.T) {
-	const (
-		capacity   = 100
-		goroutines = 100
-		iterations = 10000
-		keymax     = 8000
-	)
-
-	cnt := &testingCacheObjectCounter{}
-
-	c := NewLRUCache(capacity)
-
-	type instance struct {
-		seed       int64
-		rnd        *rand.Rand
-		nsid       uint64
-		ns         Namespace
-		effective  int
-		handles    []Handle
-		handlesMap map[uint64]int
-
-		delete          bool
-		purge           bool
-		zap             bool
-		wantDel         int
-		delfinCalled    int
-		delfinCalledAll int
-		delfinCalledEff int
-		purgefinCalled  int
-	}
-
-	instanceGet := func(p *instance, key uint64) {
-		h := p.ns.Get(key, func() (charge int, value interface{}) {
-			to := &testingCacheObject{
-				t: t, cnt: cnt,
-				ns:  p.nsid,
-				key: key,
-			}
-			p.effective++
-			cnt.createOne()
-			return 1, releaserFunc{func() {
-				to.Release()
-				p.effective--
-			}, to}
-		})
-		p.handles = append(p.handles, h)
-		p.handlesMap[key] = p.handlesMap[key] + 1
-	}
-	instanceRelease := func(p *instance, i int) {
-		h := p.handles[i]
-		key := h.Value().(releaserFunc).value.(*testingCacheObject).key
-		if n := p.handlesMap[key]; n == 0 {
-			t.Fatal("key ref == 0")
-		} else if n > 1 {
-			p.handlesMap[key] = n - 1
-		} else {
-			delete(p.handlesMap, key)
-		}
-		h.Release()
-		p.handles = append(p.handles[:i], p.handles[i+1:]...)
-		p.handles[len(p.handles) : len(p.handles)+1][0] = nil
-	}
-
-	seed := time.Now().UnixNano()
-	t.Logf("seed=%d", seed)
-
-	instances := make([]*instance, goroutines)
-	for i := range instances {
-		p := &instance{}
-		p.handlesMap = make(map[uint64]int)
-		p.seed = seed + int64(i)
-		p.rnd = rand.New(rand.NewSource(p.seed))
-		p.nsid = uint64(i)
-		p.ns = c.GetNamespace(p.nsid)
-		p.delete = i%6 == 0
-		p.purge = i%8 == 0
-		p.zap = i%12 == 0 || i%3 == 0
-		instances[i] = p
-	}
-
-	runr := rand.New(rand.NewSource(seed - 1))
-	run := func(rnd *rand.Rand, x []*instance, init func(p *instance) bool, fn func(p *instance, i int) bool) {
-		var (
-			rx []*instance
-			rn []int
-		)
-		if init == nil {
-			rx = append([]*instance{}, x...)
-			rn = make([]int, len(x))
-		} else {
-			for _, p := range x {
-				if init(p) {
-					rx = append(rx, p)
-					rn = append(rn, 0)
-				}
-			}
-		}
-		for len(rx) > 0 {
-			i := rand.Intn(len(rx))
-			if fn(rx[i], rn[i]) {
-				rn[i]++
-			} else {
-				rx = append(rx[:i], rx[i+1:]...)
-				rn = append(rn[:i], rn[i+1:]...)
-			}
+			t.Errorf("Cache.Get on #%d return nil", key)
 		}
 	}
 
-	// Get and release.
-	run(runr, instances, nil, func(p *instance, i int) bool {
-		if i < iterations {
-			if len(p.handles) == 0 || p.rnd.Int()%2 == 0 {
-				instanceGet(p, uint64(p.rnd.Intn(keymax)))
-			} else {
-				instanceRelease(p, p.rnd.Intn(len(p.handles)))
-			}
-			return true
-		} else {
-			return false
-		}
-	})
-
-	if used, cap := c.Used(), c.Capacity(); used > cap {
-		t.Errorf("Used > capacity, used=%d cap=%d", used, cap)
+	h2.Release()
+	if h := c.Get(0, 2, nil); h != nil {
+		t.Errorf("Cache.Get on #2 return non-nil: %v", h.Value())
 	}
 
-	// Check effective objects.
-	for i, p := range instances {
-		if int(p.effective) < len(p.handlesMap) {
-			t.Errorf("#%d effective objects < acquired handle, eo=%d ah=%d", i, p.effective, len(p.handlesMap))
-		}
-	}
-
-	if want := int(cnt.created - cnt.released); c.Size() != want {
-		t.Errorf("Invalid cache size, want=%d got=%d", want, c.Size())
-	}
-
-	// First delete.
-	run(runr, instances, func(p *instance) bool {
-		p.wantDel = p.effective
-		return p.delete
-	}, func(p *instance, i int) bool {
-		key := uint64(i)
-		if key < keymax {
-			_, wantExist := p.handlesMap[key]
-			gotExist := p.ns.Delete(key, func(exist, pending bool) {
-				p.delfinCalledAll++
-				if exist {
-					p.delfinCalledEff++
-				}
-			})
-			if !gotExist && wantExist {
-				t.Errorf("delete on NS#%d KEY#%d not found", p.nsid, key)
-			}
-			return true
-		} else {
-			return false
-		}
-	})
-
-	// Second delete.
-	run(runr, instances, func(p *instance) bool {
-		p.delfinCalled = 0
-		return p.delete
-	}, func(p *instance, i int) bool {
-		key := uint64(i)
-		if key < keymax {
-			gotExist := p.ns.Delete(key, func(exist, pending bool) {
-				if exist && !pending {
-					t.Errorf("delete fin on NS#%d KEY#%d exist and not pending for deletion", p.nsid, key)
-				}
-				p.delfinCalled++
-			})
-			if gotExist {
-				t.Errorf("delete on NS#%d KEY#%d found", p.nsid, key)
-			}
-			return true
-		} else {
-			if p.delfinCalled != keymax {
-				t.Errorf("(2) NS#%d not all delete fin called, diff=%d", p.nsid, keymax-p.delfinCalled)
-			}
-			return false
-		}
-	})
-
-	// Purge.
-	run(runr, instances, func(p *instance) bool {
-		return p.purge
-	}, func(p *instance, i int) bool {
-		p.ns.Purge(func(ns, key uint64) {
-			p.purgefinCalled++
-		})
-		return false
-	})
-
-	if want := int(cnt.created - cnt.released); c.Size() != want {
-		t.Errorf("Invalid cache size, want=%d got=%d", want, c.Size())
-	}
-
-	// Release.
-	run(runr, instances, func(p *instance) bool {
-		return !p.zap
-	}, func(p *instance, i int) bool {
-		if len(p.handles) > 0 {
-			instanceRelease(p, len(p.handles)-1)
-			return true
-		} else {
-			return false
-		}
-	})
-
-	if want := int(cnt.created - cnt.released); c.Size() != want {
-		t.Errorf("Invalid cache size, want=%d got=%d", want, c.Size())
-	}
-
-	// Zap.
-	run(runr, instances, func(p *instance) bool {
-		return p.zap
-	}, func(p *instance, i int) bool {
-		p.ns.Zap()
-		p.handles = nil
-		p.handlesMap = nil
-		return false
-	})
-
-	if want := int(cnt.created - cnt.released); c.Size() != want {
-		t.Errorf("Invalid cache size, want=%d got=%d", want, c.Size())
-	}
-
-	if notrel, used := int(cnt.created-cnt.released), c.Used(); notrel != used {
-		t.Errorf("Invalid used value, want=%d got=%d", notrel, used)
-	}
-
-	c.Purge(nil)
-
-	for _, p := range instances {
-		if p.delete {
-			if p.delfinCalledAll != keymax {
-				t.Errorf("#%d not all delete fin called, purge=%v zap=%v diff=%d", p.nsid, p.purge, p.zap, keymax-p.delfinCalledAll)
-			}
-			if p.delfinCalledEff != p.wantDel {
-				t.Errorf("#%d not all effective delete fin called, diff=%d", p.nsid, p.wantDel-p.delfinCalledEff)
-			}
-			if p.purge && p.purgefinCalled > 0 {
-				t.Errorf("#%d some purge fin called, delete=%v zap=%v n=%d", p.nsid, p.delete, p.zap, p.purgefinCalled)
-			}
-		} else {
-			if p.purge {
-				if p.purgefinCalled != p.wantDel {
-					t.Errorf("#%d not all purge fin called, delete=%v zap=%v diff=%d", p.nsid, p.delete, p.zap, p.wantDel-p.purgefinCalled)
-				}
-			}
-		}
-	}
-
-	if cnt.created != cnt.released {
-		t.Errorf("Some cache object weren't released, created=%d released=%d", cnt.created, cnt.released)
+	if delFuncCalled != 4 {
+		t.Errorf("delFunc isn't called 4 times: got=%d", delFuncCalled)
 	}
 }
 
-func BenchmarkLRUCache_Set(b *testing.B) {
-	c := NewLRUCache(0)
-	ns := c.GetNamespace(0)
-	b.ResetTimer()
-	for i := uint64(0); i < uint64(b.N); i++ {
-		set(ns, i, "", 1, nil)
-	}
-}
-
-func BenchmarkLRUCache_Get(b *testing.B) {
-	c := NewLRUCache(0)
-	ns := c.GetNamespace(0)
-	b.ResetTimer()
-	for i := uint64(0); i < uint64(b.N); i++ {
-		set(ns, i, "", 1, nil)
-	}
-	b.ResetTimer()
-	for i := uint64(0); i < uint64(b.N); i++ {
-		ns.Get(i, nil)
-	}
-}
-
-func BenchmarkLRUCache_Get2(b *testing.B) {
-	c := NewLRUCache(0)
-	ns := c.GetNamespace(0)
-	b.ResetTimer()
-	for i := uint64(0); i < uint64(b.N); i++ {
-		set(ns, i, "", 1, nil)
-	}
-	b.ResetTimer()
-	for i := uint64(0); i < uint64(b.N); i++ {
-		ns.Get(i, func() (charge int, value interface{}) {
-			return 0, nil
-		})
-	}
-}
-
-func BenchmarkLRUCache_Release(b *testing.B) {
-	c := NewLRUCache(0)
-	ns := c.GetNamespace(0)
-	handles := make([]Handle, b.N)
-	for i := uint64(0); i < uint64(b.N); i++ {
-		handles[i] = set(ns, i, "", 1, nil)
-	}
-	b.ResetTimer()
-	for _, h := range handles {
-		h.Release()
-	}
-}
-
-func BenchmarkLRUCache_SetRelease(b *testing.B) {
-	capacity := b.N / 100
-	if capacity <= 0 {
-		capacity = 10
-	}
-	c := NewLRUCache(capacity)
-	ns := c.GetNamespace(0)
-	b.ResetTimer()
-	for i := uint64(0); i < uint64(b.N); i++ {
-		set(ns, i, "", 1, nil).Release()
-	}
-}
-
-func BenchmarkLRUCache_SetReleaseTwice(b *testing.B) {
-	capacity := b.N / 100
-	if capacity <= 0 {
-		capacity = 10
-	}
-	c := NewLRUCache(capacity)
-	ns := c.GetNamespace(0)
-	b.ResetTimer()
-
-	na := b.N / 2
-	nb := b.N - na
-
-	for i := uint64(0); i < uint64(na); i++ {
-		set(ns, i, "", 1, nil).Release()
-	}
-
-	for i := uint64(0); i < uint64(nb); i++ {
-		set(ns, i, "", 1, nil).Release()
+func TestLRUCache_Close(t *testing.T) {
+	relFuncCalled := 0
+	relFunc := func() {
+		relFuncCalled++
+	}
+	delFuncCalled := 0
+	delFunc := func() {
+		delFuncCalled++
+	}
+
+	c := NewCache(NewLRU(2))
+	set(c, 0, 1, 1, 1, relFunc).Release()
+	set(c, 0, 2, 2, 1, relFunc).Release()
+
+	h3 := set(c, 0, 3, 3, 1, relFunc)
+	if h3 == nil {
+		t.Error("Cache.Get on #3 return nil")
+	}
+	if ok := c.Delete(0, 3, delFunc); !ok {
+		t.Error("Cache.Delete on #3 return false")
+	}
+
+	c.Close()
+
+	if relFuncCalled != 3 {
+		t.Errorf("relFunc isn't called 3 times: got=%d", relFuncCalled)
+	}
+	if delFuncCalled != 1 {
+		t.Errorf("delFunc isn't called 1 times: got=%d", delFuncCalled)
 	}
 }

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/cache/lru.go

@@ -0,0 +1,195 @@
+// Copyright (c) 2012, Suryandaru Triandana <syndtr@gmail.com>
+// All rights reserved.
+//
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+package cache
+
+import (
+	"sync"
+	"unsafe"
+)
+
+type lruNode struct {
+	n   *Node
+	h   *Handle
+	ban bool
+
+	next, prev *lruNode
+}
+
+func (n *lruNode) insert(at *lruNode) {
+	x := at.next
+	at.next = n
+	n.prev = at
+	n.next = x
+	x.prev = n
+}
+
+func (n *lruNode) remove() {
+	if n.prev != nil {
+		n.prev.next = n.next
+		n.next.prev = n.prev
+		n.prev = nil
+		n.next = nil
+	} else {
+		panic("BUG: removing removed node")
+	}
+}
+
+type lru struct {
+	mu       sync.Mutex
+	capacity int
+	used     int
+	recent   lruNode
+}
+
+func (r *lru) reset() {
+	r.recent.next = &r.recent
+	r.recent.prev = &r.recent
+	r.used = 0
+}
+
+func (r *lru) Capacity() int {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return r.capacity
+}
+
+func (r *lru) SetCapacity(capacity int) {
+	var evicted []*lruNode
+
+	r.mu.Lock()
+	r.capacity = capacity
+	for r.used > r.capacity {
+		rn := r.recent.prev
+		if rn == nil {
+			panic("BUG: invalid LRU used or capacity counter")
+		}
+		rn.remove()
+		rn.n.CacheData = nil
+		r.used -= rn.n.Size()
+		evicted = append(evicted, rn)
+	}
+	r.mu.Unlock()
+
+	for _, rn := range evicted {
+		rn.h.Release()
+	}
+}
+
+func (r *lru) Promote(n *Node) {
+	var evicted []*lruNode
+
+	r.mu.Lock()
+	if n.CacheData == nil {
+		if n.Size() <= r.capacity {
+			rn := &lruNode{n: n, h: n.GetHandle()}
+			rn.insert(&r.recent)
+			n.CacheData = unsafe.Pointer(rn)
+			r.used += n.Size()
+
+			for r.used > r.capacity {
+				rn := r.recent.prev
+				if rn == nil {
+					panic("BUG: invalid LRU used or capacity counter")
+				}
+				rn.remove()
+				rn.n.CacheData = nil
+				r.used -= rn.n.Size()
+				evicted = append(evicted, rn)
+			}
+		}
+	} else {
+		rn := (*lruNode)(n.CacheData)
+		if !rn.ban {
+			rn.remove()
+			rn.insert(&r.recent)
+		}
+	}
+	r.mu.Unlock()
+
+	for _, rn := range evicted {
+		rn.h.Release()
+	}
+}
+
+func (r *lru) Ban(n *Node) {
+	r.mu.Lock()
+	if n.CacheData == nil {
+		n.CacheData = unsafe.Pointer(&lruNode{n: n, ban: true})
+	} else {
+		rn := (*lruNode)(n.CacheData)
+		if !rn.ban {
+			rn.remove()
+			rn.ban = true
+			r.used -= rn.n.Size()
+			r.mu.Unlock()
+
+			rn.h.Release()
+			rn.h = nil
+			return
+		}
+	}
+	r.mu.Unlock()
+}
+
+func (r *lru) Evict(n *Node) {
+	r.mu.Lock()
+	rn := (*lruNode)(n.CacheData)
+	if rn == nil || rn.ban {
+		r.mu.Unlock()
+		return
+	}
+	n.CacheData = nil
+	r.mu.Unlock()
+
+	rn.h.Release()
+}
+
+func (r *lru) EvictNS(ns uint64) {
+	var evicted []*lruNode
+
+	r.mu.Lock()
+	for e := r.recent.prev; e != &r.recent; {
+		rn := e
+		e = e.prev
+		if rn.n.NS() == ns {
+			rn.remove()
+			rn.n.CacheData = nil
+			r.used -= rn.n.Size()
+			evicted = append(evicted, rn)
+		}
+	}
+	r.mu.Unlock()
+
+	for _, rn := range evicted {
+		rn.h.Release()
+	}
+}
+
+func (r *lru) EvictAll() {
+	r.mu.Lock()
+	back := r.recent.prev
+	for rn := back; rn != &r.recent; rn = rn.prev {
+		rn.n.CacheData = nil
+	}
+	r.reset()
+	r.mu.Unlock()
+
+	for rn := back; rn != &r.recent; rn = rn.prev {
+		rn.h.Release()
+	}
+}
+
+func (r *lru) Close() error {
+	return nil
+}
+
+// NewLRU create a new LRU-cache.
+func NewLRU(capacity int) Cacher {
+	r := &lru{capacity: capacity}
+	r.reset()
+	return r
+}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/cache/lru_cache.go

@@ -1,622 +0,0 @@
-// Copyright (c) 2012, Suryandaru Triandana <syndtr@gmail.com>
-// All rights reserved.
-//
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file.
-
-package cache
-
-import (
-	"sync"
-	"sync/atomic"
-
-	"github.com/syndtr/goleveldb/leveldb/util"
-)
-
-// The LLRB implementation were taken from https://github.com/petar/GoLLRB.
-// Which contains the following header:
-//
-// Copyright 2010 Petar Maymounkov. All rights reserved.
-// Use of this source code is governed by a BSD-style
-// license that can be found in the LICENSE file.
-
-// lruCache represent a LRU cache state.
-type lruCache struct {
-	mu                sync.Mutex
-	recent            lruNode
-	table             map[uint64]*lruNs
-	capacity          int
-	used, size, alive int
-}
-
-// NewLRUCache creates a new initialized LRU cache with the given capacity.
-func NewLRUCache(capacity int) Cache {
-	c := &lruCache{
-		table:    make(map[uint64]*lruNs),
-		capacity: capacity,
-	}
-	c.recent.rNext = &c.recent
-	c.recent.rPrev = &c.recent
-	return c
-}
-
-func (c *lruCache) Capacity() int {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-	return c.capacity
-}
-
-func (c *lruCache) Used() int {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-	return c.used
-}
-
-func (c *lruCache) Size() int {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-	return c.size
-}
-
-func (c *lruCache) NumObjects() int {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-	return c.alive
-}
-
-// SetCapacity set cache capacity.
-func (c *lruCache) SetCapacity(capacity int) {
-	c.mu.Lock()
-	c.capacity = capacity
-	c.evict()
-	c.mu.Unlock()
-}
-
-// GetNamespace return namespace object for given id.
-func (c *lruCache) GetNamespace(id uint64) Namespace {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-
-	if ns, ok := c.table[id]; ok {
-		return ns
-	}
-
-	ns := &lruNs{lru: c, id: id}
-	c.table[id] = ns
-	return ns
-}
-
-func (c *lruCache) ZapNamespace(id uint64) {
-	c.mu.Lock()
-	if ns, exist := c.table[id]; exist {
-		ns.zapNB()
-		delete(c.table, id)
-	}
-	c.mu.Unlock()
-}
-
-func (c *lruCache) PurgeNamespace(id uint64, fin PurgeFin) {
-	c.mu.Lock()
-	if ns, exist := c.table[id]; exist {
-		ns.purgeNB(fin)
-	}
-	c.mu.Unlock()
-}
-
-// Purge purge entire cache.
-func (c *lruCache) Purge(fin PurgeFin) {
-	c.mu.Lock()
-	for _, ns := range c.table {
-		ns.purgeNB(fin)
-	}
-	c.mu.Unlock()
-}
-
-func (c *lruCache) Zap() {
-	c.mu.Lock()
-	for _, ns := range c.table {
-		ns.zapNB()
-	}
-	c.table = make(map[uint64]*lruNs)
-	c.mu.Unlock()
-}
-
-func (c *lruCache) evict() {
-	top := &c.recent
-	for n := c.recent.rPrev; c.used > c.capacity && n != top; {
-		if n.state != nodeEffective {
-			panic("evicting non effective node")
-		}
-		n.state = nodeEvicted
-		n.rRemove()
-		n.derefNB()
-		c.used -= n.charge
-		n = c.recent.rPrev
-	}
-}
-
-type lruNs struct {
-	lru    *lruCache
-	id     uint64
-	rbRoot *lruNode
-	state  nsState
-}
-
-func (ns *lruNs) rbGetOrCreateNode(h *lruNode, key uint64) (hn, n *lruNode) {
-	if h == nil {
-		n = &lruNode{ns: ns, key: key}
-		return n, n
-	}
-
-	if key < h.key {
-		hn, n = ns.rbGetOrCreateNode(h.rbLeft, key)
-		if hn != nil {
-			h.rbLeft = hn
-		} else {
-			return nil, n
-		}
-	} else if key > h.key {
-		hn, n = ns.rbGetOrCreateNode(h.rbRight, key)
-		if hn != nil {
-			h.rbRight = hn
-		} else {
-			return nil, n
-		}
-	} else {
-		return nil, h
-	}
-
-	if rbIsRed(h.rbRight) && !rbIsRed(h.rbLeft) {
-		h = rbRotLeft(h)
-	}
-	if rbIsRed(h.rbLeft) && rbIsRed(h.rbLeft.rbLeft) {
-		h = rbRotRight(h)
-	}
-	if rbIsRed(h.rbLeft) && rbIsRed(h.rbRight) {
-		rbFlip(h)
-	}
-	return h, n
-}
-
-func (ns *lruNs) getOrCreateNode(key uint64) *lruNode {
-	hn, n := ns.rbGetOrCreateNode(ns.rbRoot, key)
-	if hn != nil {
-		ns.rbRoot = hn
-		ns.rbRoot.rbBlack = true
-	}
-	return n
-}
-
-func (ns *lruNs) rbGetNode(key uint64) *lruNode {
-	h := ns.rbRoot
-	for h != nil {
-		switch {
-		case key < h.key:
-			h = h.rbLeft
-		case key > h.key:
-			h = h.rbRight
-		default:
-			return h
-		}
-	}
-	return nil
-}
-
-func (ns *lruNs) getNode(key uint64) *lruNode {
-	return ns.rbGetNode(key)
-}
-
-func (ns *lruNs) rbDeleteNode(h *lruNode, key uint64) *lruNode {
-	if h == nil {
-		return nil
-	}
-
-	if key < h.key {
-		if h.rbLeft == nil { // key not present. Nothing to delete
-			return h
-		}
-		if !rbIsRed(h.rbLeft) && !rbIsRed(h.rbLeft.rbLeft) {
-			h = rbMoveLeft(h)
-		}
-		h.rbLeft = ns.rbDeleteNode(h.rbLeft, key)
-	} else {
-		if rbIsRed(h.rbLeft) {
-			h = rbRotRight(h)
-		}
-		// If @key equals @h.key and no right children at @h
-		if h.key == key && h.rbRight == nil {
-			return nil
-		}
-		if h.rbRight != nil && !rbIsRed(h.rbRight) && !rbIsRed(h.rbRight.rbLeft) {
-			h = rbMoveRight(h)
-		}
-		// If @key equals @h.key, and (from above) 'h.Right != nil'
-		if h.key == key {
-			var x *lruNode
-			h.rbRight, x = rbDeleteMin(h.rbRight)
-			if x == nil {
-				panic("logic")
-			}
-			x.rbLeft, h.rbLeft = h.rbLeft, nil
-			x.rbRight, h.rbRight = h.rbRight, nil
-			x.rbBlack = h.rbBlack
-			h = x
-		} else { // Else, @key is bigger than @h.key
-			h.rbRight = ns.rbDeleteNode(h.rbRight, key)
-		}
-	}
-
-	return rbFixup(h)
-}
-
-func (ns *lruNs) deleteNode(key uint64) {
-	ns.rbRoot = ns.rbDeleteNode(ns.rbRoot, key)
-	if ns.rbRoot != nil {
-		ns.rbRoot.rbBlack = true
-	}
-}
-
-func (ns *lruNs) rbIterateNodes(h *lruNode, pivot uint64, iter func(n *lruNode) bool) bool {
-	if h == nil {
-		return true
-	}
-	if h.key >= pivot {
-		if !ns.rbIterateNodes(h.rbLeft, pivot, iter) {
-			return false
-		}
-		if !iter(h) {
-			return false
-		}
-	}
-	return ns.rbIterateNodes(h.rbRight, pivot, iter)
-}
-
-func (ns *lruNs) iterateNodes(iter func(n *lruNode) bool) {
-	ns.rbIterateNodes(ns.rbRoot, 0, iter)
-}
-
-func (ns *lruNs) Get(key uint64, setf SetFunc) Handle {
-	ns.lru.mu.Lock()
-	defer ns.lru.mu.Unlock()
-
-	if ns.state != nsEffective {
-		return nil
-	}
-
-	var n *lruNode
-	if setf == nil {
-		n = ns.getNode(key)
-		if n == nil {
-			return nil
-		}
-	} else {
-		n = ns.getOrCreateNode(key)
-	}
-	switch n.state {
-	case nodeZero:
-		charge, value := setf()
-		if value == nil {
-			ns.deleteNode(key)
-			return nil
-		}
-		if charge < 0 {
-			charge = 0
-		}
-
-		n.value = value
-		n.charge = charge
-		n.state = nodeEvicted
-
-		ns.lru.size += charge
-		ns.lru.alive++
-
-		fallthrough
-	case nodeEvicted:
-		if n.charge == 0 {
-			break
-		}
-
-		// Insert to recent list.
-		n.state = nodeEffective
-		n.ref++
-		ns.lru.used += n.charge
-		ns.lru.evict()
-
-		fallthrough
-	case nodeEffective:
-		// Bump to front.
-		n.rRemove()
-		n.rInsert(&ns.lru.recent)
-	case nodeDeleted:
-		// Do nothing.
-	default:
-		panic("invalid state")
-	}
-	n.ref++
-
-	return &lruHandle{node: n}
-}
-
-func (ns *lruNs) Delete(key uint64, fin DelFin) bool {
-	ns.lru.mu.Lock()
-	defer ns.lru.mu.Unlock()
-
-	if ns.state != nsEffective {
-		if fin != nil {
-			fin(false, false)
-		}
-		return false
-	}
-
-	n := ns.getNode(key)
-	if n == nil {
-		if fin != nil {
-			fin(false, false)
-		}
-		return false
-
-	}
-
-	switch n.state {
-	case nodeEffective:
-		ns.lru.used -= n.charge
-		n.state = nodeDeleted
-		n.delfin = fin
-		n.rRemove()
-		n.derefNB()
-	case nodeEvicted:
-		n.state = nodeDeleted
-		n.delfin = fin
-	case nodeDeleted:
-		if fin != nil {
-			fin(true, true)
-		}
-		return false
-	default:
-		panic("invalid state")
-	}
-
-	return true
-}
-
-func (ns *lruNs) purgeNB(fin PurgeFin) {
-	if ns.state == nsEffective {
-		var nodes []*lruNode
-		ns.iterateNodes(func(n *lruNode) bool {
-			nodes = append(nodes, n)
-			return true
-		})
-		for _, n := range nodes {
-			switch n.state {
-			case nodeEffective:
-				ns.lru.used -= n.charge
-				n.state = nodeDeleted
-				n.purgefin = fin
-				n.rRemove()
-				n.derefNB()
-			case nodeEvicted:
-				n.state = nodeDeleted
-				n.purgefin = fin
-			case nodeDeleted:
-			default:
-				panic("invalid state")
-			}
-		}
-	}
-}
-
-func (ns *lruNs) Purge(fin PurgeFin) {
-	ns.lru.mu.Lock()
-	ns.purgeNB(fin)
-	ns.lru.mu.Unlock()
-}
-
-func (ns *lruNs) zapNB() {
-	if ns.state == nsEffective {
-		ns.state = nsZapped
-
-		ns.iterateNodes(func(n *lruNode) bool {
-			if n.state == nodeEffective {
-				ns.lru.used -= n.charge
-				n.rRemove()
-			}
-			ns.lru.size -= n.charge
-			n.state = nodeDeleted
-			n.fin()
-
-			return true
-		})
-		ns.rbRoot = nil
-	}
-}
-
-func (ns *lruNs) Zap() {
-	ns.lru.mu.Lock()
-	ns.zapNB()
-	delete(ns.lru.table, ns.id)
-	ns.lru.mu.Unlock()
-}
-
-type lruNode struct {
-	ns *lruNs
-
-	rNext, rPrev    *lruNode
-	rbLeft, rbRight *lruNode
-	rbBlack         bool
-
-	key      uint64
-	value    interface{}
-	charge   int
-	ref      int
-	state    nodeState
-	delfin   DelFin
-	purgefin PurgeFin
-}
-
-func (n *lruNode) rInsert(at *lruNode) {
-	x := at.rNext
-	at.rNext = n
-	n.rPrev = at
-	n.rNext = x
-	x.rPrev = n
-}
-
-func (n *lruNode) rRemove() bool {
-	if n.rPrev == nil {
-		return false
-	}
-
-	n.rPrev.rNext = n.rNext
-	n.rNext.rPrev = n.rPrev
-	n.rPrev = nil
-	n.rNext = nil
-
-	return true
-}
-
-func (n *lruNode) fin() {
-	if r, ok := n.value.(util.Releaser); ok {
-		r.Release()
-	}
-	if n.purgefin != nil {
-		if n.delfin != nil {
-			panic("conflicting delete and purge fin")
-		}
-		n.purgefin(n.ns.id, n.key)
-		n.purgefin = nil
-	} else if n.delfin != nil {
-		n.delfin(true, false)
-		n.delfin = nil
-	}
-}
-
-func (n *lruNode) derefNB() {
-	n.ref--
-	if n.ref == 0 {
-		if n.ns.state == nsEffective {
-			// Remove elemement.
-			n.ns.deleteNode(n.key)
-			n.ns.lru.size -= n.charge
-			n.ns.lru.alive--
-			n.fin()
-		}
-		n.value = nil
-	} else if n.ref < 0 {
-		panic("leveldb/cache: lruCache: negative node reference")
-	}
-}
-
-func (n *lruNode) deref() {
-	n.ns.lru.mu.Lock()
-	n.derefNB()
-	n.ns.lru.mu.Unlock()
-}
-
-type lruHandle struct {
-	node *lruNode
-	once uint32
-}
-
-func (h *lruHandle) Value() interface{} {
-	if atomic.LoadUint32(&h.once) == 0 {
-		return h.node.value
-	}
-	return nil
-}
-
-func (h *lruHandle) Release() {
-	if !atomic.CompareAndSwapUint32(&h.once, 0, 1) {
-		return
-	}
-	h.node.deref()
-	h.node = nil
-}
-
-func rbIsRed(h *lruNode) bool {
-	if h == nil {
-		return false
-	}
-	return !h.rbBlack
-}
-
-func rbRotLeft(h *lruNode) *lruNode {
-	x := h.rbRight
-	if x.rbBlack {
-		panic("rotating a black link")
-	}
-	h.rbRight = x.rbLeft
-	x.rbLeft = h
-	x.rbBlack = h.rbBlack
-	h.rbBlack = false
-	return x
-}
-
-func rbRotRight(h *lruNode) *lruNode {
-	x := h.rbLeft
-	if x.rbBlack {
-		panic("rotating a black link")
-	}
-	h.rbLeft = x.rbRight
-	x.rbRight = h
-	x.rbBlack = h.rbBlack
-	h.rbBlack = false
-	return x
-}
-
-func rbFlip(h *lruNode) {
-	h.rbBlack = !h.rbBlack
-	h.rbLeft.rbBlack = !h.rbLeft.rbBlack
-	h.rbRight.rbBlack = !h.rbRight.rbBlack
-}
-
-func rbMoveLeft(h *lruNode) *lruNode {
-	rbFlip(h)
-	if rbIsRed(h.rbRight.rbLeft) {
-		h.rbRight = rbRotRight(h.rbRight)
-		h = rbRotLeft(h)
-		rbFlip(h)
-	}
-	return h
-}
-
-func rbMoveRight(h *lruNode) *lruNode {
-	rbFlip(h)
-	if rbIsRed(h.rbLeft.rbLeft) {
-		h = rbRotRight(h)
-		rbFlip(h)
-	}
-	return h
-}
-
-func rbFixup(h *lruNode) *lruNode {
-	if rbIsRed(h.rbRight) {
-		h = rbRotLeft(h)
-	}
-
-	if rbIsRed(h.rbLeft) && rbIsRed(h.rbLeft.rbLeft) {
-		h = rbRotRight(h)
-	}
-
-	if rbIsRed(h.rbLeft) && rbIsRed(h.rbRight) {
-		rbFlip(h)
-	}
-
-	return h
-}
-
-func rbDeleteMin(h *lruNode) (hn, n *lruNode) {
-	if h == nil {
-		return nil, nil
-	}
-	if h.rbLeft == nil {
-		return nil, h
-	}
-
-	if !rbIsRed(h.rbLeft) && !rbIsRed(h.rbLeft.rbLeft) {
-		h = rbMoveLeft(h)
-	}
-
-	h.rbLeft, n = rbDeleteMin(h.rbLeft)
-
-	return rbFixup(h), n
-}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/corrupt_test.go

@@ -9,14 +9,12 @@ package leveldb
 import (
 	"bytes"
 	"fmt"
-	"io"
-	"math/rand"
-	"testing"
-
-	"github.com/syndtr/goleveldb/leveldb/cache"
 	"github.com/syndtr/goleveldb/leveldb/filter"
 	"github.com/syndtr/goleveldb/leveldb/opt"
 	"github.com/syndtr/goleveldb/leveldb/storage"
+	"io"
+	"math/rand"
+	"testing"
 )
 
 const ctValSize = 1000
@@ -33,8 +31,8 @@ func newDbCorruptHarnessWopt(t *testing.T, o *opt.Options) *dbCorruptHarness {
 
 func newDbCorruptHarness(t *testing.T) *dbCorruptHarness {
 	return newDbCorruptHarnessWopt(t, &opt.Options{
-		BlockCache: cache.NewLRUCache(100),
-		Strict:     opt.StrictJournalChecksum,
+		BlockCacheCapacity: 100,
+		Strict:             opt.StrictJournalChecksum,
 	})
 }
 
@@ -269,9 +267,9 @@ func TestCorruptDB_TableIndex(t *testing.T) {
 func TestCorruptDB_MissingManifest(t *testing.T) {
 	rnd := rand.New(rand.NewSource(0x0badda7a))
 	h := newDbCorruptHarnessWopt(t, &opt.Options{
-		BlockCache:  cache.NewLRUCache(100),
-		Strict:      opt.StrictJournalChecksum,
-		WriteBuffer: 1000 * 60,
+		BlockCacheCapacity: 100,
+		Strict:             opt.StrictJournalChecksum,
+		WriteBuffer:        1000 * 60,
 	})
 
 	h.build(1000)

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go

@@ -63,13 +63,14 @@ type DB struct {
 	journalAckC  chan error
 
 	// Compaction.
-	tcompCmdC   chan cCmd
-	tcompPauseC chan chan<- struct{}
-	mcompCmdC   chan cCmd
-	compErrC    chan error
-	compPerErrC chan error
-	compErrSetC chan error
-	compStats   []cStats
+	tcompCmdC        chan cCmd
+	tcompPauseC      chan chan<- struct{}
+	mcompCmdC        chan cCmd
+	compErrC         chan error
+	compPerErrC      chan error
+	compErrSetC      chan error
+	compWriteLocking bool
+	compStats        []cStats
 
 	// Close.
 	closeW sync.WaitGroup
@@ -108,28 +109,44 @@ func openDB(s *session) (*DB, error) {
 		closeC: make(chan struct{}),
 	}
 
-	if err := db.recoverJournal(); err != nil {
-		return nil, err
-	}
+	// Read-only mode.
+	readOnly := s.o.GetReadOnly()
 
-	// Remove any obsolete files.
-	if err := db.checkAndCleanFiles(); err != nil {
-		// Close journal.
-		if db.journal != nil {
-			db.journal.Close()
-			db.journalWriter.Close()
+	if readOnly {
+		// Recover journals (read-only mode).
+		if err := db.recoverJournalRO(); err != nil {
+			return nil, err
 		}
-		return nil, err
+	} else {
+		// Recover journals.
+		if err := db.recoverJournal(); err != nil {
+			return nil, err
+		}
+
+		// Remove any obsolete files.
+		if err := db.checkAndCleanFiles(); err != nil {
+			// Close journal.
+			if db.journal != nil {
+				db.journal.Close()
+				db.journalWriter.Close()
+			}
+			return nil, err
+		}
+
 	}
 
 	// Doesn't need to be included in the wait group.
 	go db.compactionError()
 	go db.mpoolDrain()
 
-	db.closeW.Add(3)
-	go db.tCompaction()
-	go db.mCompaction()
-	go db.jWriter()
+	if readOnly {
+		db.SetReadOnly()
+	} else {
+		db.closeW.Add(3)
+		go db.tCompaction()
+		go db.mCompaction()
+		go db.jWriter()
+	}
 
 	s.logf("db@open done T·%v", time.Since(start))
 
@@ -274,8 +291,9 @@ func recoverTable(s *session, o *opt.Options) error {
 
 		// We will drop corrupted table.
 		strict = o.GetStrict(opt.StrictRecovery)
+		noSync = o.GetNoSync()
 
-		rec   = &sessionRecord{numLevel: o.GetNumLevel()}
+		rec   = &sessionRecord{}
 		bpool = util.NewBufferPool(o.GetBlockSize() + 5)
 	)
 	buildTable := func(iter iterator.Iterator) (tmp storage.File, size int64, err error) {
@@ -311,9 +329,11 @@ func recoverTable(s *session, o *opt.Options) error {
 		if err != nil {
 			return
 		}
-		err = writer.Sync()
-		if err != nil {
-			return
+		if !noSync {
+			err = writer.Sync()
+			if err != nil {
+				return
+			}
 		}
 		size = int64(tw.BytesLen())
 		return
@@ -347,12 +367,14 @@ func recoverTable(s *session, o *opt.Options) error {
 			return err
 		}
 		iter := tr.NewIterator(nil, nil)
-		iter.(iterator.ErrorCallbackSetter).SetErrorCallback(func(err error) {
-			if errors.IsCorrupted(err) {
-				s.logf("table@recovery block corruption @%d %q", file.Num(), err)
-				tcorruptedBlock++
-			}
-		})
+		if itererr, ok := iter.(iterator.ErrorCallbackSetter); ok {
+			itererr.SetErrorCallback(func(err error) {
+				if errors.IsCorrupted(err) {
+					s.logf("table@recovery block corruption @%d %q", file.Num(), err)
+					tcorruptedBlock++
+				}
+			})
+		}
 
 		// Scan the table.
 		for iter.Next() {
@@ -448,132 +470,136 @@ func recoverTable(s *session, o *opt.Options) error {
 }
 
 func (db *DB) recoverJournal() error {
-	// Get all tables and sort it by file number.
-	journalFiles_, err := db.s.getFiles(storage.TypeJournal)
+	// Get all journals and sort it by file number.
+	allJournalFiles, err := db.s.getFiles(storage.TypeJournal)
 	if err != nil {
 		return err
 	}
-	journalFiles := files(journalFiles_)
-	journalFiles.sort()
+	files(allJournalFiles).sort()
 
-	// Discard older journal.
-	prev := -1
-	for i, file := range journalFiles {
-		if file.Num() >= db.s.stJournalNum {
-			if prev >= 0 {
-				i--
-				journalFiles[i] = journalFiles[prev]
-			}
-			journalFiles = journalFiles[i:]
-			break
-		} else if file.Num() == db.s.stPrevJournalNum {
-			prev = i
+	// Journals that will be recovered.
+	var recJournalFiles []storage.File
+	for _, jf := range allJournalFiles {
+		if jf.Num() >= db.s.stJournalNum || jf.Num() == db.s.stPrevJournalNum {
+			recJournalFiles = append(recJournalFiles, jf)
 		}
 	}
 
-	var jr *journal.Reader
-	var of storage.File
-	var mem *memdb.DB
-	batch := new(Batch)
-	cm := newCMem(db.s)
-	buf := new(util.Buffer)
-	// Options.
-	strict := db.s.o.GetStrict(opt.StrictJournal)
-	checksum := db.s.o.GetStrict(opt.StrictJournalChecksum)
-	writeBuffer := db.s.o.GetWriteBuffer()
-	recoverJournal := func(file storage.File) error {
-		db.logf("journal@recovery recovering @%d", file.Num())
-		reader, err := file.Open()
-		if err != nil {
-			return err
-		}
-		defer reader.Close()
+	var (
+		of  storage.File // Obsolete file.
+		rec = &sessionRecord{}
+	)
 
-		// Create/reset journal reader instance.
-		if jr == nil {
-			jr = journal.NewReader(reader, dropper{db.s, file}, strict, checksum)
-		} else {
-			jr.Reset(reader, dropper{db.s, file}, strict, checksum)
-		}
-
-		// Flush memdb and remove obsolete journal file.
-		if of != nil {
-			if mem.Len() > 0 {
-				if err := cm.flush(mem, 0); err != nil {
-					return err
-				}
-			}
-			if err := cm.commit(file.Num(), db.seq); err != nil {
-				return err
-			}
-			cm.reset()
-			of.Remove()
-			of = nil
-		}
-
-		// Replay journal to memdb.
-		mem.Reset()
-		for {
-			r, err := jr.Next()
-			if err != nil {
-				if err == io.EOF {
-					break
-				}
-				return errors.SetFile(err, file)
-			}
-
-			buf.Reset()
-			if _, err := buf.ReadFrom(r); err != nil {
-				if err == io.ErrUnexpectedEOF {
-					// This is error returned due to corruption, with strict == false.
-					continue
-				} else {
-					return errors.SetFile(err, file)
-				}
-			}
-			if err := batch.memDecodeAndReplay(db.seq, buf.Bytes(), mem); err != nil {
-				if strict || !errors.IsCorrupted(err) {
-					return errors.SetFile(err, file)
-				} else {
-					db.s.logf("journal error: %v (skipped)", err)
-					// We won't apply sequence number as it might be corrupted.
-					continue
-				}
-			}
-
-			// Save sequence number.
-			db.seq = batch.seq + uint64(batch.Len())
-
-			// Flush it if large enough.
-			if mem.Size() >= writeBuffer {
-				if err := cm.flush(mem, 0); err != nil {
-					return err
-				}
-				mem.Reset()
-			}
-		}
-
-		of = file
-		return nil
-	}
-
-	// Recover all journals.
-	if len(journalFiles) > 0 {
-		db.logf("journal@recovery F·%d", len(journalFiles))
+	// Recover journals.
+	if len(recJournalFiles) > 0 {
+		db.logf("journal@recovery F·%d", len(recJournalFiles))
 
 		// Mark file number as used.
-		db.s.markFileNum(journalFiles[len(journalFiles)-1].Num())
+		db.s.markFileNum(recJournalFiles[len(recJournalFiles)-1].Num())
 
-		mem = memdb.New(db.s.icmp, writeBuffer)
-		for _, file := range journalFiles {
-			if err := recoverJournal(file); err != nil {
+		var (
+			// Options.
+			strict      = db.s.o.GetStrict(opt.StrictJournal)
+			checksum    = db.s.o.GetStrict(opt.StrictJournalChecksum)
+			writeBuffer = db.s.o.GetWriteBuffer()
+
+			jr    *journal.Reader
+			mdb   = memdb.New(db.s.icmp, writeBuffer)
+			buf   = &util.Buffer{}
+			batch = &Batch{}
+		)
+
+		for _, jf := range recJournalFiles {
+			db.logf("journal@recovery recovering @%d", jf.Num())
+
+			fr, err := jf.Open()
+			if err != nil {
 				return err
 			}
+
+			// Create or reset journal reader instance.
+			if jr == nil {
+				jr = journal.NewReader(fr, dropper{db.s, jf}, strict, checksum)
+			} else {
+				jr.Reset(fr, dropper{db.s, jf}, strict, checksum)
+			}
+
+			// Flush memdb and remove obsolete journal file.
+			if of != nil {
+				if mdb.Len() > 0 {
+					if _, err := db.s.flushMemdb(rec, mdb, -1); err != nil {
+						fr.Close()
+						return err
+					}
+				}
+
+				rec.setJournalNum(jf.Num())
+				rec.setSeqNum(db.seq)
+				if err := db.s.commit(rec); err != nil {
+					fr.Close()
+					return err
+				}
+				rec.resetAddedTables()
+
+				of.Remove()
+				of = nil
+			}
+
+			// Replay journal to memdb.
+			mdb.Reset()
+			for {
+				r, err := jr.Next()
+				if err != nil {
+					if err == io.EOF {
+						break
+					}
+
+					fr.Close()
+					return errors.SetFile(err, jf)
+				}
+
+				buf.Reset()
+				if _, err := buf.ReadFrom(r); err != nil {
+					if err == io.ErrUnexpectedEOF {
+						// This is error returned due to corruption, with strict == false.
+						continue
+					}
+
+					fr.Close()
+					return errors.SetFile(err, jf)
+				}
+				if err := batch.memDecodeAndReplay(db.seq, buf.Bytes(), mdb); err != nil {
+					if !strict && errors.IsCorrupted(err) {
+						db.s.logf("journal error: %v (skipped)", err)
+						// We won't apply sequence number as it might be corrupted.
+						continue
+					}
+
+					fr.Close()
+					return errors.SetFile(err, jf)
+				}
+
+				// Save sequence number.
+				db.seq = batch.seq + uint64(batch.Len())
+
+				// Flush it if large enough.
+				if mdb.Size() >= writeBuffer {
+					if _, err := db.s.flushMemdb(rec, mdb, 0); err != nil {
+						fr.Close()
+						return err
+					}
+
+					mdb.Reset()
+				}
+			}
+
+			fr.Close()
+			of = jf
 		}
 
-		// Flush the last journal.
-		if mem.Len() > 0 {
-			if err := cm.flush(mem, 0); err != nil {
+		// Flush the last memdb.
+		if mdb.Len() > 0 {
+			if _, err := db.s.flushMemdb(rec, mdb, 0); err != nil {
 				return err
 			}
 		}
@@ -585,8 +611,10 @@ func (db *DB) recoverJournal() error {
 	}
 
 	// Commit.
-	if err := cm.commit(db.journalFile.Num(), db.seq); err != nil {
-		// Close journal.
+	rec.setJournalNum(db.journalFile.Num())
+	rec.setSeqNum(db.seq)
+	if err := db.s.commit(rec); err != nil {
+		// Close journal on error.
 		if db.journal != nil {
 			db.journal.Close()
 			db.journalWriter.Close()
@@ -602,6 +630,103 @@ func (db *DB) recoverJournal() error {
 	return nil
 }
 
+func (db *DB) recoverJournalRO() error {
+	// Get all journals and sort it by file number.
+	allJournalFiles, err := db.s.getFiles(storage.TypeJournal)
+	if err != nil {
+		return err
+	}
+	files(allJournalFiles).sort()
+
+	// Journals that will be recovered.
+	var recJournalFiles []storage.File
+	for _, jf := range allJournalFiles {
+		if jf.Num() >= db.s.stJournalNum || jf.Num() == db.s.stPrevJournalNum {
+			recJournalFiles = append(recJournalFiles, jf)
+		}
+	}
+
+	var (
+		// Options.
+		strict      = db.s.o.GetStrict(opt.StrictJournal)
+		checksum    = db.s.o.GetStrict(opt.StrictJournalChecksum)
+		writeBuffer = db.s.o.GetWriteBuffer()
+
+		mdb = memdb.New(db.s.icmp, writeBuffer)
+	)
+
+	// Recover journals.
+	if len(recJournalFiles) > 0 {
+		db.logf("journal@recovery RO·Mode F·%d", len(recJournalFiles))
+
+		var (
+			jr    *journal.Reader
+			buf   = &util.Buffer{}
+			batch = &Batch{}
+		)
+
+		for _, jf := range recJournalFiles {
+			db.logf("journal@recovery recovering @%d", jf.Num())
+
+			fr, err := jf.Open()
+			if err != nil {
+				return err
+			}
+
+			// Create or reset journal reader instance.
+			if jr == nil {
+				jr = journal.NewReader(fr, dropper{db.s, jf}, strict, checksum)
+			} else {
+				jr.Reset(fr, dropper{db.s, jf}, strict, checksum)
+			}
+
+			// Replay journal to memdb.
+			for {
+				r, err := jr.Next()
+				if err != nil {
+					if err == io.EOF {
+						break
+					}
+
+					fr.Close()
+					return errors.SetFile(err, jf)
+				}
+
+				buf.Reset()
+				if _, err := buf.ReadFrom(r); err != nil {
+					if err == io.ErrUnexpectedEOF {
+						// This is error returned due to corruption, with strict == false.
+						continue
+					}
+
+					fr.Close()
+					return errors.SetFile(err, jf)
+				}
+				if err := batch.memDecodeAndReplay(db.seq, buf.Bytes(), mdb); err != nil {
+					if !strict && errors.IsCorrupted(err) {
+						db.s.logf("journal error: %v (skipped)", err)
+						// We won't apply sequence number as it might be corrupted.
+						continue
+					}
+
+					fr.Close()
+					return errors.SetFile(err, jf)
+				}
+
+				// Save sequence number.
+				db.seq = batch.seq + uint64(batch.Len())
+			}
+
+			fr.Close()
+		}
+	}
+
+	// Set memDB.
+	db.mem = &memDB{db: db, DB: mdb, ref: 1}
+
+	return nil
+}
+
 func (db *DB) get(key []byte, seq uint64, ro *opt.ReadOptions) (value []byte, err error) {
 	ikey := newIkey(key, seq, ktSeek)
 
@@ -612,7 +737,7 @@ func (db *DB) get(key []byte, seq uint64, ro *opt.ReadOptions) (value []byte, er
 		}
 		defer m.decref()
 
-		mk, mv, me := m.mdb.Find(ikey)
+		mk, mv, me := m.Find(ikey)
 		if me == nil {
 			ukey, _, kt, kerr := parseIkey(mk)
 			if kerr != nil {
@@ -650,7 +775,7 @@ func (db *DB) has(key []byte, seq uint64, ro *opt.ReadOptions) (ret bool, err er
 		}
 		defer m.decref()
 
-		mk, _, me := m.mdb.Find(ikey)
+		mk, _, me := m.Find(ikey)
 		if me == nil {
 			ukey, _, kt, kerr := parseIkey(mk)
 			if kerr != nil {
@@ -782,7 +907,7 @@ func (db *DB) GetProperty(name string) (value string, err error) {
 
 	const prefix = "leveldb."
 	if !strings.HasPrefix(name, prefix) {
-		return "", errors.New("leveldb: GetProperty: unknown property: " + name)
+		return "", ErrNotFound
 	}
 	p := name[len(prefix):]
 
@@ -796,7 +921,7 @@ func (db *DB) GetProperty(name string) (value string, err error) {
 		var rest string
 		n, _ := fmt.Sscanf(p[len(numFilesPrefix):], "%d%s", &level, &rest)
 		if n != 1 || int(level) >= db.s.o.GetNumLevel() {
-			err = errors.New("leveldb: GetProperty: invalid property: " + name)
+			err = ErrNotFound
 		} else {
 			value = fmt.Sprint(v.tLen(int(level)))
 		}
@@ -823,8 +948,8 @@ func (db *DB) GetProperty(name string) (value string, err error) {
 	case p == "blockpool":
 		value = fmt.Sprintf("%v", db.s.tops.bpool)
 	case p == "cachedblock":
-		if bc := db.s.o.GetBlockCache(); bc != nil {
-			value = fmt.Sprintf("%d", bc.Size())
+		if db.s.tops.bcache != nil {
+			value = fmt.Sprintf("%d", db.s.tops.bcache.Size())
 		} else {
 			value = "<nil>"
 		}
@@ -835,7 +960,7 @@ func (db *DB) GetProperty(name string) (value string, err error) {
 	case p == "aliveiters":
 		value = fmt.Sprintf("%d", atomic.LoadInt32(&db.aliveIters))
 	default:
-		err = errors.New("leveldb: GetProperty: unknown property: " + name)
+		err = ErrNotFound
 	}
 
 	return
@@ -898,6 +1023,9 @@ func (db *DB) Close() error {
 	var err error
 	select {
 	case err = <-db.compErrC:
+		if err == ErrReadOnly {
+			err = nil
+		}
 	default:
 	}
 

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go

@@ -11,7 +11,6 @@ import (
 	"time"
 
 	"github.com/syndtr/goleveldb/leveldb/errors"
-	"github.com/syndtr/goleveldb/leveldb/memdb"
 	"github.com/syndtr/goleveldb/leveldb/opt"
 )
 
@@ -62,58 +61,8 @@ func (p *cStatsStaging) stopTimer() {
 	}
 }
 
-type cMem struct {
-	s     *session
-	level int
-	rec   *sessionRecord
-}
-
-func newCMem(s *session) *cMem {
-	return &cMem{s: s, rec: &sessionRecord{numLevel: s.o.GetNumLevel()}}
-}
-
-func (c *cMem) flush(mem *memdb.DB, level int) error {
-	s := c.s
-
-	// Write memdb to table.
-	iter := mem.NewIterator(nil)
-	defer iter.Release()
-	t, n, err := s.tops.createFrom(iter)
-	if err != nil {
-		return err
-	}
-
-	// Pick level.
-	if level < 0 {
-		v := s.version()
-		level = v.pickLevel(t.imin.ukey(), t.imax.ukey())
-		v.release()
-	}
-	c.rec.addTableFile(level, t)
-
-	s.logf("mem@flush created L%d@%d N·%d S·%s %q:%q", level, t.file.Num(), n, shortenb(int(t.size)), t.imin, t.imax)
-
-	c.level = level
-	return nil
-}
-
-func (c *cMem) reset() {
-	c.rec = &sessionRecord{numLevel: c.s.o.GetNumLevel()}
-}
-
-func (c *cMem) commit(journal, seq uint64) error {
-	c.rec.setJournalNum(journal)
-	c.rec.setSeqNum(seq)
-
-	// Commit changes.
-	return c.s.commit(c.rec)
-}
-
 func (db *DB) compactionError() {
-	var (
-		err     error
-		wlocked bool
-	)
+	var err error
 noerr:
 	// No error.
 	for {
@@ -121,7 +70,7 @@ noerr:
 		case err = <-db.compErrSetC:
 			switch {
 			case err == nil:
-			case errors.IsCorrupted(err):
+			case err == ErrReadOnly, errors.IsCorrupted(err):
 				goto hasperr
 			default:
 				goto haserr
@@ -139,7 +88,7 @@ haserr:
 			switch {
 			case err == nil:
 				goto noerr
-			case errors.IsCorrupted(err):
+			case err == ErrReadOnly, errors.IsCorrupted(err):
 				goto hasperr
 			default:
 			}
@@ -155,9 +104,9 @@ hasperr:
 		case db.compPerErrC <- err:
 		case db.writeLockC <- struct{}{}:
 			// Hold write lock, so that write won't pass-through.
-			wlocked = true
+			db.compWriteLocking = true
 		case _, _ = <-db.closeC:
-			if wlocked {
+			if db.compWriteLocking {
 				// We should release the lock or Close will hang.
 				<-db.writeLockC
 			}
@@ -287,21 +236,18 @@ func (db *DB) compactionExitTransact() {
 }
 
 func (db *DB) memCompaction() {
-	mem := db.getFrozenMem()
-	if mem == nil {
+	mdb := db.getFrozenMem()
+	if mdb == nil {
 		return
 	}
-	defer mem.decref()
+	defer mdb.decref()
 
-	c := newCMem(db.s)
-	stats := new(cStatsStaging)
-
-	db.logf("mem@flush N·%d S·%s", mem.mdb.Len(), shortenb(mem.mdb.Size()))
+	db.logf("memdb@flush N·%d S·%s", mdb.Len(), shortenb(mdb.Size()))
 
 	// Don't compact empty memdb.
-	if mem.mdb.Len() == 0 {
-		db.logf("mem@flush skipping")
-		// drop frozen mem
+	if mdb.Len() == 0 {
+		db.logf("memdb@flush skipping")
+		// drop frozen memdb
 		db.dropFrozenMem()
 		return
 	}
@@ -317,13 +263,20 @@ func (db *DB) memCompaction() {
 		return
 	}
 
-	db.compactionTransactFunc("mem@flush", func(cnt *compactionTransactCounter) (err error) {
+	var (
+		rec        = &sessionRecord{}
+		stats      = &cStatsStaging{}
+		flushLevel int
+	)
+
+	db.compactionTransactFunc("memdb@flush", func(cnt *compactionTransactCounter) (err error) {
 		stats.startTimer()
-		defer stats.stopTimer()
-		return c.flush(mem.mdb, -1)
+		flushLevel, err = db.s.flushMemdb(rec, mdb.DB, -1)
+		stats.stopTimer()
+		return
 	}, func() error {
-		for _, r := range c.rec.addedTables {
-			db.logf("mem@flush revert @%d", r.num)
+		for _, r := range rec.addedTables {
+			db.logf("memdb@flush revert @%d", r.num)
 			f := db.s.getTableFile(r.num)
 			if err := f.Remove(); err != nil {
 				return err
@@ -332,20 +285,23 @@ func (db *DB) memCompaction() {
 		return nil
 	})
 
-	db.compactionTransactFunc("mem@commit", func(cnt *compactionTransactCounter) (err error) {
+	db.compactionTransactFunc("memdb@commit", func(cnt *compactionTransactCounter) (err error) {
 		stats.startTimer()
-		defer stats.stopTimer()
-		return c.commit(db.journalFile.Num(), db.frozenSeq)
+		rec.setJournalNum(db.journalFile.Num())
+		rec.setSeqNum(db.frozenSeq)
+		err = db.s.commit(rec)
+		stats.stopTimer()
+		return
 	}, nil)
 
-	db.logf("mem@flush committed F·%d T·%v", len(c.rec.addedTables), stats.duration)
+	db.logf("memdb@flush committed F·%d T·%v", len(rec.addedTables), stats.duration)
 
-	for _, r := range c.rec.addedTables {
+	for _, r := range rec.addedTables {
 		stats.write += r.size
 	}
-	db.compStats[c.level].add(stats)
+	db.compStats[flushLevel].add(stats)
 
-	// Drop frozen mem.
+	// Drop frozen memdb.
 	db.dropFrozenMem()
 
 	// Resume table compaction.
@@ -557,7 +513,7 @@ func (b *tableCompactionBuilder) revert() error {
 func (db *DB) tableCompaction(c *compaction, noTrivial bool) {
 	defer c.release()
 
-	rec := &sessionRecord{numLevel: db.s.o.GetNumLevel()}
+	rec := &sessionRecord{}
 	rec.addCompPtr(c.level, c.imax)
 
 	if !noTrivial && c.trivial() {

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_iter.go

@@ -8,6 +8,7 @@ package leveldb
 
 import (
 	"errors"
+	"math/rand"
 	"runtime"
 	"sync"
 	"sync/atomic"
@@ -39,11 +40,11 @@ func (db *DB) newRawIterator(slice *util.Range, ro *opt.ReadOptions) iterator.It
 	ti := v.getIterators(slice, ro)
 	n := len(ti) + 2
 	i := make([]iterator.Iterator, 0, n)
-	emi := em.mdb.NewIterator(slice)
+	emi := em.NewIterator(slice)
 	emi.SetReleaser(&memdbReleaser{m: em})
 	i = append(i, emi)
 	if fm != nil {
-		fmi := fm.mdb.NewIterator(slice)
+		fmi := fm.NewIterator(slice)
 		fmi.SetReleaser(&memdbReleaser{m: fm})
 		i = append(i, fmi)
 	}
@@ -80,6 +81,10 @@ func (db *DB) newIterator(seq uint64, slice *util.Range, ro *opt.ReadOptions) *d
 	return iter
 }
 
+func (db *DB) iterSamplingRate() int {
+	return rand.Intn(2 * db.s.o.GetIteratorSamplingRate())
+}
+
 type dir int
 
 const (
@@ -98,11 +103,21 @@ type dbIter struct {
 	seq    uint64
 	strict bool
 
-	dir      dir
-	key      []byte
-	value    []byte
-	err      error
-	releaser util.Releaser
+	smaplingGap int
+	dir         dir
+	key         []byte
+	value       []byte
+	err         error
+	releaser    util.Releaser
+}
+
+func (i *dbIter) sampleSeek() {
+	ikey := i.iter.Key()
+	i.smaplingGap -= len(ikey) + len(i.iter.Value())
+	for i.smaplingGap < 0 {
+		i.smaplingGap += i.db.iterSamplingRate()
+		i.db.sampleSeek(ikey)
+	}
 }
 
 func (i *dbIter) setErr(err error) {
@@ -175,6 +190,7 @@ func (i *dbIter) Seek(key []byte) bool {
 func (i *dbIter) next() bool {
 	for {
 		if ukey, seq, kt, kerr := parseIkey(i.iter.Key()); kerr == nil {
+			i.sampleSeek()
 			if seq <= i.seq {
 				switch kt {
 				case ktDel:
@@ -225,6 +241,7 @@ func (i *dbIter) prev() bool {
 	if i.iter.Valid() {
 		for {
 			if ukey, seq, kt, kerr := parseIkey(i.iter.Key()); kerr == nil {
+				i.sampleSeek()
 				if seq <= i.seq {
 					if !del && i.icmp.uCompare(ukey, i.key) < 0 {
 						return true
@@ -266,6 +283,7 @@ func (i *dbIter) Prev() bool {
 	case dirForward:
 		for i.iter.Prev() {
 			if ukey, _, _, kerr := parseIkey(i.iter.Key()); kerr == nil {
+				i.sampleSeek()
 				if i.icmp.uCompare(ukey, i.key) < 0 {
 					goto cont
 				}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_snapshot.go

@@ -8,6 +8,7 @@ package leveldb
 
 import (
 	"container/list"
+	"fmt"
 	"runtime"
 	"sync"
 	"sync/atomic"
@@ -89,6 +90,10 @@ func (db *DB) newSnapshot() *Snapshot {
 	return snap
 }
 
+func (snap *Snapshot) String() string {
+	return fmt.Sprintf("leveldb.Snapshot{%d}", snap.elem.seq)
+}
+
 // Get gets the value for the given key. It returns ErrNotFound if
 // the DB does not contains the key.
 //

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_state.go

@@ -15,8 +15,8 @@ import (
 )
 
 type memDB struct {
-	db  *DB
-	mdb *memdb.DB
+	db *DB
+	*memdb.DB
 	ref int32
 }
 
@@ -27,12 +27,12 @@ func (m *memDB) incref() {
 func (m *memDB) decref() {
 	if ref := atomic.AddInt32(&m.ref, -1); ref == 0 {
 		// Only put back memdb with std capacity.
-		if m.mdb.Capacity() == m.db.s.o.GetWriteBuffer() {
-			m.mdb.Reset()
-			m.db.mpoolPut(m.mdb)
+		if m.Capacity() == m.db.s.o.GetWriteBuffer() {
+			m.Reset()
+			m.db.mpoolPut(m.DB)
 		}
 		m.db = nil
-		m.mdb = nil
+		m.DB = nil
 	} else if ref < 0 {
 		panic("negative memdb ref")
 	}
@@ -48,6 +48,15 @@ func (db *DB) addSeq(delta uint64) {
 	atomic.AddUint64(&db.seq, delta)
 }
 
+func (db *DB) sampleSeek(ikey iKey) {
+	v := db.s.version()
+	if v.sampleSeek(ikey) {
+		// Trigger table compaction.
+		db.compSendTrigger(db.tcompCmdC)
+	}
+	v.release()
+}
+
 func (db *DB) mpoolPut(mem *memdb.DB) {
 	defer func() {
 		recover()
@@ -117,7 +126,7 @@ func (db *DB) newMem(n int) (mem *memDB, err error) {
 	}
 	mem = &memDB{
 		db:  db,
-		mdb: mdb,
+		DB:  mdb,
 		ref: 2,
 	}
 	db.mem = mem

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_test.go

@@ -405,19 +405,21 @@ func (h *dbHarness) compactRange(min, max string) {
 	t.Log("DB range compaction done")
 }
 
-func (h *dbHarness) sizeAssert(start, limit string, low, hi uint64) {
-	t := h.t
-	db := h.db
-
-	s, err := db.SizeOf([]util.Range{
+func (h *dbHarness) sizeOf(start, limit string) uint64 {
+	sz, err := h.db.SizeOf([]util.Range{
 		{[]byte(start), []byte(limit)},
 	})
 	if err != nil {
-		t.Error("SizeOf: got error: ", err)
+		h.t.Error("SizeOf: got error: ", err)
 	}
-	if s.Sum() < low || s.Sum() > hi {
-		t.Errorf("sizeof %q to %q not in range, want %d - %d, got %d",
-			shorten(start), shorten(limit), low, hi, s.Sum())
+	return sz.Sum()
+}
+
+func (h *dbHarness) sizeAssert(start, limit string, low, hi uint64) {
+	sz := h.sizeOf(start, limit)
+	if sz < low || sz > hi {
+		h.t.Errorf("sizeOf %q to %q not in range, want %d - %d, got %d",
+			shorten(start), shorten(limit), low, hi, sz)
 	}
 }
 
@@ -1271,7 +1273,7 @@ func TestDB_DeletionMarkers2(t *testing.T) {
 }
 
 func TestDB_CompactionTableOpenError(t *testing.T) {
-	h := newDbHarnessWopt(t, &opt.Options{CachedOpenFiles: -1})
+	h := newDbHarnessWopt(t, &opt.Options{OpenFilesCacheCapacity: -1})
 	defer h.close()
 
 	im := 10
@@ -1629,8 +1631,8 @@ func TestDB_ManualCompaction(t *testing.T) {
 
 func TestDB_BloomFilter(t *testing.T) {
 	h := newDbHarnessWopt(t, &opt.Options{
-		BlockCache: opt.NoCache,
-		Filter:     filter.NewBloomFilter(10),
+		DisableBlockCache: true,
+		Filter:            filter.NewBloomFilter(10),
 	})
 	defer h.close()
 
@@ -2066,8 +2068,8 @@ func TestDB_GetProperties(t *testing.T) {
 
 func TestDB_GoleveldbIssue72and83(t *testing.T) {
 	h := newDbHarnessWopt(t, &opt.Options{
-		WriteBuffer:     1 * opt.MiB,
-		CachedOpenFiles: 3,
+		WriteBuffer:            1 * opt.MiB,
+		OpenFilesCacheCapacity: 3,
 	})
 	defer h.close()
 
@@ -2200,7 +2202,7 @@ func TestDB_GoleveldbIssue72and83(t *testing.T) {
 func TestDB_TransientError(t *testing.T) {
 	h := newDbHarnessWopt(t, &opt.Options{
 		WriteBuffer:              128 * opt.KiB,
-		CachedOpenFiles:          3,
+		OpenFilesCacheCapacity:   3,
 		DisableCompactionBackoff: true,
 	})
 	defer h.close()
@@ -2410,7 +2412,7 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
 		CompactionTableSize:         43 * opt.KiB,
 		CompactionExpandLimitFactor: 1,
 		CompactionGPOverlapsFactor:  1,
-		BlockCache:                  opt.NoCache,
+		DisableBlockCache:           true,
 	}
 	s, err := newSession(stor, o)
 	if err != nil {
@@ -2443,7 +2445,7 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
 		if err != nil {
 			t.Fatal(err)
 		}
-		rec := &sessionRecord{numLevel: s.o.GetNumLevel()}
+		rec := &sessionRecord{}
 		rec.addTableFile(i, tf)
 		if err := s.commit(rec); err != nil {
 			t.Fatal(err)
@@ -2453,7 +2455,7 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
 	// Build grandparent.
 	v := s.version()
 	c := newCompaction(s, v, 1, append(tFiles{}, v.tables[1]...))
-	rec := &sessionRecord{numLevel: s.o.GetNumLevel()}
+	rec := &sessionRecord{}
 	b := &tableCompactionBuilder{
 		s:         s,
 		c:         c,
@@ -2477,7 +2479,7 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
 	// Build level-1.
 	v = s.version()
 	c = newCompaction(s, v, 0, append(tFiles{}, v.tables[0]...))
-	rec = &sessionRecord{numLevel: s.o.GetNumLevel()}
+	rec = &sessionRecord{}
 	b = &tableCompactionBuilder{
 		s:         s,
 		c:         c,
@@ -2521,7 +2523,7 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
 	// Compaction with transient error.
 	v = s.version()
 	c = newCompaction(s, v, 1, append(tFiles{}, v.tables[1]...))
-	rec = &sessionRecord{numLevel: s.o.GetNumLevel()}
+	rec = &sessionRecord{}
 	b = &tableCompactionBuilder{
 		s:         s,
 		c:         c,
@@ -2577,3 +2579,123 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
 	}
 	v.release()
 }
+
+func testDB_IterTriggeredCompaction(t *testing.T, limitDiv int) {
+	const (
+		vSize = 200 * opt.KiB
+		tSize = 100 * opt.MiB
+		mIter = 100
+		n     = tSize / vSize
+	)
+
+	h := newDbHarnessWopt(t, &opt.Options{
+		Compression:       opt.NoCompression,
+		DisableBlockCache: true,
+	})
+	defer h.close()
+
+	key := func(x int) string {
+		return fmt.Sprintf("v%06d", x)
+	}
+
+	// Fill.
+	value := strings.Repeat("x", vSize)
+	for i := 0; i < n; i++ {
+		h.put(key(i), value)
+	}
+	h.compactMem()
+
+	// Delete all.
+	for i := 0; i < n; i++ {
+		h.delete(key(i))
+	}
+	h.compactMem()
+
+	var (
+		limit = n / limitDiv
+
+		startKey = key(0)
+		limitKey = key(limit)
+		maxKey   = key(n)
+		slice    = &util.Range{Limit: []byte(limitKey)}
+
+		initialSize0 = h.sizeOf(startKey, limitKey)
+		initialSize1 = h.sizeOf(limitKey, maxKey)
+	)
+
+	t.Logf("inital size %s [rest %s]", shortenb(int(initialSize0)), shortenb(int(initialSize1)))
+
+	for r := 0; true; r++ {
+		if r >= mIter {
+			t.Fatal("taking too long to compact")
+		}
+
+		// Iterates.
+		iter := h.db.NewIterator(slice, h.ro)
+		for iter.Next() {
+		}
+		if err := iter.Error(); err != nil {
+			t.Fatalf("Iter err: %v", err)
+		}
+		iter.Release()
+
+		// Wait compaction.
+		h.waitCompaction()
+
+		// Check size.
+		size0 := h.sizeOf(startKey, limitKey)
+		size1 := h.sizeOf(limitKey, maxKey)
+		t.Logf("#%03d size %s [rest %s]", r, shortenb(int(size0)), shortenb(int(size1)))
+		if size0 < initialSize0/10 {
+			break
+		}
+	}
+
+	if initialSize1 > 0 {
+		h.sizeAssert(limitKey, maxKey, initialSize1/4-opt.MiB, initialSize1+opt.MiB)
+	}
+}
+
+func TestDB_IterTriggeredCompaction(t *testing.T) {
+	testDB_IterTriggeredCompaction(t, 1)
+}
+
+func TestDB_IterTriggeredCompactionHalf(t *testing.T) {
+	testDB_IterTriggeredCompaction(t, 2)
+}
+
+func TestDB_ReadOnly(t *testing.T) {
+	h := newDbHarness(t)
+	defer h.close()
+
+	h.put("foo", "v1")
+	h.put("bar", "v2")
+	h.compactMem()
+
+	h.put("xfoo", "v1")
+	h.put("xbar", "v2")
+
+	t.Log("Trigger read-only")
+	if err := h.db.SetReadOnly(); err != nil {
+		h.close()
+		t.Fatalf("SetReadOnly error: %v", err)
+	}
+
+	h.stor.SetEmuErr(storage.TypeAll, tsOpCreate, tsOpReplace, tsOpRemove, tsOpWrite, tsOpWrite, tsOpSync)
+
+	ro := func(key, value, wantValue string) {
+		if err := h.db.Put([]byte(key), []byte(value), h.wo); err != ErrReadOnly {
+			t.Fatalf("unexpected error: %v", err)
+		}
+		h.getVal(key, wantValue)
+	}
+
+	ro("foo", "vx", "v1")
+
+	h.o.ReadOnly = true
+	h.reopenDB()
+
+	ro("foo", "vx", "v1")
+	ro("bar", "vx", "v2")
+	h.assertNumKeys(4)
+}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_write.go

@@ -63,24 +63,24 @@ func (db *DB) rotateMem(n int) (mem *memDB, err error) {
 	return
 }
 
-func (db *DB) flush(n int) (mem *memDB, nn int, err error) {
+func (db *DB) flush(n int) (mdb *memDB, mdbFree int, err error) {
 	delayed := false
 	flush := func() (retry bool) {
 		v := db.s.version()
 		defer v.release()
-		mem = db.getEffectiveMem()
+		mdb = db.getEffectiveMem()
 		defer func() {
 			if retry {
-				mem.decref()
-				mem = nil
+				mdb.decref()
+				mdb = nil
 			}
 		}()
-		nn = mem.mdb.Free()
+		mdbFree = mdb.Free()
 		switch {
 		case v.tLen(0) >= db.s.o.GetWriteL0SlowdownTrigger() && !delayed:
 			delayed = true
 			time.Sleep(time.Millisecond)
-		case nn >= n:
+		case mdbFree >= n:
 			return false
 		case v.tLen(0) >= db.s.o.GetWriteL0PauseTrigger():
 			delayed = true
@@ -90,15 +90,15 @@ func (db *DB) flush(n int) (mem *memDB, nn int, err error) {
 			}
 		default:
 			// Allow memdb to grow if it has no entry.
-			if mem.mdb.Len() == 0 {
-				nn = n
+			if mdb.Len() == 0 {
+				mdbFree = n
 			} else {
-				mem.decref()
-				mem, err = db.rotateMem(n)
+				mdb.decref()
+				mdb, err = db.rotateMem(n)
 				if err == nil {
-					nn = mem.mdb.Free()
+					mdbFree = mdb.Free()
 				} else {
-					nn = 0
+					mdbFree = 0
 				}
 			}
 			return false
@@ -112,9 +112,9 @@ func (db *DB) flush(n int) (mem *memDB, nn int, err error) {
 		db.writeDelay += time.Since(start)
 		db.writeDelayN++
 	} else if db.writeDelayN > 0 {
+		db.logf("db@write was delayed N·%d T·%v", db.writeDelayN, db.writeDelay)
 		db.writeDelay = 0
 		db.writeDelayN = 0
-		db.logf("db@write was delayed N·%d T·%v", db.writeDelayN, db.writeDelay)
 	}
 	return
 }
@@ -129,7 +129,7 @@ func (db *DB) Write(b *Batch, wo *opt.WriteOptions) (err error) {
 		return
 	}
 
-	b.init(wo.GetSync())
+	b.init(wo.GetSync() && !db.s.o.GetNoSync())
 
 	// The write happen synchronously.
 	select {
@@ -157,18 +157,18 @@ func (db *DB) Write(b *Batch, wo *opt.WriteOptions) (err error) {
 		}
 	}()
 
-	mem, memFree, err := db.flush(b.size())
+	mdb, mdbFree, err := db.flush(b.size())
 	if err != nil {
 		return
 	}
-	defer mem.decref()
+	defer mdb.decref()
 
 	// Calculate maximum size of the batch.
 	m := 1 << 20
 	if x := b.size(); x <= 128<<10 {
 		m = x + (128 << 10)
 	}
-	m = minInt(m, memFree)
+	m = minInt(m, mdbFree)
 
 	// Merge with other batch.
 drain:
@@ -197,7 +197,7 @@ drain:
 		select {
 		case db.journalC <- b:
 			// Write into memdb
-			if berr := b.memReplay(mem.mdb); berr != nil {
+			if berr := b.memReplay(mdb.DB); berr != nil {
 				panic(berr)
 			}
 		case err = <-db.compPerErrC:
@@ -211,7 +211,7 @@ drain:
 		case err = <-db.journalAckC:
 			if err != nil {
 				// Revert memdb if error detected
-				if berr := b.revertMemReplay(mem.mdb); berr != nil {
+				if berr := b.revertMemReplay(mdb.DB); berr != nil {
 					panic(berr)
 				}
 				return
@@ -225,7 +225,7 @@ drain:
 		if err != nil {
 			return
 		}
-		if berr := b.memReplay(mem.mdb); berr != nil {
+		if berr := b.memReplay(mdb.DB); berr != nil {
 			panic(berr)
 		}
 	}
@@ -233,7 +233,7 @@ drain:
 	// Set last seq number.
 	db.addSeq(uint64(b.Len()))
 
-	if b.size() >= memFree {
+	if b.size() >= mdbFree {
 		db.rotateMem(0)
 	}
 	return
@@ -249,8 +249,7 @@ func (db *DB) Put(key, value []byte, wo *opt.WriteOptions) error {
 	return db.Write(b, wo)
 }
 
-// Delete deletes the value for the given key. It returns ErrNotFound if
-// the DB does not contain the key.
+// Delete deletes the value for the given key.
 //
 // It is safe to modify the contents of the arguments after Delete returns.
 func (db *DB) Delete(key []byte, wo *opt.WriteOptions) error {
@@ -290,9 +289,9 @@ func (db *DB) CompactRange(r util.Range) error {
 	}
 
 	// Check for overlaps in memdb.
-	mem := db.getEffectiveMem()
-	defer mem.decref()
-	if isMemOverlaps(db.s.icmp, mem.mdb, r.Start, r.Limit) {
+	mdb := db.getEffectiveMem()
+	defer mdb.decref()
+	if isMemOverlaps(db.s.icmp, mdb.DB, r.Start, r.Limit) {
 		// Memdb compaction.
 		if _, err := db.rotateMem(0); err != nil {
 			<-db.writeLockC
@@ -309,3 +308,31 @@ func (db *DB) CompactRange(r util.Range) error {
 	// Table compaction.
 	return db.compSendRange(db.tcompCmdC, -1, r.Start, r.Limit)
 }
+
+// SetReadOnly makes DB read-only. It will stay read-only until reopened.
+func (db *DB) SetReadOnly() error {
+	if err := db.ok(); err != nil {
+		return err
+	}
+
+	// Lock writer.
+	select {
+	case db.writeLockC <- struct{}{}:
+		db.compWriteLocking = true
+	case err := <-db.compPerErrC:
+		return err
+	case _, _ = <-db.closeC:
+		return ErrClosed
+	}
+
+	// Set compaction read-only.
+	select {
+	case db.compErrSetC <- ErrReadOnly:
+	case perr := <-db.compPerErrC:
+		return perr
+	case _, _ = <-db.closeC:
+		return ErrClosed
+	}
+
+	return nil
+}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/errors.go

@@ -12,6 +12,7 @@ import (
 
 var (
 	ErrNotFound         = errors.ErrNotFound
+	ErrReadOnly         = errors.New("leveldb: read-only mode")
 	ErrSnapshotReleased = errors.New("leveldb: snapshot released")
 	ErrIterReleased     = errors.New("leveldb: iterator released")
 	ErrClosed           = errors.New("leveldb: closed")

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/errors/errors.go

@@ -52,12 +52,14 @@ func IsCorrupted(err error) bool {
 	switch err.(type) {
 	case *ErrCorrupted:
 		return true
+	case *storage.ErrCorrupted:
+		return true
 	}
 	return false
 }
 
 // ErrMissingFiles is the type that indicating a corruption due to missing
-// files.
+// files. ErrMissingFiles always wrapped with ErrCorrupted.
 type ErrMissingFiles struct {
 	Files []*storage.FileInfo
 }

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/external_test.go

@@ -17,14 +17,14 @@ import (
 var _ = testutil.Defer(func() {
 	Describe("Leveldb external", func() {
 		o := &opt.Options{
-			BlockCache:           opt.NoCache,
-			BlockRestartInterval: 5,
-			BlockSize:            80,
-			Compression:          opt.NoCompression,
-			CachedOpenFiles:      -1,
-			Strict:               opt.StrictAll,
-			WriteBuffer:          1000,
-			CompactionTableSize:  2000,
+			DisableBlockCache:      true,
+			BlockRestartInterval:   5,
+			BlockSize:              80,
+			Compression:            opt.NoCompression,
+			OpenFilesCacheCapacity: -1,
+			Strict:                 opt.StrictAll,
+			WriteBuffer:            1000,
+			CompactionTableSize:    2000,
 		}
 
 		Describe("write test", func() {

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/key.go

@@ -106,7 +106,7 @@ func (ik iKey) assert() {
 		panic("leveldb: nil iKey")
 	}
 	if len(ik) < 8 {
-		panic(fmt.Sprintf("leveldb: iKey %q, len=%d: invalid length", ik, len(ik)))
+		panic(fmt.Sprintf("leveldb: iKey %q, len=%d: invalid length", []byte(ik), len(ik)))
 	}
 }
 
@@ -124,7 +124,7 @@ func (ik iKey) parseNum() (seq uint64, kt kType) {
 	num := ik.num()
 	seq, kt = uint64(num>>8), kType(num&0xff)
 	if kt > ktVal {
-		panic(fmt.Sprintf("leveldb: iKey %q, len=%d: invalid type %#x", ik, len(ik), kt))
+		panic(fmt.Sprintf("leveldb: iKey %q, len=%d: invalid type %#x", []byte(ik), len(ik), kt))
 	}
 	return
 }

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/memdb/memdb.go

@@ -206,6 +206,7 @@ func (p *DB) randHeight() (h int) {
 	return
 }
 
+// Must hold RW-lock if prev == true, as it use shared prevNode slice.
 func (p *DB) findGE(key []byte, prev bool) (int, bool) {
 	node := 0
 	h := p.maxHeight - 1
@@ -302,7 +303,7 @@ func (p *DB) Put(key []byte, value []byte) error {
 	node := len(p.nodeData)
 	p.nodeData = append(p.nodeData, kvOffset, len(key), len(value), h)
 	for i, n := range p.prevNode[:h] {
-		m := n + 4 + i
+		m := n + nNext + i
 		p.nodeData = append(p.nodeData, p.nodeData[m])
 		p.nodeData[m] = node
 	}
@@ -434,20 +435,22 @@ func (p *DB) Len() int {
 
 // Reset resets the DB to initial empty state. Allows reuse the buffer.
 func (p *DB) Reset() {
+	p.mu.Lock()
 	p.rnd = rand.New(rand.NewSource(0xdeadbeef))
 	p.maxHeight = 1
 	p.n = 0
 	p.kvSize = 0
 	p.kvData = p.kvData[:0]
-	p.nodeData = p.nodeData[:4+tMaxHeight]
+	p.nodeData = p.nodeData[:nNext+tMaxHeight]
 	p.nodeData[nKV] = 0
 	p.nodeData[nKey] = 0
 	p.nodeData[nVal] = 0
 	p.nodeData[nHeight] = tMaxHeight
 	for n := 0; n < tMaxHeight; n++ {
-		p.nodeData[4+n] = 0
+		p.nodeData[nNext+n] = 0
 		p.prevNode[n] = 0
 	}
+	p.mu.Unlock()
 }
 
 // New creates a new initalized in-memory key/value DB. The capacity

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/opt/options.go

@@ -20,8 +20,9 @@ const (
 	GiB = MiB * 1024
 )
 
-const (
-	DefaultBlockCacheSize                = 8 * MiB
+var (
+	DefaultBlockCacher                   = LRUCacher
+	DefaultBlockCacheCapacity            = 8 * MiB
 	DefaultBlockRestartInterval          = 16
 	DefaultBlockSize                     = 4 * KiB
 	DefaultCompactionExpandLimitFactor   = 25
@@ -33,30 +34,43 @@ const (
 	DefaultCompactionTotalSize           = 10 * MiB
 	DefaultCompactionTotalSizeMultiplier = 10.0
 	DefaultCompressionType               = SnappyCompression
-	DefaultCachedOpenFiles               = 500
+	DefaultIteratorSamplingRate          = 1 * MiB
 	DefaultMaxMemCompationLevel          = 2
 	DefaultNumLevel                      = 7
+	DefaultOpenFilesCacher               = LRUCacher
+	DefaultOpenFilesCacheCapacity        = 500
 	DefaultWriteBuffer                   = 4 * MiB
 	DefaultWriteL0PauseTrigger           = 12
 	DefaultWriteL0SlowdownTrigger        = 8
 )
 
-type noCache struct{}
+// Cacher is a caching algorithm.
+type Cacher interface {
+	New(capacity int) cache.Cacher
+}
 
-func (noCache) SetCapacity(capacity int)                     {}
-func (noCache) Capacity() int                                { return 0 }
-func (noCache) Used() int                                    { return 0 }
-func (noCache) Size() int                                    { return 0 }
-func (noCache) NumObjects() int                              { return 0 }
-func (noCache) GetNamespace(id uint64) cache.Namespace       { return nil }
-func (noCache) PurgeNamespace(id uint64, fin cache.PurgeFin) {}
-func (noCache) ZapNamespace(id uint64)                       {}
-func (noCache) Purge(fin cache.PurgeFin)                     {}
-func (noCache) Zap()                                         {}
+type CacherFunc struct {
+	NewFunc func(capacity int) cache.Cacher
+}
 
-var NoCache cache.Cache = noCache{}
+func (f *CacherFunc) New(capacity int) cache.Cacher {
+	if f.NewFunc != nil {
+		return f.NewFunc(capacity)
+	}
+	return nil
+}
 
-// Compression is the per-block compression algorithm to use.
+func noCacher(int) cache.Cacher { return nil }
+
+var (
+	// LRUCacher is the LRU-cache algorithm.
+	LRUCacher = &CacherFunc{cache.NewLRU}
+
+	// NoCacher is the value to disable caching algorithm.
+	NoCacher = &CacherFunc{}
+)
+
+// Compression is the 'sorted table' block compression algorithm to use.
 type Compression uint
 
 func (c Compression) String() string {
@@ -133,16 +147,17 @@ type Options struct {
 	// The default value is nil
 	AltFilters []filter.Filter
 
-	// BlockCache provides per-block caching for LevelDB. Specify NoCache to
-	// disable block caching.
+	// BlockCacher provides cache algorithm for LevelDB 'sorted table' block caching.
+	// Specify NoCacher to disable caching algorithm.
 	//
-	// By default LevelDB will create LRU-cache with capacity of BlockCacheSize.
-	BlockCache cache.Cache
+	// The default value is LRUCacher.
+	BlockCacher Cacher
 
-	// BlockCacheSize defines the capacity of the default 'block cache'.
+	// BlockCacheCapacity defines the capacity of the 'sorted table' block caching.
+	// Use -1 for zero, this has same effect as specifying NoCacher to BlockCacher.
 	//
 	// The default value is 8MiB.
-	BlockCacheSize int
+	BlockCacheCapacity int
 
 	// BlockRestartInterval is the number of keys between restart points for
 	// delta encoding of keys.
@@ -156,13 +171,6 @@ type Options struct {
 	// The default value is 4KiB.
 	BlockSize int
 
-	// CachedOpenFiles defines number of open files to kept around when not
-	// in-use, the counting includes still in-use files.
-	// Set this to negative value to disable caching.
-	//
-	// The default value is 500.
-	CachedOpenFiles int
-
 	// CompactionExpandLimitFactor limits compaction size after expanded.
 	// This will be multiplied by table size limit at compaction target level.
 	//
@@ -237,11 +245,22 @@ type Options struct {
 	// The default value uses the same ordering as bytes.Compare.
 	Comparer comparer.Comparer
 
-	// Compression defines the per-block compression to use.
+	// Compression defines the 'sorted table' block compression to use.
 	//
 	// The default value (DefaultCompression) uses snappy compression.
 	Compression Compression
 
+	// DisableBufferPool allows disable use of util.BufferPool functionality.
+	//
+	// The default value is false.
+	DisableBufferPool bool
+
+	// DisableBlockCache allows disable use of cache.Cache functionality on
+	// 'sorted table' block.
+	//
+	// The default value is false.
+	DisableBlockCache bool
+
 	// DisableCompactionBackoff allows disable compaction retry backoff.
 	//
 	// The default value is false.
@@ -275,6 +294,13 @@ type Options struct {
 	// The default value is nil.
 	Filter filter.Filter
 
+	// IteratorSamplingRate defines approximate gap (in bytes) between read
+	// sampling of an iterator. The samples will be used to determine when
+	// compaction should be triggered.
+	//
+	// The default is 1MiB.
+	IteratorSamplingRate int
+
 	// MaxMemCompationLevel defines maximum level a newly compacted 'memdb'
 	// will be pushed into if doesn't creates overlap. This should less than
 	// NumLevel. Use -1 for level-0.
@@ -282,12 +308,34 @@ type Options struct {
 	// The default is 2.
 	MaxMemCompationLevel int
 
+	// NoSync allows completely disable fsync.
+	//
+	// The default is false.
+	NoSync bool
+
 	// NumLevel defines number of database level. The level shouldn't changed
 	// between opens, or the database will panic.
 	//
 	// The default is 7.
 	NumLevel int
 
+	// OpenFilesCacher provides cache algorithm for open files caching.
+	// Specify NoCacher to disable caching algorithm.
+	//
+	// The default value is LRUCacher.
+	OpenFilesCacher Cacher
+
+	// OpenFilesCacheCapacity defines the capacity of the open files caching.
+	// Use -1 for zero, this has same effect as specifying NoCacher to OpenFilesCacher.
+	//
+	// The default value is 500.
+	OpenFilesCacheCapacity int
+
+	// If true then opens DB in read-only mode.
+	//
+	// The default value is false.
+	ReadOnly bool
+
 	// Strict defines the DB strict level.
 	Strict Strict
 
@@ -320,18 +368,22 @@ func (o *Options) GetAltFilters() []filter.Filter {
 	return o.AltFilters
 }
 
-func (o *Options) GetBlockCache() cache.Cache {
-	if o == nil {
+func (o *Options) GetBlockCacher() Cacher {
+	if o == nil || o.BlockCacher == nil {
+		return DefaultBlockCacher
+	} else if o.BlockCacher == NoCacher {
 		return nil
 	}
-	return o.BlockCache
+	return o.BlockCacher
 }
 
-func (o *Options) GetBlockCacheSize() int {
-	if o == nil || o.BlockCacheSize <= 0 {
-		return DefaultBlockCacheSize
+func (o *Options) GetBlockCacheCapacity() int {
+	if o == nil || o.BlockCacheCapacity == 0 {
+		return DefaultBlockCacheCapacity
+	} else if o.BlockCacheCapacity < 0 {
+		return 0
 	}
-	return o.BlockCacheSize
+	return o.BlockCacheCapacity
 }
 
 func (o *Options) GetBlockRestartInterval() int {
@@ -348,15 +400,6 @@ func (o *Options) GetBlockSize() int {
 	return o.BlockSize
 }
 
-func (o *Options) GetCachedOpenFiles() int {
-	if o == nil || o.CachedOpenFiles == 0 {
-		return DefaultCachedOpenFiles
-	} else if o.CachedOpenFiles < 0 {
-		return 0
-	}
-	return o.CachedOpenFiles
-}
-
 func (o *Options) GetCompactionExpandLimit(level int) int {
 	factor := DefaultCompactionExpandLimitFactor
 	if o != nil && o.CompactionExpandLimitFactor > 0 {
@@ -444,6 +487,20 @@ func (o *Options) GetCompression() Compression {
 	return o.Compression
 }
 
+func (o *Options) GetDisableBufferPool() bool {
+	if o == nil {
+		return false
+	}
+	return o.DisableBufferPool
+}
+
+func (o *Options) GetDisableBlockCache() bool {
+	if o == nil {
+		return false
+	}
+	return o.DisableBlockCache
+}
+
 func (o *Options) GetDisableCompactionBackoff() bool {
 	if o == nil {
 		return false
@@ -472,12 +529,19 @@ func (o *Options) GetFilter() filter.Filter {
 	return o.Filter
 }
 
+func (o *Options) GetIteratorSamplingRate() int {
+	if o == nil || o.IteratorSamplingRate <= 0 {
+		return DefaultIteratorSamplingRate
+	}
+	return o.IteratorSamplingRate
+}
+
 func (o *Options) GetMaxMemCompationLevel() int {
 	level := DefaultMaxMemCompationLevel
 	if o != nil {
 		if o.MaxMemCompationLevel > 0 {
 			level = o.MaxMemCompationLevel
-		} else if o.MaxMemCompationLevel == -1 {
+		} else if o.MaxMemCompationLevel < 0 {
 			level = 0
 		}
 	}
@@ -487,6 +551,13 @@ func (o *Options) GetMaxMemCompationLevel() int {
 	return level
 }
 
+func (o *Options) GetNoSync() bool {
+	if o == nil {
+		return false
+	}
+	return o.NoSync
+}
+
 func (o *Options) GetNumLevel() int {
 	if o == nil || o.NumLevel <= 0 {
 		return DefaultNumLevel
@@ -494,6 +565,32 @@ func (o *Options) GetNumLevel() int {
 	return o.NumLevel
 }
 
+func (o *Options) GetOpenFilesCacher() Cacher {
+	if o == nil || o.OpenFilesCacher == nil {
+		return DefaultOpenFilesCacher
+	}
+	if o.OpenFilesCacher == NoCacher {
+		return nil
+	}
+	return o.OpenFilesCacher
+}
+
+func (o *Options) GetOpenFilesCacheCapacity() int {
+	if o == nil || o.OpenFilesCacheCapacity == 0 {
+		return DefaultOpenFilesCacheCapacity
+	} else if o.OpenFilesCacheCapacity < 0 {
+		return 0
+	}
+	return o.OpenFilesCacheCapacity
+}
+
+func (o *Options) GetReadOnly() bool {
+	if o == nil {
+		return false
+	}
+	return o.ReadOnly
+}
+
 func (o *Options) GetStrict(strict Strict) bool {
 	if o == nil || o.Strict == 0 {
 		return DefaultStrict&strict != 0

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/options.go

@@ -7,7 +7,6 @@
 package leveldb
 
 import (
-	"github.com/syndtr/goleveldb/leveldb/cache"
 	"github.com/syndtr/goleveldb/leveldb/filter"
 	"github.com/syndtr/goleveldb/leveldb/opt"
 )
@@ -32,13 +31,6 @@ func (s *session) setOptions(o *opt.Options) {
 			no.AltFilters[i] = &iFilter{filter}
 		}
 	}
-	// Block cache.
-	switch o.GetBlockCache() {
-	case nil:
-		no.BlockCache = cache.NewLRUCache(o.GetBlockCacheSize())
-	case opt.NoCache:
-		no.BlockCache = nil
-	}
 	// Comparer.
 	s.icmp = &iComparer{o.GetComparer()}
 	no.Comparer = s.icmp

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/session.go

@@ -11,10 +11,8 @@ import (
 	"io"
 	"os"
 	"sync"
-	"sync/atomic"
 
 	"github.com/syndtr/goleveldb/leveldb/errors"
-	"github.com/syndtr/goleveldb/leveldb/iterator"
 	"github.com/syndtr/goleveldb/leveldb/journal"
 	"github.com/syndtr/goleveldb/leveldb/opt"
 	"github.com/syndtr/goleveldb/leveldb/storage"
@@ -73,7 +71,7 @@ func newSession(stor storage.Storage, o *opt.Options) (s *session, err error) {
 		stCompPtrs: make([]iKey, o.GetNumLevel()),
 	}
 	s.setOptions(o)
-	s.tops = newTableOps(s, s.o.GetCachedOpenFiles())
+	s.tops = newTableOps(s)
 	s.setVersion(newVersion(s))
 	s.log("log@legend F·NumFile S·FileSize N·Entry C·BadEntry B·BadBlock Ke·KeyError D·DroppedEntry L·Level Q·SeqNum T·TimeElapsed")
 	return
@@ -82,9 +80,6 @@ func newSession(stor storage.Storage, o *opt.Options) (s *session, err error) {
 // Close session.
 func (s *session) close() {
 	s.tops.close()
-	if bc := s.o.GetBlockCache(); bc != nil {
-		bc.Purge(nil)
-	}
 	if s.manifest != nil {
 		s.manifest.Close()
 	}
@@ -130,11 +125,16 @@ func (s *session) recover() (err error) {
 		return
 	}
 	defer reader.Close()
-	strict := s.o.GetStrict(opt.StrictManifest)
-	jr := journal.NewReader(reader, dropper{s, m}, strict, true)
 
-	staging := s.stVersion.newStaging()
-	rec := &sessionRecord{numLevel: s.o.GetNumLevel()}
+	var (
+		// Options.
+		numLevel = s.o.GetNumLevel()
+		strict   = s.o.GetStrict(opt.StrictManifest)
+
+		jr      = journal.NewReader(reader, dropper{s, m}, strict, true)
+		rec     = &sessionRecord{}
+		staging = s.stVersion.newStaging()
+	)
 	for {
 		var r io.Reader
 		r, err = jr.Next()
@@ -146,7 +146,7 @@ func (s *session) recover() (err error) {
 			return errors.SetFile(err, m)
 		}
 
-		err = rec.decode(r)
+		err = rec.decode(r, numLevel)
 		if err == nil {
 			// save compact pointers
 			for _, r := range rec.compPtrs {
@@ -209,250 +209,3 @@ func (s *session) commit(r *sessionRecord) (err error) {
 
 	return
 }
-
-// Pick a compaction based on current state; need external synchronization.
-func (s *session) pickCompaction() *compaction {
-	v := s.version()
-
-	var level int
-	var t0 tFiles
-	if v.cScore >= 1 {
-		level = v.cLevel
-		cptr := s.stCompPtrs[level]
-		tables := v.tables[level]
-		for _, t := range tables {
-			if cptr == nil || s.icmp.Compare(t.imax, cptr) > 0 {
-				t0 = append(t0, t)
-				break
-			}
-		}
-		if len(t0) == 0 {
-			t0 = append(t0, tables[0])
-		}
-	} else {
-		if p := atomic.LoadPointer(&v.cSeek); p != nil {
-			ts := (*tSet)(p)
-			level = ts.level
-			t0 = append(t0, ts.table)
-		} else {
-			v.release()
-			return nil
-		}
-	}
-
-	return newCompaction(s, v, level, t0)
-}
-
-// Create compaction from given level and range; need external synchronization.
-func (s *session) getCompactionRange(level int, umin, umax []byte) *compaction {
-	v := s.version()
-
-	t0 := v.tables[level].getOverlaps(nil, s.icmp, umin, umax, level == 0)
-	if len(t0) == 0 {
-		v.release()
-		return nil
-	}
-
-	// Avoid compacting too much in one shot in case the range is large.
-	// But we cannot do this for level-0 since level-0 files can overlap
-	// and we must not pick one file and drop another older file if the
-	// two files overlap.
-	if level > 0 {
-		limit := uint64(v.s.o.GetCompactionSourceLimit(level))
-		total := uint64(0)
-		for i, t := range t0 {
-			total += t.size
-			if total >= limit {
-				s.logf("table@compaction limiting F·%d -> F·%d", len(t0), i+1)
-				t0 = t0[:i+1]
-				break
-			}
-		}
-	}
-
-	return newCompaction(s, v, level, t0)
-}
-
-func newCompaction(s *session, v *version, level int, t0 tFiles) *compaction {
-	c := &compaction{
-		s:             s,
-		v:             v,
-		level:         level,
-		tables:        [2]tFiles{t0, nil},
-		maxGPOverlaps: uint64(s.o.GetCompactionGPOverlaps(level)),
-		tPtrs:         make([]int, s.o.GetNumLevel()),
-	}
-	c.expand()
-	c.save()
-	return c
-}
-
-// compaction represent a compaction state.
-type compaction struct {
-	s *session
-	v *version
-
-	level         int
-	tables        [2]tFiles
-	maxGPOverlaps uint64
-
-	gp                tFiles
-	gpi               int
-	seenKey           bool
-	gpOverlappedBytes uint64
-	imin, imax        iKey
-	tPtrs             []int
-	released          bool
-
-	snapGPI               int
-	snapSeenKey           bool
-	snapGPOverlappedBytes uint64
-	snapTPtrs             []int
-}
-
-func (c *compaction) save() {
-	c.snapGPI = c.gpi
-	c.snapSeenKey = c.seenKey
-	c.snapGPOverlappedBytes = c.gpOverlappedBytes
-	c.snapTPtrs = append(c.snapTPtrs[:0], c.tPtrs...)
-}
-
-func (c *compaction) restore() {
-	c.gpi = c.snapGPI
-	c.seenKey = c.snapSeenKey
-	c.gpOverlappedBytes = c.snapGPOverlappedBytes
-	c.tPtrs = append(c.tPtrs[:0], c.snapTPtrs...)
-}
-
-func (c *compaction) release() {
-	if !c.released {
-		c.released = true
-		c.v.release()
-	}
-}
-
-// Expand compacted tables; need external synchronization.
-func (c *compaction) expand() {
-	limit := uint64(c.s.o.GetCompactionExpandLimit(c.level))
-	vt0, vt1 := c.v.tables[c.level], c.v.tables[c.level+1]
-
-	t0, t1 := c.tables[0], c.tables[1]
-	imin, imax := t0.getRange(c.s.icmp)
-	// We expand t0 here just incase ukey hop across tables.
-	t0 = vt0.getOverlaps(t0, c.s.icmp, imin.ukey(), imax.ukey(), c.level == 0)
-	if len(t0) != len(c.tables[0]) {
-		imin, imax = t0.getRange(c.s.icmp)
-	}
-	t1 = vt1.getOverlaps(t1, c.s.icmp, imin.ukey(), imax.ukey(), false)
-	// Get entire range covered by compaction.
-	amin, amax := append(t0, t1...).getRange(c.s.icmp)
-
-	// See if we can grow the number of inputs in "level" without
-	// changing the number of "level+1" files we pick up.
-	if len(t1) > 0 {
-		exp0 := vt0.getOverlaps(nil, c.s.icmp, amin.ukey(), amax.ukey(), c.level == 0)
-		if len(exp0) > len(t0) && t1.size()+exp0.size() < limit {
-			xmin, xmax := exp0.getRange(c.s.icmp)
-			exp1 := vt1.getOverlaps(nil, c.s.icmp, xmin.ukey(), xmax.ukey(), false)
-			if len(exp1) == len(t1) {
-				c.s.logf("table@compaction expanding L%d+L%d (F·%d S·%s)+(F·%d S·%s) -> (F·%d S·%s)+(F·%d S·%s)",
-					c.level, c.level+1, len(t0), shortenb(int(t0.size())), len(t1), shortenb(int(t1.size())),
-					len(exp0), shortenb(int(exp0.size())), len(exp1), shortenb(int(exp1.size())))
-				imin, imax = xmin, xmax
-				t0, t1 = exp0, exp1
-				amin, amax = append(t0, t1...).getRange(c.s.icmp)
-			}
-		}
-	}
-
-	// Compute the set of grandparent files that overlap this compaction
-	// (parent == level+1; grandparent == level+2)
-	if c.level+2 < c.s.o.GetNumLevel() {
-		c.gp = c.v.tables[c.level+2].getOverlaps(c.gp, c.s.icmp, amin.ukey(), amax.ukey(), false)
-	}
-
-	c.tables[0], c.tables[1] = t0, t1
-	c.imin, c.imax = imin, imax
-}
-
-// Check whether compaction is trivial.
-func (c *compaction) trivial() bool {
-	return len(c.tables[0]) == 1 && len(c.tables[1]) == 0 && c.gp.size() <= c.maxGPOverlaps
-}
-
-func (c *compaction) baseLevelForKey(ukey []byte) bool {
-	for level, tables := range c.v.tables[c.level+2:] {
-		for c.tPtrs[level] < len(tables) {
-			t := tables[c.tPtrs[level]]
-			if c.s.icmp.uCompare(ukey, t.imax.ukey()) <= 0 {
-				// We've advanced far enough.
-				if c.s.icmp.uCompare(ukey, t.imin.ukey()) >= 0 {
-					// Key falls in this file's range, so definitely not base level.
-					return false
-				}
-				break
-			}
-			c.tPtrs[level]++
-		}
-	}
-	return true
-}
-
-func (c *compaction) shouldStopBefore(ikey iKey) bool {
-	for ; c.gpi < len(c.gp); c.gpi++ {
-		gp := c.gp[c.gpi]
-		if c.s.icmp.Compare(ikey, gp.imax) <= 0 {
-			break
-		}
-		if c.seenKey {
-			c.gpOverlappedBytes += gp.size
-		}
-	}
-	c.seenKey = true
-
-	if c.gpOverlappedBytes > c.maxGPOverlaps {
-		// Too much overlap for current output; start new output.
-		c.gpOverlappedBytes = 0
-		return true
-	}
-	return false
-}
-
-// Creates an iterator.
-func (c *compaction) newIterator() iterator.Iterator {
-	// Creates iterator slice.
-	icap := len(c.tables)
-	if c.level == 0 {
-		// Special case for level-0
-		icap = len(c.tables[0]) + 1
-	}
-	its := make([]iterator.Iterator, 0, icap)
-
-	// Options.
-	ro := &opt.ReadOptions{
-		DontFillCache: true,
-		Strict:        opt.StrictOverride,
-	}
-	strict := c.s.o.GetStrict(opt.StrictCompaction)
-	if strict {
-		ro.Strict |= opt.StrictReader
-	}
-
-	for i, tables := range c.tables {
-		if len(tables) == 0 {
-			continue
-		}
-
-		// Level-0 is not sorted and may overlaps each other.
-		if c.level+i == 0 {
-			for _, t := range tables {
-				its = append(its, c.s.tops.newIterator(t, nil, ro))
-			}
-		} else {
-			it := iterator.NewIndexedIterator(tables.newIndexIterator(c.s.tops, c.s.icmp, nil, ro), strict)
-			its = append(its, it)
-		}
-	}
-
-	return iterator.NewMergedIterator(its, c.s.icmp, strict)
-}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/session_compaction.go

@@ -0,0 +1,287 @@
+// Copyright (c) 2012, Suryandaru Triandana <syndtr@gmail.com>
+// All rights reserved.
+//
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+package leveldb
+
+import (
+	"sync/atomic"
+
+	"github.com/syndtr/goleveldb/leveldb/iterator"
+	"github.com/syndtr/goleveldb/leveldb/memdb"
+	"github.com/syndtr/goleveldb/leveldb/opt"
+)
+
+func (s *session) pickMemdbLevel(umin, umax []byte) int {
+	v := s.version()
+	defer v.release()
+	return v.pickMemdbLevel(umin, umax)
+}
+
+func (s *session) flushMemdb(rec *sessionRecord, mdb *memdb.DB, level int) (level_ int, err error) {
+	// Create sorted table.
+	iter := mdb.NewIterator(nil)
+	defer iter.Release()
+	t, n, err := s.tops.createFrom(iter)
+	if err != nil {
+		return level, err
+	}
+
+	// Pick level and add to record.
+	if level < 0 {
+		level = s.pickMemdbLevel(t.imin.ukey(), t.imax.ukey())
+	}
+	rec.addTableFile(level, t)
+
+	s.logf("memdb@flush created L%d@%d N·%d S·%s %q:%q", level, t.file.Num(), n, shortenb(int(t.size)), t.imin, t.imax)
+	return level, nil
+}
+
+// Pick a compaction based on current state; need external synchronization.
+func (s *session) pickCompaction() *compaction {
+	v := s.version()
+
+	var level int
+	var t0 tFiles
+	if v.cScore >= 1 {
+		level = v.cLevel
+		cptr := s.stCompPtrs[level]
+		tables := v.tables[level]
+		for _, t := range tables {
+			if cptr == nil || s.icmp.Compare(t.imax, cptr) > 0 {
+				t0 = append(t0, t)
+				break
+			}
+		}
+		if len(t0) == 0 {
+			t0 = append(t0, tables[0])
+		}
+	} else {
+		if p := atomic.LoadPointer(&v.cSeek); p != nil {
+			ts := (*tSet)(p)
+			level = ts.level
+			t0 = append(t0, ts.table)
+		} else {
+			v.release()
+			return nil
+		}
+	}
+
+	return newCompaction(s, v, level, t0)
+}
+
+// Create compaction from given level and range; need external synchronization.
+func (s *session) getCompactionRange(level int, umin, umax []byte) *compaction {
+	v := s.version()
+
+	t0 := v.tables[level].getOverlaps(nil, s.icmp, umin, umax, level == 0)
+	if len(t0) == 0 {
+		v.release()
+		return nil
+	}
+
+	// Avoid compacting too much in one shot in case the range is large.
+	// But we cannot do this for level-0 since level-0 files can overlap
+	// and we must not pick one file and drop another older file if the
+	// two files overlap.
+	if level > 0 {
+		limit := uint64(v.s.o.GetCompactionSourceLimit(level))
+		total := uint64(0)
+		for i, t := range t0 {
+			total += t.size
+			if total >= limit {
+				s.logf("table@compaction limiting F·%d -> F·%d", len(t0), i+1)
+				t0 = t0[:i+1]
+				break
+			}
+		}
+	}
+
+	return newCompaction(s, v, level, t0)
+}
+
+func newCompaction(s *session, v *version, level int, t0 tFiles) *compaction {
+	c := &compaction{
+		s:             s,
+		v:             v,
+		level:         level,
+		tables:        [2]tFiles{t0, nil},
+		maxGPOverlaps: uint64(s.o.GetCompactionGPOverlaps(level)),
+		tPtrs:         make([]int, s.o.GetNumLevel()),
+	}
+	c.expand()
+	c.save()
+	return c
+}
+
+// compaction represent a compaction state.
+type compaction struct {
+	s *session
+	v *version
+
+	level         int
+	tables        [2]tFiles
+	maxGPOverlaps uint64
+
+	gp                tFiles
+	gpi               int
+	seenKey           bool
+	gpOverlappedBytes uint64
+	imin, imax        iKey
+	tPtrs             []int
+	released          bool
+
+	snapGPI               int
+	snapSeenKey           bool
+	snapGPOverlappedBytes uint64
+	snapTPtrs             []int
+}
+
+func (c *compaction) save() {
+	c.snapGPI = c.gpi
+	c.snapSeenKey = c.seenKey
+	c.snapGPOverlappedBytes = c.gpOverlappedBytes
+	c.snapTPtrs = append(c.snapTPtrs[:0], c.tPtrs...)
+}
+
+func (c *compaction) restore() {
+	c.gpi = c.snapGPI
+	c.seenKey = c.snapSeenKey
+	c.gpOverlappedBytes = c.snapGPOverlappedBytes
+	c.tPtrs = append(c.tPtrs[:0], c.snapTPtrs...)
+}
+
+func (c *compaction) release() {
+	if !c.released {
+		c.released = true
+		c.v.release()
+	}
+}
+
+// Expand compacted tables; need external synchronization.
+func (c *compaction) expand() {
+	limit := uint64(c.s.o.GetCompactionExpandLimit(c.level))
+	vt0, vt1 := c.v.tables[c.level], c.v.tables[c.level+1]
+
+	t0, t1 := c.tables[0], c.tables[1]
+	imin, imax := t0.getRange(c.s.icmp)
+	// We expand t0 here just incase ukey hop across tables.
+	t0 = vt0.getOverlaps(t0, c.s.icmp, imin.ukey(), imax.ukey(), c.level == 0)
+	if len(t0) != len(c.tables[0]) {
+		imin, imax = t0.getRange(c.s.icmp)
+	}
+	t1 = vt1.getOverlaps(t1, c.s.icmp, imin.ukey(), imax.ukey(), false)
+	// Get entire range covered by compaction.
+	amin, amax := append(t0, t1...).getRange(c.s.icmp)
+
+	// See if we can grow the number of inputs in "level" without
+	// changing the number of "level+1" files we pick up.
+	if len(t1) > 0 {
+		exp0 := vt0.getOverlaps(nil, c.s.icmp, amin.ukey(), amax.ukey(), c.level == 0)
+		if len(exp0) > len(t0) && t1.size()+exp0.size() < limit {
+			xmin, xmax := exp0.getRange(c.s.icmp)
+			exp1 := vt1.getOverlaps(nil, c.s.icmp, xmin.ukey(), xmax.ukey(), false)
+			if len(exp1) == len(t1) {
+				c.s.logf("table@compaction expanding L%d+L%d (F·%d S·%s)+(F·%d S·%s) -> (F·%d S·%s)+(F·%d S·%s)",
+					c.level, c.level+1, len(t0), shortenb(int(t0.size())), len(t1), shortenb(int(t1.size())),
+					len(exp0), shortenb(int(exp0.size())), len(exp1), shortenb(int(exp1.size())))
+				imin, imax = xmin, xmax
+				t0, t1 = exp0, exp1
+				amin, amax = append(t0, t1...).getRange(c.s.icmp)
+			}
+		}
+	}
+
+	// Compute the set of grandparent files that overlap this compaction
+	// (parent == level+1; grandparent == level+2)
+	if c.level+2 < c.s.o.GetNumLevel() {
+		c.gp = c.v.tables[c.level+2].getOverlaps(c.gp, c.s.icmp, amin.ukey(), amax.ukey(), false)
+	}
+
+	c.tables[0], c.tables[1] = t0, t1
+	c.imin, c.imax = imin, imax
+}
+
+// Check whether compaction is trivial.
+func (c *compaction) trivial() bool {
+	return len(c.tables[0]) == 1 && len(c.tables[1]) == 0 && c.gp.size() <= c.maxGPOverlaps
+}
+
+func (c *compaction) baseLevelForKey(ukey []byte) bool {
+	for level, tables := range c.v.tables[c.level+2:] {
+		for c.tPtrs[level] < len(tables) {
+			t := tables[c.tPtrs[level]]
+			if c.s.icmp.uCompare(ukey, t.imax.ukey()) <= 0 {
+				// We've advanced far enough.
+				if c.s.icmp.uCompare(ukey, t.imin.ukey()) >= 0 {
+					// Key falls in this file's range, so definitely not base level.
+					return false
+				}
+				break
+			}
+			c.tPtrs[level]++
+		}
+	}
+	return true
+}
+
+func (c *compaction) shouldStopBefore(ikey iKey) bool {
+	for ; c.gpi < len(c.gp); c.gpi++ {
+		gp := c.gp[c.gpi]
+		if c.s.icmp.Compare(ikey, gp.imax) <= 0 {
+			break
+		}
+		if c.seenKey {
+			c.gpOverlappedBytes += gp.size
+		}
+	}
+	c.seenKey = true
+
+	if c.gpOverlappedBytes > c.maxGPOverlaps {
+		// Too much overlap for current output; start new output.
+		c.gpOverlappedBytes = 0
+		return true
+	}
+	return false
+}
+
+// Creates an iterator.
+func (c *compaction) newIterator() iterator.Iterator {
+	// Creates iterator slice.
+	icap := len(c.tables)
+	if c.level == 0 {
+		// Special case for level-0.
+		icap = len(c.tables[0]) + 1
+	}
+	its := make([]iterator.Iterator, 0, icap)
+
+	// Options.
+	ro := &opt.ReadOptions{
+		DontFillCache: true,
+		Strict:        opt.StrictOverride,
+	}
+	strict := c.s.o.GetStrict(opt.StrictCompaction)
+	if strict {
+		ro.Strict |= opt.StrictReader
+	}
+
+	for i, tables := range c.tables {
+		if len(tables) == 0 {
+			continue
+		}
+
+		// Level-0 is not sorted and may overlaps each other.
+		if c.level+i == 0 {
+			for _, t := range tables {
+				its = append(its, c.s.tops.newIterator(t, nil, ro))
+			}
+		} else {
+			it := iterator.NewIndexedIterator(tables.newIndexIterator(c.s.tops, c.s.icmp, nil, ro), strict)
+			its = append(its, it)
+		}
+	}
+
+	return iterator.NewMergedIterator(its, c.s.icmp, strict)
+}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/session_record.go

@@ -52,8 +52,6 @@ type dtRecord struct {
 }
 
 type sessionRecord struct {
-	numLevel int
-
 	hasRec         int
 	comparer       string
 	journalNum     uint64
@@ -230,7 +228,7 @@ func (p *sessionRecord) readBytes(field string, r byteReader) []byte {
 	return x
 }
 
-func (p *sessionRecord) readLevel(field string, r io.ByteReader) int {
+func (p *sessionRecord) readLevel(field string, r io.ByteReader, numLevel int) int {
 	if p.err != nil {
 		return 0
 	}
@@ -238,14 +236,14 @@ func (p *sessionRecord) readLevel(field string, r io.ByteReader) int {
 	if p.err != nil {
 		return 0
 	}
-	if x >= uint64(p.numLevel) {
+	if x >= uint64(numLevel) {
 		p.err = errors.NewErrCorrupted(nil, &ErrManifestCorrupted{field, "invalid level number"})
 		return 0
 	}
 	return int(x)
 }
 
-func (p *sessionRecord) decode(r io.Reader) error {
+func (p *sessionRecord) decode(r io.Reader, numLevel int) error {
 	br, ok := r.(byteReader)
 	if !ok {
 		br = bufio.NewReader(r)
@@ -286,13 +284,13 @@ func (p *sessionRecord) decode(r io.Reader) error {
 				p.setSeqNum(x)
 			}
 		case recCompPtr:
-			level := p.readLevel("comp-ptr.level", br)
+			level := p.readLevel("comp-ptr.level", br, numLevel)
 			ikey := p.readBytes("comp-ptr.ikey", br)
 			if p.err == nil {
 				p.addCompPtr(level, iKey(ikey))
 			}
 		case recAddTable:
-			level := p.readLevel("add-table.level", br)
+			level := p.readLevel("add-table.level", br, numLevel)
 			num := p.readUvarint("add-table.num", br)
 			size := p.readUvarint("add-table.size", br)
 			imin := p.readBytes("add-table.imin", br)
@@ -301,7 +299,7 @@ func (p *sessionRecord) decode(r io.Reader) error {
 				p.addTable(level, num, size, imin, imax)
 			}
 		case recDelTable:
-			level := p.readLevel("del-table.level", br)
+			level := p.readLevel("del-table.level", br, numLevel)
 			num := p.readUvarint("del-table.num", br)
 			if p.err == nil {
 				p.delTable(level, num)

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/session_record_test.go

@@ -19,8 +19,8 @@ func decodeEncode(v *sessionRecord) (res bool, err error) {
 	if err != nil {
 		return
 	}
-	v2 := &sessionRecord{numLevel: opt.DefaultNumLevel}
-	err = v.decode(b)
+	v2 := &sessionRecord{}
+	err = v.decode(b, opt.DefaultNumLevel)
 	if err != nil {
 		return
 	}
@@ -34,7 +34,7 @@ func decodeEncode(v *sessionRecord) (res bool, err error) {
 
 func TestSessionRecord_EncodeDecode(t *testing.T) {
 	big := uint64(1) << 50
-	v := &sessionRecord{numLevel: opt.DefaultNumLevel}
+	v := &sessionRecord{}
 	i := uint64(0)
 	test := func() {
 		res, err := decodeEncode(v)

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/session_util.go

@@ -182,7 +182,7 @@ func (s *session) newManifest(rec *sessionRecord, v *version) (err error) {
 		defer v.release()
 	}
 	if rec == nil {
-		rec = &sessionRecord{numLevel: s.o.GetNumLevel()}
+		rec = &sessionRecord{}
 	}
 	s.fillRecord(rec, true)
 	v.fillRecord(rec)
@@ -240,9 +240,11 @@ func (s *session) flushManifest(rec *sessionRecord) (err error) {
 	if err != nil {
 		return
 	}
-	err = s.manifestWriter.Sync()
-	if err != nil {
-		return
+	if !s.o.GetNoSync() {
+		err = s.manifestWriter.Sync()
+		if err != nil {
+			return
+		}
 	}
 	s.recordCommited(rec)
 	return

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go

@@ -221,7 +221,7 @@ func (fs *fileStorage) GetManifest() (f File, err error) {
 					fs.log(fmt.Sprintf("skipping %s: invalid file name", fn))
 					continue
 				}
-				if _, e1 := strconv.ParseUint(fn[7:], 10, 0); e1 != nil {
+				if _, e1 := strconv.ParseUint(fn[8:], 10, 0); e1 != nil {
 					fs.log(fmt.Sprintf("skipping %s: invalid file num: %v", fn, e1))
 					continue
 				}
@@ -243,7 +243,10 @@ func (fs *fileStorage) GetManifest() (f File, err error) {
 					rem = append(rem, fn)
 				}
 				if !pend1 || cerr == nil {
-					cerr = fmt.Errorf("leveldb/storage: corrupted or incomplete %s file", fn)
+					cerr = &ErrCorrupted{
+						File: fsParseName(filepath.Base(fn)),
+						Err:  errors.New("leveldb/storage: corrupted or incomplete manifest file"),
+					}
 				}
 			} else if f != nil && f1.Num() < f.Num() {
 				fs.log(fmt.Sprintf("skipping %s: obsolete", fn))
@@ -326,8 +329,7 @@ func (fs *fileStorage) Close() error {
 	runtime.SetFinalizer(fs, nil)
 
 	if fs.open > 0 {
-		fs.log(fmt.Sprintf("refuse to close, %d files still open", fs.open))
-		return fmt.Errorf("leveldb/storage: cannot close, %d files still open", fs.open)
+		fs.log(fmt.Sprintf("close: warning, %d files still open", fs.open))
 	}
 	fs.open = -1
 	e1 := fs.logw.Close()
@@ -505,30 +507,37 @@ func (f *file) path() string {
 	return filepath.Join(f.fs.path, f.name())
 }
 
-func (f *file) parse(name string) bool {
-	var num uint64
+func fsParseName(name string) *FileInfo {
+	fi := &FileInfo{}
 	var tail string
-	_, err := fmt.Sscanf(name, "%d.%s", &num, &tail)
+	_, err := fmt.Sscanf(name, "%d.%s", &fi.Num, &tail)
 	if err == nil {
 		switch tail {
 		case "log":
-			f.t = TypeJournal
+			fi.Type = TypeJournal
 		case "ldb", "sst":
-			f.t = TypeTable
+			fi.Type = TypeTable
 		case "tmp":
-			f.t = TypeTemp
+			fi.Type = TypeTemp
 		default:
-			return false
+			return nil
 		}
-		f.num = num
-		return true
+		return fi
 	}
-	n, _ := fmt.Sscanf(name, "MANIFEST-%d%s", &num, &tail)
+	n, _ := fmt.Sscanf(name, "MANIFEST-%d%s", &fi.Num, &tail)
 	if n == 1 {
-		f.t = TypeManifest
-		f.num = num
-		return true
+		fi.Type = TypeManifest
+		return fi
 	}
-
-	return false
+	return nil
+}
+
+func (f *file) parse(name string) bool {
+	fi := fsParseName(name)
+	if fi == nil {
+		return false
+	}
+	f.t = fi.Type
+	f.num = fi.Num
+	return true
 }

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage_unix.go

@@ -50,13 +50,23 @@ func rename(oldpath, newpath string) error {
 	return os.Rename(oldpath, newpath)
 }
 
+func isErrInvalid(err error) bool {
+	if err == os.ErrInvalid {
+		return true
+	}
+	if syserr, ok := err.(*os.SyscallError); ok && syserr.Err == syscall.EINVAL {
+		return true
+	}
+	return false
+}
+
 func syncDir(name string) error {
 	f, err := os.Open(name)
 	if err != nil {
 		return err
 	}
 	defer f.Close()
-	if err := f.Sync(); err != nil {
+	if err := f.Sync(); err != nil && !isErrInvalid(err) {
 		return err
 	}
 	return nil

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/storage/storage.go

@@ -46,6 +46,22 @@ var (
 	ErrClosed      = errors.New("leveldb/storage: closed")
 )
 
+// ErrCorrupted is the type that wraps errors that indicate corruption of
+// a file. Package storage has its own type instead of using
+// errors.ErrCorrupted to prevent circular import.
+type ErrCorrupted struct {
+	File *FileInfo
+	Err  error
+}
+
+func (e *ErrCorrupted) Error() string {
+	if e.File != nil {
+		return fmt.Sprintf("%v [file=%v]", e.Err, e.File)
+	} else {
+		return e.Err.Error()
+	}
+}
+
 // Syncer is the interface that wraps basic Sync method.
 type Syncer interface {
 	// Sync commits the current contents of the file to stable storage.

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/storage_test.go

@@ -42,6 +42,8 @@ type tsOp uint
 const (
 	tsOpOpen tsOp = iota
 	tsOpCreate
+	tsOpReplace
+	tsOpRemove
 	tsOpRead
 	tsOpReadAt
 	tsOpWrite
@@ -241,6 +243,10 @@ func (tf tsFile) Replace(newfile storage.File) (err error) {
 	if err != nil {
 		return
 	}
+	if tf.shouldErr(tsOpReplace) {
+		err = errors.New("leveldb.testStorage: emulated create error")
+		return
+	}
 	err = tf.File.Replace(newfile.(tsFile).File)
 	if err != nil {
 		ts.t.Errorf("E: cannot replace file, num=%d type=%v: %v", tf.Num(), tf.Type(), err)
@@ -258,6 +264,10 @@ func (tf tsFile) Remove() (err error) {
 	if err != nil {
 		return
 	}
+	if tf.shouldErr(tsOpRemove) {
+		err = errors.New("leveldb.testStorage: emulated create error")
+		return
+	}
 	err = tf.File.Remove()
 	if err != nil {
 		ts.t.Errorf("E: cannot remove file, num=%d type=%v: %v", tf.Num(), tf.Type(), err)

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table.go

@@ -286,10 +286,11 @@ func (x *tFilesSortByNum) Less(i, j int) bool {
 
 // Table operations.
 type tOps struct {
-	s       *session
-	cache   cache.Cache
-	cacheNS cache.Namespace
-	bpool   *util.BufferPool
+	s      *session
+	noSync bool
+	cache  *cache.Cache
+	bcache *cache.Cache
+	bpool  *util.BufferPool
 }
 
 // Creates an empty table and returns table writer.
@@ -338,26 +339,28 @@ func (t *tOps) createFrom(src iterator.Iterator) (f *tFile, n int, err error) {
 
 // Opens table. It returns a cache handle, which should
 // be released after use.
-func (t *tOps) open(f *tFile) (ch cache.Handle, err error) {
+func (t *tOps) open(f *tFile) (ch *cache.Handle, err error) {
 	num := f.file.Num()
-	ch = t.cacheNS.Get(num, func() (charge int, value interface{}) {
+	ch = t.cache.Get(0, num, func() (size int, value cache.Value) {
 		var r storage.Reader
 		r, err = f.file.Open()
 		if err != nil {
 			return 0, nil
 		}
 
-		var bcacheNS cache.Namespace
-		if bc := t.s.o.GetBlockCache(); bc != nil {
-			bcacheNS = bc.GetNamespace(num)
+		var bcache *cache.CacheGetter
+		if t.bcache != nil {
+			bcache = &cache.CacheGetter{Cache: t.bcache, NS: num}
 		}
+
 		var tr *table.Reader
-		tr, err = table.NewReader(r, int64(f.size), storage.NewFileInfo(f.file), bcacheNS, t.bpool, t.s.o.Options)
+		tr, err = table.NewReader(r, int64(f.size), storage.NewFileInfo(f.file), bcache, t.bpool, t.s.o.Options)
 		if err != nil {
 			r.Close()
 			return 0, nil
 		}
 		return 1, tr
+
 	})
 	if ch == nil && err == nil {
 		err = ErrClosed
@@ -412,16 +415,14 @@ func (t *tOps) newIterator(f *tFile, slice *util.Range, ro *opt.ReadOptions) ite
 // no one use the the table.
 func (t *tOps) remove(f *tFile) {
 	num := f.file.Num()
-	t.cacheNS.Delete(num, func(exist, pending bool) {
-		if !pending {
-			if err := f.file.Remove(); err != nil {
-				t.s.logf("table@remove removing @%d %q", num, err)
-			} else {
-				t.s.logf("table@remove removed @%d", num)
-			}
-			if bc := t.s.o.GetBlockCache(); bc != nil {
-				bc.ZapNamespace(num)
-			}
+	t.cache.Delete(0, num, func() {
+		if err := f.file.Remove(); err != nil {
+			t.s.logf("table@remove removing @%d %q", num, err)
+		} else {
+			t.s.logf("table@remove removed @%d", num)
+		}
+		if t.bcache != nil {
+			t.bcache.EvictNS(num)
 		}
 	})
 }
@@ -429,18 +430,39 @@ func (t *tOps) remove(f *tFile) {
 // Closes the table ops instance. It will close all tables,
 // regadless still used or not.
 func (t *tOps) close() {
-	t.cache.Zap()
 	t.bpool.Close()
+	t.cache.Close()
+	if t.bcache != nil {
+		t.bcache.Close()
+	}
 }
 
 // Creates new initialized table ops instance.
-func newTableOps(s *session, cacheCap int) *tOps {
-	c := cache.NewLRUCache(cacheCap)
+func newTableOps(s *session) *tOps {
+	var (
+		cacher cache.Cacher
+		bcache *cache.Cache
+		bpool  *util.BufferPool
+	)
+	if s.o.GetOpenFilesCacheCapacity() > 0 {
+		cacher = cache.NewLRU(s.o.GetOpenFilesCacheCapacity())
+	}
+	if !s.o.GetDisableBlockCache() {
+		var bcacher cache.Cacher
+		if s.o.GetBlockCacheCapacity() > 0 {
+			bcacher = cache.NewLRU(s.o.GetBlockCacheCapacity())
+		}
+		bcache = cache.NewCache(bcacher)
+	}
+	if !s.o.GetDisableBufferPool() {
+		bpool = util.NewBufferPool(s.o.GetBlockSize() + 5)
+	}
 	return &tOps{
-		s:       s,
-		cache:   c,
-		cacheNS: c.GetNamespace(0),
-		bpool:   util.NewBufferPool(s.o.GetBlockSize() + 5),
+		s:      s,
+		noSync: s.o.GetNoSync(),
+		cache:  cache.NewCache(cacher),
+		bcache: bcache,
+		bpool:  bpool,
 	}
 }
 
@@ -485,9 +507,11 @@ func (w *tWriter) finish() (f *tFile, err error) {
 	if err != nil {
 		return
 	}
-	err = w.w.Sync()
-	if err != nil {
-		return
+	if !w.t.noSync {
+		err = w.w.Sync()
+		if err != nil {
+			return
+		}
 	}
 	f = newTableFile(w.file, uint64(w.tw.BytesLen()), iKey(w.first), iKey(w.last))
 	return

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go

@@ -14,7 +14,7 @@ import (
 	"strings"
 	"sync"
 
-	"github.com/syndtr/gosnappy/snappy"
+	"github.com/golang/snappy"
 
 	"github.com/syndtr/goleveldb/leveldb/cache"
 	"github.com/syndtr/goleveldb/leveldb/comparer"
@@ -509,7 +509,7 @@ type Reader struct {
 	mu     sync.RWMutex
 	fi     *storage.FileInfo
 	reader io.ReaderAt
-	cache  cache.Namespace
+	cache  *cache.CacheGetter
 	err    error
 	bpool  *util.BufferPool
 	// Options
@@ -613,18 +613,22 @@ func (r *Reader) readBlock(bh blockHandle, verifyChecksum bool) (*block, error)
 
 func (r *Reader) readBlockCached(bh blockHandle, verifyChecksum, fillCache bool) (*block, util.Releaser, error) {
 	if r.cache != nil {
-		var err error
-		ch := r.cache.Get(bh.offset, func() (charge int, value interface{}) {
-			if !fillCache {
-				return 0, nil
-			}
-			var b *block
-			b, err = r.readBlock(bh, verifyChecksum)
-			if err != nil {
-				return 0, nil
-			}
-			return cap(b.data), b
-		})
+		var (
+			err error
+			ch  *cache.Handle
+		)
+		if fillCache {
+			ch = r.cache.Get(bh.offset, func() (size int, value cache.Value) {
+				var b *block
+				b, err = r.readBlock(bh, verifyChecksum)
+				if err != nil {
+					return 0, nil
+				}
+				return cap(b.data), b
+			})
+		} else {
+			ch = r.cache.Get(bh.offset, nil)
+		}
 		if ch != nil {
 			b, ok := ch.Value().(*block)
 			if !ok {
@@ -667,18 +671,22 @@ func (r *Reader) readFilterBlock(bh blockHandle) (*filterBlock, error) {
 
 func (r *Reader) readFilterBlockCached(bh blockHandle, fillCache bool) (*filterBlock, util.Releaser, error) {
 	if r.cache != nil {
-		var err error
-		ch := r.cache.Get(bh.offset, func() (charge int, value interface{}) {
-			if !fillCache {
-				return 0, nil
-			}
-			var b *filterBlock
-			b, err = r.readFilterBlock(bh)
-			if err != nil {
-				return 0, nil
-			}
-			return cap(b.data), b
-		})
+		var (
+			err error
+			ch  *cache.Handle
+		)
+		if fillCache {
+			ch = r.cache.Get(bh.offset, func() (size int, value cache.Value) {
+				var b *filterBlock
+				b, err = r.readFilterBlock(bh)
+				if err != nil {
+					return 0, nil
+				}
+				return cap(b.data), b
+			})
+		} else {
+			ch = r.cache.Get(bh.offset, nil)
+		}
 		if ch != nil {
 			b, ok := ch.Value().(*filterBlock)
 			if !ok {
@@ -980,7 +988,7 @@ func (r *Reader) Release() {
 // The fi, cache and bpool is optional and can be nil.
 //
 // The returned table reader instance is goroutine-safe.
-func NewReader(f io.ReaderAt, size int64, fi *storage.FileInfo, cache cache.Namespace, bpool *util.BufferPool, o *opt.Options) (*Reader, error) {
+func NewReader(f io.ReaderAt, size int64, fi *storage.FileInfo, cache *cache.CacheGetter, bpool *util.BufferPool, o *opt.Options) (*Reader, error) {
 	if f == nil {
 		return nil, errors.New("leveldb/table: nil file")
 	}

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/writer.go

@@ -12,7 +12,7 @@ import (
 	"fmt"
 	"io"
 
-	"github.com/syndtr/gosnappy/snappy"
+	"github.com/golang/snappy"
 
 	"github.com/syndtr/goleveldb/leveldb/comparer"
 	"github.com/syndtr/goleveldb/leveldb/filter"
@@ -167,11 +167,7 @@ func (w *Writer) writeBlock(buf *util.Buffer, compression opt.Compression) (bh b
 		if n := snappy.MaxEncodedLen(buf.Len()) + blockTrailerLen; len(w.compressionScratch) < n {
 			w.compressionScratch = make([]byte, n)
 		}
-		var compressed []byte
-		compressed, err = snappy.Encode(w.compressionScratch, buf.Bytes())
-		if err != nil {
-			return
-		}
+		compressed := snappy.Encode(w.compressionScratch, buf.Bytes())
 		n := len(compressed)
 		b = compressed[:n+blockTrailerLen]
 		b[n] = blockTypeSnappyCompression

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go

@@ -201,6 +201,7 @@ func (p *BufferPool) String() string {
 
 func (p *BufferPool) drain() {
 	ticker := time.NewTicker(2 * time.Second)
+	defer ticker.Stop()
 	for {
 		select {
 		case <-ticker.C:

Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/version.go

@@ -136,9 +136,8 @@ func (v *version) get(ikey iKey, ro *opt.ReadOptions, noValue bool) (value []byt
 		if !tseek {
 			if tset == nil {
 				tset = &tSet{level, t}
-			} else if tset.table.consumeSeek() <= 0 {
+			} else {
 				tseek = true
-				tcomp = atomic.CompareAndSwapPointer(&v.cSeek, nil, unsafe.Pointer(tset))
 			}
 		}
 
@@ -203,6 +202,28 @@ func (v *version) get(ikey iKey, ro *opt.ReadOptions, noValue bool) (value []byt
 		return true
 	})
 
+	if tseek && tset.table.consumeSeek() <= 0 {
+		tcomp = atomic.CompareAndSwapPointer(&v.cSeek, nil, unsafe.Pointer(tset))
+	}
+
+	return
+}
+
+func (v *version) sampleSeek(ikey iKey) (tcomp bool) {
+	var tset *tSet
+
+	v.walkOverlapping(ikey, func(level int, t *tFile) bool {
+		if tset == nil {
+			tset = &tSet{level, t}
+			return true
+		} else {
+			if tset.table.consumeSeek() <= 0 {
+				tcomp = atomic.CompareAndSwapPointer(&v.cSeek, nil, unsafe.Pointer(tset))
+			}
+			return false
+		}
+	}, nil)
+
 	return
 }
 
@@ -279,7 +300,7 @@ func (v *version) offsetOf(ikey iKey) (n uint64, err error) {
 	return
 }
 
-func (v *version) pickLevel(umin, umax []byte) (level int) {
+func (v *version) pickMemdbLevel(umin, umax []byte) (level int) {
 	if !v.tables[0].overlaps(v.s.icmp, umin, umax, true) {
 		var overlaps tFiles
 		maxLevel := v.s.o.GetMaxMemCompationLevel()

Godeps/_workspace/src/github.com/syndtr/gosnappy/snappy/decode.go

@@ -1,124 +0,0 @@
-// Copyright 2011 The Snappy-Go Authors. All rights reserved.
-// Use of this source code is governed by a BSD-style
-// license that can be found in the LICENSE file.
-
-package snappy
-
-import (
-	"encoding/binary"
-	"errors"
-)
-
-// ErrCorrupt reports that the input is invalid.
-var ErrCorrupt = errors.New("snappy: corrupt input")
-
-// DecodedLen returns the length of the decoded block.
-func DecodedLen(src []byte) (int, error) {
-	v, _, err := decodedLen(src)
-	return v, err
-}
-
-// decodedLen returns the length of the decoded block and the number of bytes
-// that the length header occupied.
-func decodedLen(src []byte) (blockLen, headerLen int, err error) {
-	v, n := binary.Uvarint(src)
-	if n == 0 {
-		return 0, 0, ErrCorrupt
-	}
-	if uint64(int(v)) != v {
-		return 0, 0, errors.New("snappy: decoded block is too large")
-	}
-	return int(v), n, nil
-}
-
-// Decode returns the decoded form of src. The returned slice may be a sub-
-// slice of dst if dst was large enough to hold the entire decoded block.
-// Otherwise, a newly allocated slice will be returned.
-// It is valid to pass a nil dst.
-func Decode(dst, src []byte) ([]byte, error) {
-	dLen, s, err := decodedLen(src)
-	if err != nil {
-		return nil, err
-	}
-	if len(dst) < dLen {
-		dst = make([]byte, dLen)
-	}
-
-	var d, offset, length int
-	for s < len(src) {
-		switch src[s] & 0x03 {
-		case tagLiteral:
-			x := uint(src[s] >> 2)
-			switch {
-			case x < 60:
-				s += 1
-			case x == 60:
-				s += 2
-				if s > len(src) {
-					return nil, ErrCorrupt
-				}
-				x = uint(src[s-1])
-			case x == 61:
-				s += 3
-				if s > len(src) {
-					return nil, ErrCorrupt
-				}
-				x = uint(src[s-2]) | uint(src[s-1])<<8
-			case x == 62:
-				s += 4
-				if s > len(src) {
-					return nil, ErrCorrupt
-				}
-				x = uint(src[s-3]) | uint(src[s-2])<<8 | uint(src[s-1])<<16
-			case x == 63:
-				s += 5
-				if s > len(src) {
-					return nil, ErrCorrupt
-				}
-				x = uint(src[s-4]) | uint(src[s-3])<<8 | uint(src[s-2])<<16 | uint(src[s-1])<<24
-			}
-			length = int(x + 1)
-			if length <= 0 {
-				return nil, errors.New("snappy: unsupported literal length")
-			}
-			if length > len(dst)-d || length > len(src)-s {
-				return nil, ErrCorrupt
-			}
-			copy(dst[d:], src[s:s+length])
-			d += length
-			s += length
-			continue
-
-		case tagCopy1:
-			s += 2
-			if s > len(src) {
-				return nil, ErrCorrupt
-			}
-			length = 4 + int(src[s-2])>>2&0x7
-			offset = int(src[s-2])&0xe0<<3 | int(src[s-1])
-
-		case tagCopy2:
-			s += 3
-			if s > len(src) {
-				return nil, ErrCorrupt
-			}
-			length = 1 + int(src[s-3])>>2
-			offset = int(src[s-2]) | int(src[s-1])<<8
-
-		case tagCopy4:
-			return nil, errors.New("snappy: unsupported COPY_4 tag")
-		}
-
-		end := d + length
-		if offset > d || end > len(dst) {
-			return nil, ErrCorrupt
-		}
-		for ; d < end; d++ {
-			dst[d] = dst[d-offset]
-		}
-	}
-	if d != dLen {
-		return nil, ErrCorrupt
-	}
-	return dst[:d], nil
-}

Godeps/_workspace/src/github.com/thejerf/suture/.travis.yml

@@ -0,0 +1,7 @@
+language: go
+go:
+  - 1.1
+  - 1.2
+  - 1.3
+  - 1.4
+  - tip

Godeps/_workspace/src/github.com/thejerf/suture/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2014-2015 Barracuda Networks, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

Godeps/_workspace/src/github.com/thejerf/suture/README.md

@@ -0,0 +1,50 @@
+Suture
+======
+
+[![Build Status](https://travis-ci.org/thejerf/suture.png?branch=master)](https://travis-ci.org/thejerf/suture)
+
+Suture provides Erlang-ish supervisor trees for Go. "Supervisor trees" ->
+"sutree" -> "suture" -> holds your code together when it's trying to die.
+
+This library has hit maturity, and isn't expected to be changed
+radically. This can also be imported via gopkg.in/thejerf/suture.v1 .
+
+It is intended to deal gracefully with the real failure cases that can
+occur with supervision trees (such as burning all your CPU time endlessly
+restarting dead services), while also making no unnecessary demands on the
+"service" code, and providing hooks to perform adequate logging with in a
+production environment.
+
+[A blog post describing the design decisions](http://www.jerf.org/iri/post/2930)
+is available.
+
+This module is fully covered with [godoc](http://godoc.org/github.com/thejerf/suture),
+including an example, usage, and everything else you might expect from a
+README.md on GitHub. (DRY.)
+
+Code Signing
+------------
+
+Starting with the commit after ac7cf8591b, I will be signing this repository
+with the ["jerf" keybase account](https://keybase.io/jerf).
+
+Aspiration
+----------
+
+One of the big wins the Erlang community has with their pervasive OTP
+support is that it makes it easy for them to distribute libraries that
+easily fit into the OTP paradigm. It ought to someday be considered a good
+idea to distribute libraries that provide some sort of supervisor tree
+functionality out of the box. It is possible to provide this functionality
+without explicitly depending on the Suture library.
+
+Changelog
+---------
+
+suture uses semantic versioning.
+
+1. 1.0.0
+  * Initial release.
+2. 1.0.1
+  * Fixed data race on the .state variable.
+

Godeps/_workspace/src/github.com/thejerf/suture/pre-commit

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+GOLINTOUT=$(golint *go)
+
+if [ ! -z "$GOLINTOUT" -o "$?" != 0 ]; then
+    echo golint failed:
+    echo $GOLINTOUT
+    exit 1
+fi
+
+go test
+

Godeps/_workspace/src/github.com/thejerf/suture/suture.go

@@ -0,0 +1,690 @@
+/*
+
+Package suture provides Erlang-like supervisor trees.
+
+This implements Erlang-esque supervisor trees, as adapted for Go. This is
+intended to be an industrial-strength implementation, but it has not yet
+been deployed in a hostile environment. (It's headed there, though.)
+
+Supervisor Tree -> SuTree -> suture -> holds your code together when it's
+trying to fall apart.
+
+Why use Suture?
+
+ * You want to write bullet-resistant services that will remain available
+   despite unforeseen failure.
+ * You need the code to be smart enough not to consume 100% of the CPU
+   restarting things.
+ * You want to easily compose multiple such services in one program.
+ * You want the Erlang programmers to stop lording their supervision
+   trees over you.
+
+Suture has 100% test coverage, and is golint clean. This doesn't prove it
+free of bugs, but it shows I care.
+
+A blog post describing the design decisions is available at
+http://www.jerf.org/iri/post/2930 .
+
+Using Suture
+
+To idiomatically use Suture, create a Supervisor which is your top level
+"application" supervisor. This will often occur in your program's "main"
+function.
+
+Create "Service"s, which implement the Service interface. .Add() them
+to your Supervisor. Supervisors are also services, so you can create a
+tree structure here, depending on the exact combination of restarts
+you want to create.
+
+As a special case, when adding Supervisors to Supervisors, the "sub"
+supervisor will have the "super" supervisor's Log function copied.
+This allows you to set one log function on the "top" supervisor, and
+have it propagate down to all the sub-supervisors. This also allows
+libraries or modules to provide Supervisors without having to commit
+their users to a particular logging method.
+
+Finally, as what is probably the last line of your main() function, call
+.Serve() on your top level supervisor. This will start all the services
+you've defined.
+
+See the Example for an example, using a simple service that serves out
+incrementing integers.
+
+*/
+package suture
+
+import (
+	"errors"
+	"fmt"
+	"log"
+	"math"
+	"runtime"
+	"sync"
+	"sync/atomic"
+	"time"
+)
+
+const (
+	notRunning = iota
+	normal
+	paused
+)
+
+type supervisorID uint32
+type serviceID uint32
+
+var currentSupervisorID uint32
+
+// ErrWrongSupervisor is returned by the (*Supervisor).Remove method
+// if you pass a ServiceToken from the wrong Supervisor.
+var ErrWrongSupervisor = errors.New("wrong supervisor for this service token, no service removed")
+
+// ServiceToken is an opaque identifier that can be used to terminate a service that
+// has been Add()ed to a Supervisor.
+type ServiceToken struct {
+	id uint64
+}
+
+/*
+Supervisor is the core type of the module that represents a Supervisor.
+
+Supervisors should be constructed either by New or NewSimple.
+
+Once constructed, a Supervisor should be started in one of three ways:
+
+ 1. Calling .Serve().
+ 2. Calling .ServeBackground().
+ 3. Adding it to an existing Supervisor.
+
+Calling Serve will cause the supervisor to run until it is shut down by
+an external user calling Stop() on it. If that never happens, it simply
+runs forever. I suggest creating your services in Supervisors, then making
+a Serve() call on your top-level Supervisor be the last line of your main
+func.
+
+Calling ServeBackground will CORRECTLY start the supervisor running in a
+new goroutine. You do not want to just:
+
+  go supervisor.Serve()
+
+because that will briefly create a race condition as it starts up, if you
+try to .Add() services immediately afterward.
+
+*/
+type Supervisor struct {
+	Name string
+	id   supervisorID
+
+	failureDecay     float64
+	failureThreshold float64
+	failureBackoff   time.Duration
+	timeout          time.Duration
+	log              func(string)
+	services         map[serviceID]Service
+	lastFail         time.Time
+	failures         float64
+	restartQueue     []serviceID
+	serviceCounter   serviceID
+	control          chan supervisorMessage
+	resumeTimer      <-chan time.Time
+
+	// The testing uses the ability to grab these individual logging functions
+	// and get inside of suture's handling at a deep level.
+	// If you ever come up with some need to get into these, submit a pull
+	// request to make them public and some smidge of justification, and
+	// I'll happily do it.
+	// But since I've now changed the signature on these once, I'm glad I
+	// didn't start with them public... :)
+	logBadStop func(*Supervisor, Service)
+	logFailure func(supervisor *Supervisor, service Service, currentFailures float64, failureThreshold float64, restarting bool, error interface{}, stacktrace []byte)
+	logBackoff func(*Supervisor, bool)
+
+	// avoid a dependency on github.com/thejerf/abtime by just implementing
+	// a minimal chunk.
+	getNow    func() time.Time
+	getResume func(time.Duration) <-chan time.Time
+
+	sync.Mutex
+	state uint8
+}
+
+// Spec is used to pass arguments to the New function to create a
+// supervisor. See the New function for full documentation.
+type Spec struct {
+	Log              func(string)
+	FailureDecay     float64
+	FailureThreshold float64
+	FailureBackoff   time.Duration
+	Timeout          time.Duration
+}
+
+/*
+
+New is the full constructor function for a supervisor.
+
+The name is a friendly human name for the supervisor, used in logging. Suture
+does not care if this is unique, but it is good for your sanity if it is.
+
+If not set, the following values are used:
+
+ * Log:               A function is created that uses log.Print.
+ * FailureDecay:      30 seconds
+ * FailureThreshold:  5 failures
+ * FailureBackoff:    15 seconds
+ * Timeout:           10 seconds
+
+The Log function will be called when errors occur. Suture will log the
+following:
+
+ * When a service has failed, with a descriptive message about the
+   current backoff status, and whether it was immediately restarted
+ * When the supervisor has gone into its backoff mode, and when it
+   exits it
+ * When a service fails to stop
+
+The failureRate, failureThreshold, and failureBackoff controls how failures
+are handled, in order to avoid the supervisor failure case where the
+program does nothing but restarting failed services. If you do not
+care how failures behave, the default values should be fine for the
+vast majority of services, but if you want the details:
+
+The supervisor tracks the number of failures that have occurred, with an
+exponential decay on the count. Every FailureDecay seconds, the number of
+failures that have occurred is cut in half. (This is done smoothly with an
+exponential function.) When a failure occurs, the number of failures
+is incremented by one. When the number of failures passes the
+FailureThreshold, the entire service waits for FailureBackoff seconds
+before attempting any further restarts, at which point it resets its
+failure count to zero.
+
+Timeout is how long Suture will wait for a service to properly terminate.
+
+*/
+func New(name string, spec Spec) (s *Supervisor) {
+	s = new(Supervisor)
+
+	s.Name = name
+	s.id = supervisorID(atomic.AddUint32(&currentSupervisorID, 1))
+
+	if spec.Log == nil {
+		s.log = func(msg string) {
+			log.Print(fmt.Sprintf("Supervisor %s: %s", s.Name, msg))
+		}
+	} else {
+		s.log = spec.Log
+	}
+
+	if spec.FailureDecay == 0 {
+		s.failureDecay = 30
+	} else {
+		s.failureDecay = spec.FailureDecay
+	}
+	if spec.FailureThreshold == 0 {
+		s.failureThreshold = 5
+	} else {
+		s.failureThreshold = spec.FailureThreshold
+	}
+	if spec.FailureBackoff == 0 {
+		s.failureBackoff = time.Second * 15
+	} else {
+		s.failureBackoff = spec.FailureBackoff
+	}
+	if spec.Timeout == 0 {
+		s.timeout = time.Second * 10
+	} else {
+		s.timeout = spec.Timeout
+	}
+
+	// overriding these allows for testing the threshold behavior
+	s.getNow = time.Now
+	s.getResume = time.After
+
+	s.control = make(chan supervisorMessage)
+	s.services = make(map[serviceID]Service)
+	s.restartQueue = make([]serviceID, 0, 1)
+	s.resumeTimer = make(chan time.Time)
+
+	// set up the default logging handlers
+	s.logBadStop = func(supervisor *Supervisor, service Service) {
+		s.log(fmt.Sprintf("%s: Service %s failed to terminate in a timely manner", serviceName(supervisor), serviceName(service)))
+	}
+	s.logFailure = func(supervisor *Supervisor, service Service, failures float64, threshold float64, restarting bool, err interface{}, st []byte) {
+		var errString string
+
+		e, canError := err.(error)
+		if canError {
+			errString = e.Error()
+		} else {
+			errString = fmt.Sprintf("%#v", err)
+		}
+
+		s.log(fmt.Sprintf("%s: Failed service '%s' (%f failures of %f), restarting: %#v, error: %s, stacktrace: %s", serviceName(supervisor), serviceName(service), failures, threshold, restarting, errString, string(st)))
+	}
+	s.logBackoff = func(s *Supervisor, entering bool) {
+		if entering {
+			s.log("Entering the backoff state.")
+		} else {
+			s.log("Exiting backoff state.")
+		}
+	}
+
+	return
+}
+
+func serviceName(service Service) (serviceName string) {
+	stringer, canStringer := service.(fmt.Stringer)
+	if canStringer {
+		serviceName = stringer.String()
+	} else {
+		serviceName = fmt.Sprintf("%#v", service)
+	}
+	return
+}
+
+// NewSimple is a convenience function to create a service with just a name
+// and the sensible defaults.
+func NewSimple(name string) *Supervisor {
+	return New(name, Spec{})
+}
+
+/*
+Service is the interface that describes a service to a Supervisor.
+
+Serve Method
+
+The Serve method is called by a Supervisor to start the service.
+The service should execute within the goroutine that this is
+called in. If this function either returns or panics, the Supervisor
+will call it again.
+
+A Serve method SHOULD do as much cleanup of the state as possible,
+to prevent any corruption in the previous state from crashing the
+service again.
+
+Stop Method
+
+This method is used by the supervisor to stop the service. Calling this
+directly on a Service given to a Supervisor will simply result in the
+Service being restarted; use the Supervisor's .Remove(ServiceToken) method
+to stop a service. A supervisor will call .Stop() only once. Thus, it may
+be as destructive as it likes to get the service to stop.
+
+Once Stop has been called on a Service, the Service SHOULD NOT be
+reused in any other supervisor! Because of the impossibility of
+guaranteeing that the service has actually stopped in Go, you can't
+prove that you won't be starting two goroutines using the exact
+same memory to store state, causing completely unpredictable behavior.
+
+Stop should not return until the service has actually stopped.
+"Stopped" here is defined as "the service will stop servicing any
+further requests in the future". For instance, a common implementation
+is to receive a message on a dedicated "stop" channel and immediately
+returning. Once the stop command has been processed, the service is
+stopped.
+
+Another common Stop implementation is to forcibly close an open socket
+or other resource, which will cause detectable errors to manifest in the
+service code. Bear in mind that to perfectly correctly use this
+approach requires a bit more work to handle the chance of a Stop
+command coming in before the resource has been created.
+
+If a service does not Stop within the supervisor's timeout duration, a log
+entry will be made with a descriptive string to that effect. This does
+not guarantee that the service is hung; it may still get around to being
+properly stopped in the future. Until the service is fully stopped,
+both the service and the spawned goroutine trying to stop it will be
+"leaked".
+
+Stringer Interface
+
+It is not mandatory to implement the fmt.Stringer interface on your
+service, but if your Service does happen to implement that, the log
+messages that describe your service will use that when naming the
+service. Otherwise, you'll see the GoString of your service object,
+obtained via fmt.Sprintf("%#v", service).
+
+*/
+type Service interface {
+	Serve()
+	Stop()
+}
+
+/*
+Add adds a service to this supervisor.
+
+If the supervisor is currently running, the service will be started
+immediately. If the supervisor is not currently running, the service
+will be started when the supervisor is.
+
+The returned ServiceID may be passed to the Remove method of the Supervisor
+to terminate the service.
+
+As a special behavior, if the service added is itself a supervisor, the
+supervisor being added will copy the Log function from the Supervisor it
+is being added to. This allows factoring out providing a Supervisor
+from its logging.
+
+*/
+func (s *Supervisor) Add(service Service) ServiceToken {
+	if s == nil {
+		panic("can't add service to nil *suture.Supervisor")
+	}
+
+	if supervisor, isSupervisor := service.(*Supervisor); isSupervisor {
+		supervisor.logBadStop = s.logBadStop
+		supervisor.logFailure = s.logFailure
+		supervisor.logBackoff = s.logBackoff
+	}
+
+	s.Lock()
+	if s.state == notRunning {
+		id := s.serviceCounter
+		s.serviceCounter++
+
+		s.services[id] = service
+		s.restartQueue = append(s.restartQueue, id)
+
+		s.Unlock()
+		return ServiceToken{uint64(s.id)<<32 | uint64(id)}
+	}
+	s.Unlock()
+
+	response := make(chan serviceID)
+	s.control <- addService{service, response}
+	return ServiceToken{uint64(s.id)<<32 | uint64(<-response)}
+}
+
+// ServeBackground starts running a supervisor in its own goroutine. This
+// method does not return until it is safe to use .Add() on the Supervisor.
+func (s *Supervisor) ServeBackground() {
+	go s.Serve()
+	s.sync()
+}
+
+/*
+Serve starts the supervisor. You should call this on the top-level supervisor,
+but nothing else.
+*/
+func (s *Supervisor) Serve() {
+	if s == nil {
+		panic("Can't serve with a nil *suture.Supervisor")
+	}
+	if s.id == 0 {
+		panic("Can't call Serve on an incorrectly-constructed *suture.Supervisor")
+	}
+
+	defer func() {
+		s.Lock()
+		s.state = notRunning
+		s.Unlock()
+	}()
+
+	s.Lock()
+	if s.state != notRunning {
+		s.Unlock()
+		panic("Running a supervisor while it is already running?")
+	}
+
+	s.state = normal
+	s.Unlock()
+
+	// for all the services I currently know about, start them
+	for _, id := range s.restartQueue {
+		service, present := s.services[id]
+		if present {
+			s.runService(service, id)
+		}
+	}
+	s.restartQueue = make([]serviceID, 0, 1)
+
+	for {
+		select {
+		case m := <-s.control:
+			switch msg := m.(type) {
+			case serviceFailed:
+				s.handleFailedService(msg.id, msg.err, msg.stacktrace)
+			case serviceEnded:
+				service, monitored := s.services[msg.id]
+				if monitored {
+					s.handleFailedService(msg.id, fmt.Sprintf("%s returned unexpectedly", service), []byte("[unknown stack trace]"))
+				}
+			case addService:
+				id := s.serviceCounter
+				s.serviceCounter++
+
+				s.services[id] = msg.service
+				s.runService(msg.service, id)
+
+				msg.response <- id
+			case removeService:
+				s.removeService(msg.id)
+			case stopSupervisor:
+				for id := range s.services {
+					s.removeService(id)
+				}
+				return
+			case listServices:
+				services := []Service{}
+				for _, service := range s.services {
+					services = append(services, service)
+				}
+				msg.c <- services
+			case syncSupervisor:
+				// this does nothing on purpose; its sole purpose is to
+				// introduce a sync point via the channel receive
+			case panicSupervisor:
+				// used only by tests
+				panic("Panicking as requested!")
+			}
+		case _ = <-s.resumeTimer:
+			// We're resuming normal operation after a pause due to
+			// excessive thrashing
+			// FIXME: Ought to permit some spacing of these functions, rather
+			// than simply hammering through them
+			s.Lock()
+			s.state = normal
+			s.Unlock()
+			s.failures = 0
+			s.logBackoff(s, false)
+			for _, id := range s.restartQueue {
+				service, present := s.services[id]
+				if present {
+					s.runService(service, id)
+				}
+			}
+			s.restartQueue = make([]serviceID, 0, 1)
+		}
+	}
+}
+
+func (s *Supervisor) handleFailedService(id serviceID, err interface{}, stacktrace []byte) {
+	now := s.getNow()
+
+	if s.lastFail.IsZero() {
+		s.lastFail = now
+		s.failures = 1.0
+	} else {
+		sinceLastFail := now.Sub(s.lastFail).Seconds()
+		intervals := sinceLastFail / s.failureDecay
+		s.failures = s.failures*math.Pow(.5, intervals) + 1
+	}
+
+	if s.failures > s.failureThreshold {
+		s.Lock()
+		s.state = paused
+		s.Unlock()
+		s.logBackoff(s, true)
+		s.resumeTimer = s.getResume(s.failureBackoff)
+	}
+
+	s.lastFail = now
+
+	failedService, monitored := s.services[id]
+
+	// It is possible for a service to be no longer monitored
+	// by the time we get here. In that case, just ignore it.
+	if monitored {
+		// this may look dangerous because the state could change, but this
+		// code is only ever run in the one goroutine that is permitted to
+		// change the state, so nothing else will.
+		s.Lock()
+		curState := s.state
+		s.Unlock()
+		if curState == normal {
+			s.runService(failedService, id)
+			s.logFailure(s, failedService, s.failures, s.failureThreshold, true, err, stacktrace)
+		} else {
+			// FIXME: When restarting, check that the service still
+			// exists (it may have been stopped in the meantime)
+			s.restartQueue = append(s.restartQueue, id)
+			s.logFailure(s, failedService, s.failures, s.failureThreshold, false, err, stacktrace)
+		}
+	}
+}
+
+func (s *Supervisor) runService(service Service, id serviceID) {
+	go func() {
+		defer func() {
+			if r := recover(); r != nil {
+				buf := make([]byte, 65535, 65535)
+				written := runtime.Stack(buf, false)
+				buf = buf[:written]
+				s.fail(id, r, buf)
+			}
+		}()
+
+		service.Serve()
+
+		s.serviceEnded(id)
+	}()
+}
+
+func (s *Supervisor) removeService(id serviceID) {
+	service, present := s.services[id]
+	if present {
+		delete(s.services, id)
+		go func() {
+			successChan := make(chan bool)
+			go func() {
+				service.Stop()
+				successChan <- true
+			}()
+
+			failChan := s.getResume(s.timeout)
+
+			select {
+			case <-successChan:
+				// Life is good!
+			case <-failChan:
+				s.logBadStop(s, service)
+			}
+		}()
+	}
+}
+
+// String implements the fmt.Stringer interface.
+func (s *Supervisor) String() string {
+	return s.Name
+}
+
+// sum type pattern for type-safe message passing; see
+// http://www.jerf.org/iri/post/2917
+
+type supervisorMessage interface {
+	isSupervisorMessage()
+}
+
+/*
+Remove will remove the given service from the Supervisor, and attempt to Stop() it.
+The ServiceID token comes from the Add() call.
+*/
+func (s *Supervisor) Remove(id ServiceToken) error {
+	sID := supervisorID(id.id >> 32)
+	if sID != s.id {
+		return ErrWrongSupervisor
+	}
+	s.control <- removeService{serviceID(id.id & 0xffffffff)}
+	return nil
+}
+
+/*
+
+Services returns a []Service containing a snapshot of the services this
+Supervisor is managing.
+
+*/
+func (s *Supervisor) Services() []Service {
+	ls := listServices{make(chan []Service)}
+	s.control <- ls
+	return <-ls.c
+}
+
+type listServices struct {
+	c chan []Service
+}
+
+func (ls listServices) isSupervisorMessage() {}
+
+type removeService struct {
+	id serviceID
+}
+
+func (rs removeService) isSupervisorMessage() {}
+
+func (s *Supervisor) sync() {
+	s.control <- syncSupervisor{}
+}
+
+type syncSupervisor struct {
+}
+
+func (ss syncSupervisor) isSupervisorMessage() {}
+
+func (s *Supervisor) fail(id serviceID, err interface{}, stacktrace []byte) {
+	s.control <- serviceFailed{id, err, stacktrace}
+}
+
+type serviceFailed struct {
+	id         serviceID
+	err        interface{}
+	stacktrace []byte
+}
+
+func (sf serviceFailed) isSupervisorMessage() {}
+
+func (s *Supervisor) serviceEnded(id serviceID) {
+	s.control <- serviceEnded{id}
+}
+
+type serviceEnded struct {
+	id serviceID
+}
+
+func (s serviceEnded) isSupervisorMessage() {}
+
+// added by the Add() method
+type addService struct {
+	service  Service
+	response chan serviceID
+}
+
+func (as addService) isSupervisorMessage() {}
+
+// Stop stops the Supervisor.
+func (s *Supervisor) Stop() {
+	s.control <- stopSupervisor{}
+}
+
+type stopSupervisor struct {
+}
+
+func (ss stopSupervisor) isSupervisorMessage() {}
+
+func (s *Supervisor) panic() {
+	s.control <- panicSupervisor{}
+}
+
+type panicSupervisor struct {
+}
+
+func (ps panicSupervisor) isSupervisorMessage() {}

Godeps/_workspace/src/github.com/thejerf/suture/suture_simple_test.go

@@ -0,0 +1,49 @@
+package suture
+
+import "fmt"
+
+type Incrementor struct {
+	current int
+	next    chan int
+	stop    chan bool
+}
+
+func (i *Incrementor) Stop() {
+	fmt.Println("Stopping the service")
+	i.stop <- true
+}
+
+func (i *Incrementor) Serve() {
+	for {
+		select {
+		case i.next <- i.current:
+			i.current++
+		case <-i.stop:
+			// We sync here just to guarantee the output of "Stopping the service",
+			// so this passes the test reliably.
+			// Most services would simply "return" here.
+			i.stop <- true
+			return
+		}
+	}
+}
+
+func ExampleNew_simple() {
+	supervisor := NewSimple("Supervisor")
+	service := &Incrementor{0, make(chan int), make(chan bool)}
+	supervisor.Add(service)
+
+	go supervisor.ServeBackground()
+
+	fmt.Println("Got:", <-service.next)
+	fmt.Println("Got:", <-service.next)
+	supervisor.Stop()
+
+	// We sync here just to guarantee the output of "Stopping the service"
+	<-service.stop
+
+	// Output:
+	// Got: 0
+	// Got: 1
+	// Stopping the service
+}

Godeps/_workspace/src/github.com/thejerf/suture/suture_test.go

@@ -0,0 +1,616 @@
+package suture
+
+import (
+	"errors"
+	"fmt"
+	"reflect"
+	"sync"
+	"testing"
+	"time"
+)
+
+const (
+	Happy = iota
+	Fail
+	Panic
+	Hang
+	UseStopChan
+)
+
+var everMultistarted = false
+
+// Test that supervisors work perfectly when everything is hunky dory.
+func TestTheHappyCase(t *testing.T) {
+	t.Parallel()
+
+	s := NewSimple("A")
+	if s.String() != "A" {
+		t.Fatal("Can't get name from a supervisor")
+	}
+	service := NewService("B")
+
+	s.Add(service)
+
+	go s.Serve()
+
+	<-service.started
+
+	// If we stop the service, it just gets restarted
+	service.Stop()
+	<-service.started
+
+	// And it is shut down when we stop the supervisor
+	service.take <- UseStopChan
+	s.Stop()
+	<-service.stop
+}
+
+// Test that adding to a running supervisor does indeed start the service.
+func TestAddingToRunningSupervisor(t *testing.T) {
+	t.Parallel()
+
+	s := NewSimple("A1")
+
+	s.ServeBackground()
+	defer s.Stop()
+
+	service := NewService("B1")
+	s.Add(service)
+
+	<-service.started
+
+	services := s.Services()
+	if !reflect.DeepEqual([]Service{service}, services) {
+		t.Fatal("Can't get list of services as expected.")
+	}
+}
+
+// Test what happens when services fail.
+func TestFailures(t *testing.T) {
+	t.Parallel()
+
+	s := NewSimple("A2")
+	s.failureThreshold = 3.5
+
+	go s.Serve()
+	defer func() {
+		// to avoid deadlocks during shutdown, we have to not try to send
+		// things out on channels while we're shutting down (this undoes the
+		// logFailure overide about 25 lines down)
+		s.logFailure = func(*Supervisor, Service, float64, float64, bool, interface{}, []byte) {}
+		s.Stop()
+	}()
+	s.sync()
+
+	service1 := NewService("B2")
+	service2 := NewService("C2")
+
+	s.Add(service1)
+	<-service1.started
+	s.Add(service2)
+	<-service2.started
+
+	nowFeeder := NewNowFeeder()
+	pastVal := time.Unix(1000000, 0)
+	nowFeeder.appendTimes(pastVal)
+	s.getNow = nowFeeder.getter
+
+	resumeChan := make(chan time.Time)
+	s.getResume = func(d time.Duration) <-chan time.Time {
+		return resumeChan
+	}
+
+	failNotify := make(chan bool)
+	// use this to synchronize on here
+	s.logFailure = func(supervisor *Supervisor, s Service, cf float64, ft float64, r bool, error interface{}, stacktrace []byte) {
+		failNotify <- r
+	}
+
+	// All that setup was for this: Service1, please return now.
+	service1.take <- Fail
+	restarted := <-failNotify
+	<-service1.started
+
+	if !restarted || s.failures != 1 || s.lastFail != pastVal {
+		t.Fatal("Did not fail in the expected manner")
+	}
+	// Getting past this means the service was restarted.
+	service1.take <- Happy
+
+	// Service2, your turn.
+	service2.take <- Fail
+	nowFeeder.appendTimes(pastVal)
+	restarted = <-failNotify
+	<-service2.started
+	if !restarted || s.failures != 2 || s.lastFail != pastVal {
+		t.Fatal("Did not fail in the expected manner")
+	}
+	// And you're back. (That is, the correct service was restarted.)
+	service2.take <- Happy
+
+	// Now, one failureDecay later, is everything working correctly?
+	oneDecayLater := time.Unix(1000030, 0)
+	nowFeeder.appendTimes(oneDecayLater)
+	service2.take <- Fail
+	restarted = <-failNotify
+	<-service2.started
+	// playing a bit fast and loose here with floating point, but...
+	// we get 2 by taking the current failure value of 2, decaying it
+	// by one interval, which cuts it in half to 1, then adding 1 again,
+	// all of which "should" be precise
+	if !restarted || s.failures != 2 || s.lastFail != oneDecayLater {
+		t.Fatal("Did not decay properly", s.lastFail, oneDecayLater)
+	}
+
+	// For a change of pace, service1 would you be so kind as to panic?
+	nowFeeder.appendTimes(oneDecayLater)
+	service1.take <- Panic
+	restarted = <-failNotify
+	<-service1.started
+	if !restarted || s.failures != 3 || s.lastFail != oneDecayLater {
+		t.Fatal("Did not correctly recover from a panic")
+	}
+
+	nowFeeder.appendTimes(oneDecayLater)
+	backingoff := make(chan bool)
+	s.logBackoff = func(s *Supervisor, backingOff bool) {
+		backingoff <- backingOff
+	}
+
+	// And with this failure, we trigger the backoff code.
+	service1.take <- Fail
+	backoff := <-backingoff
+	restarted = <-failNotify
+
+	if !backoff || restarted || s.failures != 4 {
+		t.Fatal("Broke past the threshold but did not log correctly", s.failures)
+	}
+	if service1.existing != 0 {
+		t.Fatal("service1 still exists according to itself?")
+	}
+
+	// service2 is still running, because we don't shut anything down in a
+	// backoff, we just stop restarting.
+	service2.take <- Happy
+
+	var correct bool
+	timer := time.NewTimer(time.Millisecond * 10)
+	// verify the service has not been restarted
+	// hard to get around race conditions here without simply using a timer...
+	select {
+	case service1.take <- Happy:
+		correct = false
+	case <-timer.C:
+		correct = true
+	}
+	if !correct {
+		t.Fatal("Restarted the service during the backoff interval")
+	}
+
+	// tell the supervisor the restart interval has passed
+	resumeChan <- time.Time{}
+	backoff = <-backingoff
+	<-service1.started
+	s.sync()
+	if s.failures != 0 {
+		t.Fatal("Did not reset failure count after coming back from timeout.")
+	}
+
+	nowFeeder.appendTimes(oneDecayLater)
+	service1.take <- Fail
+	restarted = <-failNotify
+	<-service1.started
+	if !restarted || backoff {
+		t.Fatal("For some reason, got that we were backing off again.", restarted, backoff)
+	}
+}
+
+func TestRunningAlreadyRunning(t *testing.T) {
+	t.Parallel()
+
+	s := NewSimple("A3")
+	go s.Serve()
+	defer s.Stop()
+
+	// ensure the supervisor has made it to its main loop
+	s.sync()
+	var errored bool
+	func() {
+		defer func() {
+			if r := recover(); r != nil {
+				errored = true
+			}
+		}()
+
+		s.Serve()
+	}()
+	if !errored {
+		t.Fatal("Supervisor failed to prevent itself from double-running.")
+	}
+}
+
+func TestFullConstruction(t *testing.T) {
+	t.Parallel()
+
+	s := New("Moo", Spec{
+		Log:              func(string) {},
+		FailureDecay:     1,
+		FailureThreshold: 2,
+		FailureBackoff:   3,
+		Timeout:          time.Second * 29,
+	})
+	if s.String() != "Moo" || s.failureDecay != 1 || s.failureThreshold != 2 || s.failureBackoff != 3 || s.timeout != time.Second*29 {
+		t.Fatal("Full construction failed somehow")
+	}
+}
+
+// This is mostly for coverage testing.
+func TestDefaultLogging(t *testing.T) {
+	t.Parallel()
+
+	s := NewSimple("A4")
+
+	service := NewService("B4")
+	s.Add(service)
+
+	s.failureThreshold = .5
+	s.failureBackoff = time.Millisecond * 25
+	go s.Serve()
+	s.sync()
+
+	<-service.started
+
+	resumeChan := make(chan time.Time)
+	s.getResume = func(d time.Duration) <-chan time.Time {
+		return resumeChan
+	}
+
+	service.take <- UseStopChan
+	service.take <- Fail
+	<-service.stop
+	resumeChan <- time.Time{}
+
+	<-service.started
+
+	service.take <- Happy
+
+	serviceName(&BarelyService{})
+
+	s.logBadStop(s, service)
+	s.logFailure(s, service, 1, 1, true, errors.New("test error"), []byte{})
+
+	s.Stop()
+}
+
+func TestNestedSupervisors(t *testing.T) {
+	t.Parallel()
+
+	super1 := NewSimple("Top5")
+	super2 := NewSimple("Nested5")
+	service := NewService("Service5")
+
+	super2.logBadStop = func(*Supervisor, Service) {
+		panic("Failed to copy logBadStop")
+	}
+
+	super1.Add(super2)
+	super2.Add(service)
+
+	// test the functions got copied from super1; if this panics, it didn't
+	// get copied
+	super2.logBadStop(super2, service)
+
+	go super1.Serve()
+	super1.sync()
+
+	<-service.started
+	service.take <- Happy
+
+	super1.Stop()
+}
+
+func TestStoppingSupervisorStopsServices(t *testing.T) {
+	t.Parallel()
+
+	s := NewSimple("Top6")
+	service := NewService("Service 6")
+
+	s.Add(service)
+
+	go s.Serve()
+	s.sync()
+
+	<-service.started
+
+	service.take <- UseStopChan
+
+	s.Stop()
+	<-service.stop
+}
+
+func TestStoppingStillWorksWithHungServices(t *testing.T) {
+	t.Parallel()
+
+	s := NewSimple("Top7")
+	service := NewService("Service WillHang7")
+
+	s.Add(service)
+
+	go s.Serve()
+
+	<-service.started
+
+	service.take <- UseStopChan
+	service.take <- Hang
+
+	resumeChan := make(chan time.Time)
+	s.getResume = func(d time.Duration) <-chan time.Time {
+		return resumeChan
+	}
+	failNotify := make(chan struct{})
+	s.logBadStop = func(supervisor *Supervisor, s Service) {
+		failNotify <- struct{}{}
+	}
+
+	s.Stop()
+
+	resumeChan <- time.Time{}
+	<-failNotify
+	service.release <- true
+	<-service.stop
+}
+
+func TestRemoveService(t *testing.T) {
+	t.Parallel()
+
+	s := NewSimple("Top")
+	service := NewService("ServiceToRemove8")
+
+	id := s.Add(service)
+
+	go s.Serve()
+
+	<-service.started
+	service.take <- UseStopChan
+
+	err := s.Remove(id)
+	if err != nil {
+		t.Fatal("Removing service somehow failed")
+	}
+	<-service.stop
+
+	err = s.Remove(ServiceToken{1<<36 + 1})
+	if err != ErrWrongSupervisor {
+		t.Fatal("Did not detect that the ServiceToken was wrong")
+	}
+}
+
+func TestFailureToConstruct(t *testing.T) {
+	t.Parallel()
+
+	var s *Supervisor
+
+	panics(func() {
+		s.Serve()
+	})
+
+	s = new(Supervisor)
+	panics(func() {
+		s.Serve()
+	})
+}
+
+func TestFailingSupervisors(t *testing.T) {
+	t.Parallel()
+
+	// This is a bit of a complicated test, so let me explain what
+	// all this is doing:
+	// 1. Set up a top-level supervisor with a hair-trigger backoff.
+	// 2. Add a supervisor to that.
+	// 3. To that supervisor, add a service.
+	// 4. Panic the supervisor in the middle, sending the top-level into
+	//    backoff.
+	// 5. Kill the lower level service too.
+	// 6. Verify that when the top-level service comes out of backoff,
+	//    the service ends up restarted as expected.
+
+	// Ultimately, we can't have more than a best-effort recovery here.
+	// A panic'ed supervisor can't really be trusted to have consistent state,
+	// and without *that*, we can't trust it to do anything sensible with
+	// the children it may have been running. So unlike Erlang, we can't
+	// can't really expect to be able to safely restart them or anything.
+	// Really, the "correct" answer is that the Supervisor must never panic,
+	// but in the event that it does, this verifies that it at least tries
+	// to get on with life.
+
+	// This also tests that if a Supervisor itself panics, and one of its
+	// monitored services goes down in the meantime, that the monitored
+	// service also gets correctly restarted when the supervisor does.
+
+	s1 := NewSimple("Top9")
+	s2 := NewSimple("Nested9")
+	service := NewService("Service9")
+
+	s1.Add(s2)
+	s2.Add(service)
+
+	go s1.Serve()
+	<-service.started
+
+	s1.failureThreshold = .5
+
+	// let us control precisely when s1 comes back
+	resumeChan := make(chan time.Time)
+	s1.getResume = func(d time.Duration) <-chan time.Time {
+		return resumeChan
+	}
+	failNotify := make(chan string)
+	// use this to synchronize on here
+	s1.logFailure = func(supervisor *Supervisor, s Service, cf float64, ft float64, r bool, error interface{}, stacktrace []byte) {
+		failNotify <- fmt.Sprintf("%s", s)
+	}
+
+	s2.panic()
+
+	failing := <-failNotify
+	// that's enough sync to guarantee this:
+	if failing != "Nested9" || s1.state != paused {
+		t.Fatal("Top-level supervisor did not go into backoff as expected")
+	}
+
+	service.take <- Fail
+
+	resumeChan <- time.Time{}
+	<-service.started
+}
+
+func TestNilSupervisorAdd(t *testing.T) {
+	t.Parallel()
+
+	var s *Supervisor
+
+	defer func() {
+		if r := recover(); r == nil {
+			t.Fatal("did not panic as expected on nil add")
+		}
+	}()
+
+	s.Add(s)
+}
+
+// https://github.com/thejerf/suture/issues/11
+//
+// The purpose of this test is to verify that it does not cause data races,
+// so there are no obvious assertions.
+func TestIssue11(t *testing.T) {
+	t.Parallel()
+
+	s := NewSimple("main")
+	s.ServeBackground()
+
+	subsuper := NewSimple("sub")
+	s.Add(subsuper)
+
+	subsuper.Add(NewService("may cause data race"))
+}
+
+// http://golangtutorials.blogspot.com/2011/10/gotest-unit-testing-and-benchmarking-go.html
+// claims test function are run in the same order as the source file...
+// I'm not sure if this is part of the contract, though. Especially in the
+// face of "t.Parallel()"...
+func TestEverMultistarted(t *testing.T) {
+	if everMultistarted {
+		t.Fatal("Seem to have multistarted a service at some point, bummer.")
+	}
+}
+
+// A test service that can be induced to fail, panic, or hang on demand.
+func NewService(name string) *FailableService {
+	return &FailableService{name, make(chan bool), make(chan int),
+		make(chan bool, 1), make(chan bool), make(chan bool), 0}
+}
+
+type FailableService struct {
+	name     string
+	started  chan bool
+	take     chan int
+	shutdown chan bool
+	release  chan bool
+	stop     chan bool
+	existing int
+}
+
+func (s *FailableService) Serve() {
+	if s.existing != 0 {
+		everMultistarted = true
+		panic("Multi-started the same service! " + s.name)
+	}
+	s.existing++
+
+	s.started <- true
+
+	useStopChan := false
+
+	for {
+		select {
+		case val := <-s.take:
+			switch val {
+			case Happy:
+				// Do nothing on purpose. Life is good!
+			case Fail:
+				s.existing--
+				if useStopChan {
+					s.stop <- true
+				}
+				return
+			case Panic:
+				s.existing--
+				panic("Panic!")
+			case Hang:
+				// or more specifically, "hang until I release you"
+				<-s.release
+			case UseStopChan:
+				useStopChan = true
+			}
+		case <-s.shutdown:
+			s.existing--
+			if useStopChan {
+				s.stop <- true
+			}
+			return
+		}
+	}
+}
+
+func (s *FailableService) String() string {
+	return s.name
+}
+
+func (s *FailableService) Stop() {
+	s.shutdown <- true
+}
+
+type NowFeeder struct {
+	values []time.Time
+	getter func() time.Time
+	m      sync.Mutex
+}
+
+// This is used to test serviceName; it's a service without a Stringer.
+type BarelyService struct{}
+
+func (bs *BarelyService) Serve() {}
+func (bs *BarelyService) Stop()  {}
+
+func NewNowFeeder() (nf *NowFeeder) {
+	nf = new(NowFeeder)
+	nf.getter = func() time.Time {
+		nf.m.Lock()
+		defer nf.m.Unlock()
+		if len(nf.values) > 0 {
+			ret := nf.values[0]
+			nf.values = nf.values[1:]
+			return ret
+		}
+		panic("Ran out of values for NowFeeder")
+	}
+	return
+}
+
+func (nf *NowFeeder) appendTimes(t ...time.Time) {
+	nf.m.Lock()
+	defer nf.m.Unlock()
+	nf.values = append(nf.values, t...)
+}
+
+func panics(doesItPanic func()) (panics bool) {
+	defer func() {
+		if r := recover(); r != nil {
+			panics = true
+		}
+	}()
+
+	doesItPanic()
+
+	return
+}

Godeps/_workspace/src/golang.org/x/net/internal/iana/const.go

@@ -0,0 +1,181 @@
+// go generate gen.go
+// GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
+
+// Package iana provides protocol number resources managed by the Internet Assigned Numbers Authority (IANA).
+package iana // import "golang.org/x/net/internal/iana"
+
+// Differentiated Services Field Codepoints (DSCP), Updated: 2013-06-25
+const (
+	DiffServCS0        = 0x0  // CS0
+	DiffServCS1        = 0x20 // CS1
+	DiffServCS2        = 0x40 // CS2
+	DiffServCS3        = 0x60 // CS3
+	DiffServCS4        = 0x80 // CS4
+	DiffServCS5        = 0xa0 // CS5
+	DiffServCS6        = 0xc0 // CS6
+	DiffServCS7        = 0xe0 // CS7
+	DiffServAF11       = 0x28 // AF11
+	DiffServAF12       = 0x30 // AF12
+	DiffServAF13       = 0x38 // AF13
+	DiffServAF21       = 0x48 // AF21
+	DiffServAF22       = 0x50 // AF22
+	DiffServAF23       = 0x58 // AF23
+	DiffServAF31       = 0x68 // AF31
+	DiffServAF32       = 0x70 // AF32
+	DiffServAF33       = 0x78 // AF33
+	DiffServAF41       = 0x88 // AF41
+	DiffServAF42       = 0x90 // AF42
+	DiffServAF43       = 0x98 // AF43
+	DiffServEFPHB      = 0xb8 // EF PHB
+	DiffServVOICEADMIT = 0xb0 // VOICE-ADMIT
+)
+
+// IPv4 TOS Byte and IPv6 Traffic Class Octet, Updated: 2001-09-06
+const (
+	NotECNTransport       = 0x0 // Not-ECT (Not ECN-Capable Transport)
+	ECNTransport1         = 0x1 // ECT(1) (ECN-Capable Transport(1))
+	ECNTransport0         = 0x2 // ECT(0) (ECN-Capable Transport(0))
+	CongestionExperienced = 0x3 // CE (Congestion Experienced)
+)
+
+// Protocol Numbers, Updated: 2015-06-23
+const (
+	ProtocolIP             = 0   // IPv4 encapsulation, pseudo protocol number
+	ProtocolHOPOPT         = 0   // IPv6 Hop-by-Hop Option
+	ProtocolICMP           = 1   // Internet Control Message
+	ProtocolIGMP           = 2   // Internet Group Management
+	ProtocolGGP            = 3   // Gateway-to-Gateway
+	ProtocolIPv4           = 4   // IPv4 encapsulation
+	ProtocolST             = 5   // Stream
+	ProtocolTCP            = 6   // Transmission Control
+	ProtocolCBT            = 7   // CBT
+	ProtocolEGP            = 8   // Exterior Gateway Protocol
+	ProtocolIGP            = 9   // any private interior gateway (used by Cisco for their IGRP)
+	ProtocolBBNRCCMON      = 10  // BBN RCC Monitoring
+	ProtocolNVPII          = 11  // Network Voice Protocol
+	ProtocolPUP            = 12  // PUP
+	ProtocolARGUS          = 13  // ARGUS
+	ProtocolEMCON          = 14  // EMCON
+	ProtocolXNET           = 15  // Cross Net Debugger
+	ProtocolCHAOS          = 16  // Chaos
+	ProtocolUDP            = 17  // User Datagram
+	ProtocolMUX            = 18  // Multiplexing
+	ProtocolDCNMEAS        = 19  // DCN Measurement Subsystems
+	ProtocolHMP            = 20  // Host Monitoring
+	ProtocolPRM            = 21  // Packet Radio Measurement
+	ProtocolXNSIDP         = 22  // XEROX NS IDP
+	ProtocolTRUNK1         = 23  // Trunk-1
+	ProtocolTRUNK2         = 24  // Trunk-2
+	ProtocolLEAF1          = 25  // Leaf-1
+	ProtocolLEAF2          = 26  // Leaf-2
+	ProtocolRDP            = 27  // Reliable Data Protocol
+	ProtocolIRTP           = 28  // Internet Reliable Transaction
+	ProtocolISOTP4         = 29  // ISO Transport Protocol Class 4
+	ProtocolNETBLT         = 30  // Bulk Data Transfer Protocol
+	ProtocolMFENSP         = 31  // MFE Network Services Protocol
+	ProtocolMERITINP       = 32  // MERIT Internodal Protocol
+	ProtocolDCCP           = 33  // Datagram Congestion Control Protocol
+	Protocol3PC            = 34  // Third Party Connect Protocol
+	ProtocolIDPR           = 35  // Inter-Domain Policy Routing Protocol
+	ProtocolXTP            = 36  // XTP
+	ProtocolDDP            = 37  // Datagram Delivery Protocol
+	ProtocolIDPRCMTP       = 38  // IDPR Control Message Transport Proto
+	ProtocolTPPP           = 39  // TP++ Transport Protocol
+	ProtocolIL             = 40  // IL Transport Protocol
+	ProtocolIPv6           = 41  // IPv6 encapsulation
+	ProtocolSDRP           = 42  // Source Demand Routing Protocol
+	ProtocolIPv6Route      = 43  // Routing Header for IPv6
+	ProtocolIPv6Frag       = 44  // Fragment Header for IPv6
+	ProtocolIDRP           = 45  // Inter-Domain Routing Protocol
+	ProtocolRSVP           = 46  // Reservation Protocol
+	ProtocolGRE            = 47  // Generic Routing Encapsulation
+	ProtocolDSR            = 48  // Dynamic Source Routing Protocol
+	ProtocolBNA            = 49  // BNA
+	ProtocolESP            = 50  // Encap Security Payload
+	ProtocolAH             = 51  // Authentication Header
+	ProtocolINLSP          = 52  // Integrated Net Layer Security  TUBA
+	ProtocolNARP           = 54  // NBMA Address Resolution Protocol
+	ProtocolMOBILE         = 55  // IP Mobility
+	ProtocolTLSP           = 56  // Transport Layer Security Protocol using Kryptonet key management
+	ProtocolSKIP           = 57  // SKIP
+	ProtocolIPv6ICMP       = 58  // ICMP for IPv6
+	ProtocolIPv6NoNxt      = 59  // No Next Header for IPv6
+	ProtocolIPv6Opts       = 60  // Destination Options for IPv6
+	ProtocolCFTP           = 62  // CFTP
+	ProtocolSATEXPAK       = 64  // SATNET and Backroom EXPAK
+	ProtocolKRYPTOLAN      = 65  // Kryptolan
+	ProtocolRVD            = 66  // MIT Remote Virtual Disk Protocol
+	ProtocolIPPC           = 67  // Internet Pluribus Packet Core
+	ProtocolSATMON         = 69  // SATNET Monitoring
+	ProtocolVISA           = 70  // VISA Protocol
+	ProtocolIPCV           = 71  // Internet Packet Core Utility
+	ProtocolCPNX           = 72  // Computer Protocol Network Executive
+	ProtocolCPHB           = 73  // Computer Protocol Heart Beat
+	ProtocolWSN            = 74  // Wang Span Network
+	ProtocolPVP            = 75  // Packet Video Protocol
+	ProtocolBRSATMON       = 76  // Backroom SATNET Monitoring
+	ProtocolSUNND          = 77  // SUN ND PROTOCOL-Temporary
+	ProtocolWBMON          = 78  // WIDEBAND Monitoring
+	ProtocolWBEXPAK        = 79  // WIDEBAND EXPAK
+	ProtocolISOIP          = 80  // ISO Internet Protocol
+	ProtocolVMTP           = 81  // VMTP
+	ProtocolSECUREVMTP     = 82  // SECURE-VMTP
+	ProtocolVINES          = 83  // VINES
+	ProtocolTTP            = 84  // Transaction Transport Protocol
+	ProtocolIPTM           = 84  // Internet Protocol Traffic Manager
+	ProtocolNSFNETIGP      = 85  // NSFNET-IGP
+	ProtocolDGP            = 86  // Dissimilar Gateway Protocol
+	ProtocolTCF            = 87  // TCF
+	ProtocolEIGRP          = 88  // EIGRP
+	ProtocolOSPFIGP        = 89  // OSPFIGP
+	ProtocolSpriteRPC      = 90  // Sprite RPC Protocol
+	ProtocolLARP           = 91  // Locus Address Resolution Protocol
+	ProtocolMTP            = 92  // Multicast Transport Protocol
+	ProtocolAX25           = 93  // AX.25 Frames
+	ProtocolIPIP           = 94  // IP-within-IP Encapsulation Protocol
+	ProtocolSCCSP          = 96  // Semaphore Communications Sec. Pro.
+	ProtocolETHERIP        = 97  // Ethernet-within-IP Encapsulation
+	ProtocolENCAP          = 98  // Encapsulation Header
+	ProtocolGMTP           = 100 // GMTP
+	ProtocolIFMP           = 101 // Ipsilon Flow Management Protocol
+	ProtocolPNNI           = 102 // PNNI over IP
+	ProtocolPIM            = 103 // Protocol Independent Multicast
+	ProtocolARIS           = 104 // ARIS
+	ProtocolSCPS           = 105 // SCPS
+	ProtocolQNX            = 106 // QNX
+	ProtocolAN             = 107 // Active Networks
+	ProtocolIPComp         = 108 // IP Payload Compression Protocol
+	ProtocolSNP            = 109 // Sitara Networks Protocol
+	ProtocolCompaqPeer     = 110 // Compaq Peer Protocol
+	ProtocolIPXinIP        = 111 // IPX in IP
+	ProtocolVRRP           = 112 // Virtual Router Redundancy Protocol
+	ProtocolPGM            = 113 // PGM Reliable Transport Protocol
+	ProtocolL2TP           = 115 // Layer Two Tunneling Protocol
+	ProtocolDDX            = 116 // D-II Data Exchange (DDX)
+	ProtocolIATP           = 117 // Interactive Agent Transfer Protocol
+	ProtocolSTP            = 118 // Schedule Transfer Protocol
+	ProtocolSRP            = 119 // SpectraLink Radio Protocol
+	ProtocolUTI            = 120 // UTI
+	ProtocolSMP            = 121 // Simple Message Protocol
+	ProtocolPTP            = 123 // Performance Transparency Protocol
+	ProtocolISIS           = 124 // ISIS over IPv4
+	ProtocolFIRE           = 125 // FIRE
+	ProtocolCRTP           = 126 // Combat Radio Transport Protocol
+	ProtocolCRUDP          = 127 // Combat Radio User Datagram
+	ProtocolSSCOPMCE       = 128 // SSCOPMCE
+	ProtocolIPLT           = 129 // IPLT
+	ProtocolSPS            = 130 // Secure Packet Shield
+	ProtocolPIPE           = 131 // Private IP Encapsulation within IP
+	ProtocolSCTP           = 132 // Stream Control Transmission Protocol
+	ProtocolFC             = 133 // Fibre Channel
+	ProtocolRSVPE2EIGNORE  = 134 // RSVP-E2E-IGNORE
+	ProtocolMobilityHeader = 135 // Mobility Header
+	ProtocolUDPLite        = 136 // UDPLite
+	ProtocolMPLSinIP       = 137 // MPLS-in-IP
+	ProtocolMANET          = 138 // MANET Protocols
+	ProtocolHIP            = 139 // Host Identity Protocol
+	ProtocolShim6          = 140 // Shim6 Protocol
+	ProtocolWESP           = 141 // Wrapped Encapsulating Security Payload
+	ProtocolROHC           = 142 // Robust Header Compression
+	ProtocolReserved       = 255 // Reserved
+)