derp.Client.ForwardPacket flushed on every call, so each packet a
derper forwards to a mesh peer cost its own TLS record and write
syscall, serialized behind the client's write mutex with the syscall
inside the critical section.
ForwardPacket now appends the frame to one of two 16 KiB buffers and
returns. A flusher goroutine, started if none is running, swaps the
buffer for the spare under the mutex and writes it with the mutex
released, so callers keep filling the spare while a write is in
progress and frames from concurrent callers share a write. A caller
that finds the spare full waits for the write to complete, preserving
backpressure. The buffers are allocated on first forward, so only mesh
clients carry them.
Forward buffers are written directly to the connection rather than
through the client's bufio.Writer. A new mutex serializes those writes
with the client's other senders, since not every Conn tolerates
concurrent writes and bufio can split one frame over several.
A failed write is logged with the number of frames dropped, closes the
connection, wakes waiting callers with the error, and is returned by
every later call, so derphttp reconnects as before.
Adds BenchmarkForwardPacket, which also reports percentiles of
ForwardPacket's own latency, unit tests for batching, backpressure,
oversized frames, write failure, and write serialization, and an
end-to-end test that concurrently forwarded packets all reach the
destination.
Updates tailscale/corp#50628
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add --tcp-save-syn on Linux to record client MSS in a Prometheus histogram. Count retrieval outcomes separately and limit saved-SYN error logs to one per ten minutes across clients. IPv6 extension headers are unsupported.
Estimate the network-namespace SYN rate from kernel counters once per second. Disable TCP_SAVE_SYN on listeners above 100,000/s or if rate measurement fails, leaving it off until restart.
Updates tailscale/corp#49139
Signed-off-by: James Tucker <james@tailscale.com>
Add three things to the connected clients debug page:
app=NAME narrows any of the existing filters to connections that
advertised that app name, and may be repeated to match any of several
(app=tailcat-server&app=tailcat-client). On its own it applies to all
connections. An empty app= matches connections that sent no app name.
App names in the table link to their filter, and the next-page and
sort links carry the app filter along.
sort=connected walks by connection time, ascending being longest
connected first, with -connected for newest first. The next-page
links use the connection time in Unix nanoseconds as the cursor; by
hand, after= also accepts a duration such as 30m, meaning connections
that have been up that long, which is the natural way to ask for
"everything older than half an hour".
format=json returns the page as a JSON object with the filter
description, the matching connection and key counts, how many
connections remain after the page, the next page's relative URL, and
the client rows, so the page can be walked from curl or a script the
same way a browser follows the next links. Rows gain a connectedAt
timestamp alongside the rounded connected duration.
Updates tailscale/corp#48933
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib5d2e7c40a9f13e8a6d7c2b5f9e0a4d3c8b17e62
The derper debug pages had no way to see which clients were connected.
The expvar gauges only give counts, /debug/check only says whether the
counts agree, and /debug/traffic only reports connections that moved
bytes since its last tick, and only if ss is installed.
Add /debug/clients/, which by default serves an index page with a form
to pick one of four filters: ?all lists every connection, ?ip=1.2.3.4
and ?cidr=1.2.0.0/16 list connections from an address or prefix, and
?key=nodekey:... lists the connection(s) for one node key. Each row
shows the connection number, key, remote address, connection age,
flags (home, mesh, prober, notideal, dup/active/disabled), protocol
version, app name, per-connection rx/tx packet and byte counts, and
the estimated unique sender count.
Big derpers have far too many connections for one page, so results
are paginated with keyset cursors rather than page numbers: sort=key,
ip, conn, rx, tx, rxpkts, or txpkts (with a leading - for descending)
picks the walk order, limit=N the page size, and after=X resumes after
that value of the sort field. The next-page links add afterconn=N so a
page boundary that falls among connections sharing a value (duplicate
keys, one IP with many ports, equal counters) resumes exactly. Column
headers link to the other sort orders.
The walk under Server.mu does only a filter match, a cursor comparison,
and at most a bounded-heap operation per connection, so connections
before the cursor are discarded without being copied and at most limit
entries are ever kept. Only the summary counts (matching connections
and keys) look at every connection. Snapshots are taken and the page
rendered after the lock is released, so a slow debug client can't
stall the server. A benchmark with 100k connections takes about 10ms
per page.
There were no per-connection traffic counters before, only the
server-wide ones, so sclient gains four atomic.Uint64 counters (rx/tx
packets and bytes, counting data packets like the server-wide ones)
bumped alongside them. That's 32 bytes per connection. For the counter
sorts, the value is loaded once per connection during the walk and
used for both the cursor test and the heap order, so the order stays
consistent while the counters keep changing.
The sclient preferred field becomes an atomic.Bool so the page can
report which connections are the client's home DERP; it was previously
only touched by the run goroutine.
Updates tailscale/corp#48933
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I4e9b7c2d5a83f61b0e7d2c94a5f8b3e16d7c0a29
When two connections share a node key, they form a dup client set and
noteClientActivity records each sending connection in the set's
sendHistory. It appended on every frame whenever the sender was not the
immediately previous one, and nothing trimmed the slice while both
connections stayed alive. Two connections taking turns sending therefore
grew sendHistory by one *sclient per frame without bound.
Under the default lastWriterIsActive policy nothing ever stops that
growth, so a malicious client (which controls its own node key) or a
buggy one that keeps two connections alive and both sending could leak
server memory, roughly 8 bytes per frame, for the life of the
connection pair.
Record the sender by moving it to the end of sendHistory and dropping any
earlier occurrence, so each connection appears at most once and the slice
stays bounded by the number of connections in the set. This preserves the
existing behavior: the fighting check under disableFighters still runs
before the move and still disables everyone on the first repeat, and
removeClient still promotes the previous speaker from the slice tail.
Fixestailscale/corp#48884
Change-Id: I06198178e6ab0d7e2f04c1dc4c09eafcb16ace46
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The per-client receive rate limiter charges at most
minRateLimitTokenBucketSize tokens per frame, on the assumption that any
frame larger than that would fail validation and close the connection.
That held for every known frame type, but not for unknown ones:
handleUnknownFrame discarded exactly the declared length, up to 4 GiB,
and returned nil, keeping the connection open.
A client could therefore send unknown frame types with huge declared
lengths and have the server read the bytes off the socket while being
charged only 64 KiB of tokens per frame, bypassing the operator's
configured per-client rate limit by a factor of about 65,000.
Reject unknown frames whose declared length exceeds the largest frame a
regular client can send today, which makes the limiter's assumption true
for all frame types while still tolerating small, reasonably sized
frames from newer clients. Add a test that the server still tolerates
a small unknown frame and closes the connection on a huge one.
Fixestailscale/corp#48889
Updates tailscale/corp#40171
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3f6a2c9d8e1b4a7f5c0d2e9b8a6f4c1d7e3b5a90
ServeDebugTraffic held s.mu while JSON-encoding each record straight
to the ResponseWriter, so a debug client on a slow connection could
block the network write with the server mutex held and stall the
whole DERP server.
Encode into a private bytes.Buffer instead and, once it passes a
threshold size, release s.mu, write the buffer to the network, and
re-take the lock before continuing. That keeps the lock off the
network path without toggling it around every record.
Fixestailscale/corp#48890
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I7c3e91a4d2f58b06e1a9c4f7d3b28e5a61f09c4d
Each client connection ran two goroutines for its lifetime: the reader
in sclient.run and a sendLoop blocked in a select over its send
queues, pong, peer gone, mesh update, and keepalive channels. Almost
all clients are idle at any moment, so the second goroutine mostly
pinned memory: a 4 KiB stack, a g struct, a sudog per select case,
three channels, and a context and errgroup. At 100k idle connections
that was about 8 KB of a client's 22 KB RSS.
Instead, start up the sendLoop only as needed, letting the goroutine
go away otherwise, like Go 1.28-dev's http2 code
(golang/go@5c51011e82) with similar parking to
https://go.dev/cl/834084 but DERP's producers are all non-blocking, so
a kick bit replaces that http2 code's send count.
Measured with 100k idle TLS connections, server RSS per client went
from 22.3 KB to 14.5 KB (22.9 KB to 15.3 KB after each connection had
carried a packet), goroutines dropped from 2 to 1, and with no change
in BenchmarkSendRecv throughput or allocations and no change in the
time to do 100k serial round trips, each of which parks and wakes the
writer. (it's super cheap to start goroutines)
Updates #21064
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I7c3e9a51d4b8f2607a1e5c3d9f8b2a4e6c0d1f3b
Each connected client kept a HyperLogLog sketch of the peers that had
sent it packets, and every relayed packet was inserted into it under a
mutex. That costs memory per client and time on the packet path for a
debug-only estimate that few servers look at.
Keep the accounting but only allocate the sketch when the new
TS_DERP_SENDER_CARDINALITY environment variable is set. When it is
unset, EstimatedUniqueSenders reports 0 and the debug traffic page
omits the field as before.
Updates tailscale/corp#24681
Change-Id: I471bf816e5461069d50b97696e8d91ec3cbe3389
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Every packet the server relayed allocated a fresh []byte for its
payload in recvPacket or recvForwardPacket and dropped it once the
destination's sendLoop had written it. On one busy server, this was
observed allocating about 160 MB/sec of short-lived garbage, and GC
plus malloc were about 5% of the process CPU profile.
Instead, take payload buffers from a size-classed sync.Pool on the
Server, with power-of-two classes from 1 KiB up to derp.MaxPacketSize,
and return them once the packet has been written, forwarded, or
dropped. sync.Pool holds nothing per connection and is trimmed by the
GC, so idle clients pin no memory; only packets actually in flight
hold a buffer. A compile-time assertion ties the largest size class to
derp.MaxPacketSize, and the get and put helpers panic on sizes outside
the pool's classes rather than indexing past it.
Because the memory is now reused, PacketForwarder implementations must
not retain the payload after ForwardPacket returns. Make that explicit
in the signature: the payload is passed as a new derp.LoanedBytes
value, which exposes only Len, WriteTo, and Clone, so an implementation
has to copy to keep it. derp.Client and derphttp.Client, the real
implementations, already wrote it out synchronously; the test-only
channelFwd now clones.
BenchmarkSendRecv shows one fewer allocation per relayed packet and,
for 1000-byte packets, B/op down from 1278 to 263. ns/op on the
loopback benchmarks is dominated by syscalls and is unchanged within
noise.
Updates #21064
Change-Id: Ie40c82388ddb5d22f75fa828749b53fcaba9adde
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
sclient.debugLogf and Server.debugLogf check a debug flag before
logging, but Go evaluates and boxes their arguments before the call.
The per-packet call sites in run, handleFrameSendPacket,
handleFrameForwardPacket, sendPkt, recordDrop, and sendPacket's
deferred stats func were therefore calling key.NodePublic.ShortString
and boxing frame headers on every relayed packet, all for messages
that were then discarded.
On one busy server's heap profile, those discarded arguments were
about half of all objects allocated by the process. Guard each hot
call site with the debug flag so nothing is built unless it will be
logged, and document that requirement on both debugLogf methods.
While here, give the sender cardinality sketch its key bytes from a
stack array rather than an AppendTo(nil) allocation per packet.
BenchmarkSendRecv drops from 10 or 11 allocations per relayed packet
to 3, and BenchmarkConcurrentStreams from 11 to 4.
Updates #21064
Change-Id: Ibb7c4fbff546b6f1b40ce0a21d41ae0976705941
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Each connected client held two buffered channels of 32 pkts each (its
regular and disco send queues), about 5.5 KB per client for the
lifetime of the connection, even though almost all clients are idle at
any given moment.
Replace the channels with pktQueue, a mutex-guarded ring buffer whose
backing array comes from a per-server sync.Pool on first enqueue and
goes back when the queue drains empty, so an idle client holds no
queue memory at all. Enqueuers wake the client's sendLoop through a
one-slot channel.
Updates #21064
Change-Id: I2d7e8f4a1b6c3950e2a7d4b8f1c5e3a9d6b0c2e4
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Previously the DERP handler served each connection for its lifetime
on its hijacked connection's net/http handler goroutine. That
goroutine's conn.serve stack frame kept the dead HTTP/1 server state
reachable for the whole DERP connection: the http.conn and its 4KB
bufio.Writer (hijack hands over c.bufw but conn.serve still references
it, so derpserver returning it to its flush pool never made it
collectable), the 4KB bufio.Reader, the upgrade *http.Request with its
parsed headers, and the request context chain. The goroutine also kept
the stack growth from the TLS handshake and HTTP request parsing.
Instead, hand the connection off to a new goroutine and return from
the handler (ala tailscale/corp@dc09e27aef), letting all the HTTP
upgrade state be collected. Give Accept a smaller 1KB frame reader,
draining and releasing the hijacked reader if it contains buffered
bytes from a fast-start client, and a nil bufio.Writer so writes go
through pooled buffers held only for the duration of a write instead
of a per-connection buffer.
Because the handler now returns at handoff time, cmd/derper's
gauge_derper_tls_active_version decrement can no longer be deferred
to handler return: intercept Hijack in the TLS metrics wrapper and,
for hijacked connections (DERP, its WebSocket flavor, and CONNECT),
decrement the gauge once when the hijacked connection closes,
restoring the gauge's connection-lifetime semantics. Teach
derpserver's TCP RTT stats to unwrap the close-hook conn so they
still find the underlying *net.TCPConn.
Also soften the UntypedHexString deprecation notices in types/key to
warnings: the untyped hex string format is the DERP wire protocol's
key encoding, so these call sites are legitimate and permanent, and
a Deprecated marker just makes them light up in editors and linters.
The cautionary text about the format's risks remains.
Measured with 100,000 idle TLS DERP connections on linux/amd64:
standing memory drops from 55.6KB to 32.6KB per connection (-41%).
Updates #21064
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: If05a0c6ea79134807e9e8872861db216
Clients can advertise an opaque app name in their ClientInfo but the
server previously did nothing with it.
Constrain app names to at most 32 bytes of printable ASCII, enforced
both in derp.NewClient and by the server when it parses the ClientInfo.
Extend the peerPresent frame, following its existing pattern of
appending optional fields, with a length-prefixed app name after the
flags byte, so trusted mesh watchers (other DERP nodes and stats
tools) can attribute connections by app. Old clients ignore the extra
bytes; old servers send frames without them.
Also add a derper --disallow-app-names flag taking a comma-separated
list of app names whose connections are refused, except for trusted
mesh peers.
Updates tailscale/corp#24454
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I6e721258675145833aafa1355fabf7fc05a5a204
Add an AppName field to the DERP ClientInfo so DERP servers can
attribute connections to the application making them, primarily for
best effort stats purposes. The value is plumbed per engine instance
rather than via a process global, so a process hosting multiple stacks
can attribute each one's DERP connections separately:
wgengine.Config.DERPAppName flows through magicsock.Options and
derphttp.Client into the naclbox-sealed ClientInfo JSON. Old servers
ignore the unknown field.
There are no callers in the tree yet setting the name.
Updates tailscale/corp#24454
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ia7d3e9c2b6f8140e5a9d7c3b2e6f1a8d4c0b5e9f
ModifyTLSConfigToAddMetaCert (and its inline copy in cmd/derper) appended
the DERP meta cert directly to the *tls.Certificate returned by the
underlying GetCertificate. autocert returns a certificate sharing a cached
chain slice (and, on the TLS-ALPN token path, the same pointer) across
concurrent handshakes, so the in-place append was a data race and could
grow the served chain unboundedly.
Return a shallow copy with the meta cert appended to a fresh backing
array instead, and have cmd/derper reuse ModifyTLSConfigToAddMetaCert
rather than duplicating the wrapper.
Fixes#20352
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add Go tests that drive a real headless Chromium (via chromedp) against
the built cmd/tsconnect/pkg/ artifact and verify the @tailscale/connect
public API surface end-to-end. The package has not been republished in
three years, in part because no test exercises the produced artifact at
runtime — only tsc --noEmit and a Go build run in CI.
TestCreateIPN loads pkg.js into the browser, calls createIPN with a junk
auth key, and asserts that pkg.createIPN / pkg.runSSHSession are
functions and that createIPN() returns an IPN with the documented
run/login/logout/ssh/fetch methods. No control-plane traffic.
TestFetchTailnetPeer stands up a full local tailnet (testcontrol +
DERP + a tsnet.Server peer) and verifies that the browser-side WASM
client can join over WebSocket-noise to the same control, connect to
DERP over WSS, and then ipn.fetch() an HTTP service hosted on the tsnet
peer through the tailnet. The test asserts the response body matches a
known string. Browser state transitions are logged: NoState -> NeedsLogin
-> Starting -> Running.
Tests are opt-in via --run-headless-browser-tests (matching the existing
--run-vm-tests pattern in tstest/natlab/vmtest) so they never fire in
casual `go test ./...` runs. When the flag is set, a test is skipped if
cmd/tsconnect/pkg/ has not been built, and fails with t.Error if no
chromium binary is found on $PATH (honoring $CHROME_BIN as an override).
findChromium also falls back to /Applications/Google Chrome.app and
/Applications/Chromium.app on darwin, since macOS Chrome's executable
lives inside an .app bundle and is not on $PATH by default. The
.github/workflows/test.yml wasm job is extended to install
google-chrome-stable and run the tests with the flag after build-pkg.
To prevent silently testing a stale pkg/main.wasm (built from an older
checkout than the rest of the test invocation), build-pkg now writes
pkg/build-info.json recording the sha256 of the raw (pre-wasm-opt)
go-build output. The test does its own `go build` of
cmd/tsconnect/wasm with the same -tags/-trimpath/-ldflags (factored
into a new cmd/tsconnect/wasmbuild package shared by both call sites)
and t.Fatalfs with a "rebuild" instruction on mismatch. Cost is
near-zero because the Go build cache from the prior build-pkg makes
the rebuild a cache hit.
The new wasmbuild package also replaces cmd/tsconnect's hardcoded -tags
string with a minimal-feature-set computation. wasmbuild.Keep names the
small set of feature/featuretags entries the browser client actually
needs (netstack, logtail, dns, health, c2n, ipnbus); wasmbuild.Tags()
emits a ts_omit_<f> for every other
omittable feature in feature/featuretags.Features, with transitive deps
expanded via featuretags.Requires. An init() panics if Keep references
a feature unknown to feature/featuretags so a rename there fails
loudly. Net effect on size: 32M raw / 9.4M brotli before this change,
25M raw / 4.4M brotli after — vs the last-published 1.39.98 at 21M /
3.8M. The transitive package-import graph is unchanged (176
tailscale.com/* packages either way): featuretags omits eliminate
dead code via `const HasX = false`, not imports. Trimming the import
graph would require a separate, larger refactor splitting interface
packages by build tag.
Writing TestFetchTailnetPeer surfaced several real issues, all fixed
here:
* cmd/tsconnect built the wasm with the nethttpomithttp2 tag, but
control/ts2021 (since commit 1d93bdce2, "control/controlclient:
remove x/net/http2, use net/http", Oct 2025) requires HTTP/2 from
net/http's bundled implementation. With nethttpomithttp2 set, the
bundle is excluded and the wasm client cannot speak HTTP/2 to any
control plane, including production. Drop the tag. Wasm size grows
~1 MB raw / ~300 KB brotli (more than offset by the feature
pruning above). The last published @tailscale/connect (1.39.98,
early 2023) pre-dates the regression, which is why no consumer has
reported the breakage.
* tstest/integration/testcontrol.Server's /ts2021 noise upgrade
endpoint rejected anything but POST. WebSocket clients (the only
transport available to browser-WASM) come in as GET. Allow both;
the controlhttp AcceptHTTP path dispatches on the Upgrade header,
so the websocket library still enforces GET for WS upgrades.
This matches production, where the same controlhttpserver.AcceptHTTP
routes purely on the Upgrade header without checking method.
* derp/derphttp's urlString built the DERP URL from node.HostName
only, dropping node.DERPPort. Non-WS clients use a separate code
path (connectToHost) that honors DERPPort, but WebSocket-only
clients (browser-WASM) went through urlString and so could not
reach a DERP running on any port other than 443. Include the port
when it differs from the scheme default.
Also move addWebSocketSupport from cmd/derper (where it was main-only)
to derp/derpserver.AddWebSocketSupport so tstest/integration.RunDERPAndSTUN
can wrap its DERP handler with WebSocket support — without that, the
test DERP would not accept the browser's wss connection.
Fixes#9394
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Iff9cdee303e3b239924249b5bffb2fd04e02f391
Server.clientsAtomic was introduced in 6b729795c3 as a lock-free
mirror of Server.clients to skip Server.mu on the packet send hot
path. This drops the non-concurrent map and makes all the existing
callers of the old plain map just use the concurrent map, but still
holding Server.mu.
BenchmarkLookupDestHashTrie is unchanged at ~2ns/op.
Fixes#19726
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I0894e4d86914d152b9b5fef969a3184bcb96f678
Replace the process-global Server.mu lookup in the packet send hot path
with a global hashtriemap mirror of local clientSet entries. The
authoritative clients map remains guarded by Server.mu; clientsAtomic is
only a lock-free fast path for active local clients.
Misses, stale inactive client sets, duplicate accounting, and mesh
forwarding still fall back to lookupDestUncached. This avoids taking
Server.mu for the common local active-client send path, at the cost of
adding one global concurrent map that mirrors Server.clients for local
peers.
The benchmark uses four destination peers. The before run sets
TS_DEBUG_DERP_DISABLE_PEER_HASHTRIE=true to force the old mutex lookup
path; the after run uses the hashtrie fast path.
goos: linux
goarch: amd64
pkg: tailscale.com/derp/derpserver
cpu: Intel(R) Xeon(R) 6975P-C
│ before │ after │
│ sec/op │ sec/op vs base │
LookupDestHashTrie-16 176.050n ± 1% 1.904n ± 6% -98.92% (p=0.000 n=10)
│ before │ after │
│ B/op │ B/op vs base │
LookupDestHashTrie-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
│ before │ after │
│ allocs/op │ allocs/op vs base │
LookupDestHashTrie-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
Updates #3560 (very indirectly, historically)
Updates #19713 (as an alternative to that PR)
Change-Id: Ifb72e5c9854ad00e938cd24c6ab9c27312f297e8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This commit enables the operator to set a global rate limit without any
per-client.
Updates tailscale/corp#40962
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Expvars track count of rate limiters exceeding their threshold.
Covers (1) global rate limiter and (2) total of local rate limiters.
Also publish optional rate-limit metrics during ExpVar() call
if -rate-config is specified. Fixes current rate-limit metrics
being published outside of "derp" in /debug/vars.
Updates tailscale/corp#38509
Change-Id: Ic7f5a1e890d0d7d3d7b679daa4b5f8926a6a6964
Signed-off-by: Alex Valiushko <alexvaliushko@tailscale.com>
By adding a server-global parent bucket. Per-client rate limiting is
subject to the parent bucket if global rate limiting is enabled.
This implementation is experimental, and all related APIs should be
considered unstable.
Updates tailscale/corp#40291
Signed-off-by: Jordan Whited <jordan@tailscale.com>
And cap WaitN calls to prevent token bucket errors. Frame length is
inclusive of DERP key for FrameSendPacket frames.
Updates tailscale/corp#40171
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Add a --rate-config flag pointing to a JSON file for per-client receive
rate limits (bytes/sec and burst bytes). The config is reloaded on SIGHUP,
updating all existing client connections live. The --per-client-rate-limit
and --per-client-rate-burst flags are removed in favor of the config file.
In derpserver, rate limiting uses an atomic.Pointer[xrate.Limiter] per
client: nil when unlimited or mesh (zero overhead), non-nil when
rate-limited.
Document that clientSet.activeClient Store operations require Server.mu.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add server-side per-client bandwidth enforcement using TCP backpressure.
When configured, the server calls WaitN after reading each DERP frame,
which delays the next read, fills the TCP receive buffer, shrinks
the TCP window, and naturally throttles the sender — no packets are dropped.
- Rate limiting is on the receive (inbound) side, which is what an abusive
client controls
- Mesh peers are exempt since they are trusted infrastructure
- The burst size is at least MaxPacketSize (64KB) to ensure a
single max-size frame can always be processed
Also refactors sclient to store a context.Context directly instead of a
done channel, which simplifies the rate limiter's WaitN call.
Flags added to cmd/derper:
--per-client-rate-limit (bytes/sec, default 0 = unlimited)
--per-client-rate-burst (bytes, default 0 = 2x rate limit)
Example for 10Mbps: --per-client-rate-limit=1250000
Updates #38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Replace byte-at-a-time ReadByte loops with Peek+Discard in the DERP
read path. Peek returns a slice into bufio's internal buffer without
allocating, and Discard advances the read pointer without copying.
Introduce util/bufiox with a BufferedReader interface and ReadFull
helper that uses Peek+copy+Discard as an allocation-free alternative
to io.ReadFull.
- derp.ReadFrameHeader: replace 5× ReadByte with Peek(5)+Discard(5),
reading the frame type and length directly from the peeked slice.
Remove now-unused readUint32 helper.
name old ns/op new ns/op speedup
ReadFrameHeader-8 24.2 12.4 ~2x
(0 allocs/op in both)
- key.NodePublic.ReadRawWithoutAllocating: replace 32× ReadByte with
bufiox.ReadFull. Addresses the "Dear future" comment about switching
away from byte-at-a-time reads once a non-escaping alternative exists.
name old ns/op new ns/op speedup
NodeReadRawWithoutAllocating-8 140 43.6 ~3.2x
(0 allocs/op in both)
- derpserver.handleFramePing: replace io.ReadFull with bufiox.ReadFull.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
I omitted a lot of the min/max modernizers because they didn't
result in more clear code.
Some of it's older "for x := range 123".
Also: errors.AsType, any, fmt.Appendf, etc.
Updates #18682
Change-Id: I83a451577f33877f962766a5b65ce86f7696471c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This file was never truly necessary and has never actually been used in
the history of Tailscale's open source releases.
A Brief History of AUTHORS files
---
The AUTHORS file was a pattern developed at Google, originally for
Chromium, then adopted by Go and a bunch of other projects. The problem
was that Chromium originally had a copyright line only recognizing
Google as the copyright holder. Because Google (and most open source
projects) do not require copyright assignemnt for contributions, each
contributor maintains their copyright. Some large corporate contributors
then tried to add their own name to the copyright line in the LICENSE
file or in file headers. This quickly becomes unwieldy, and puts a
tremendous burden on anyone building on top of Chromium, since the
license requires that they keep all copyright lines intact.
The compromise was to create an AUTHORS file that would list all of the
copyright holders. The LICENSE file and source file headers would then
include that list by reference, listing the copyright holder as "The
Chromium Authors".
This also become cumbersome to simply keep the file up to date with a
high rate of new contributors. Plus it's not always obvious who the
copyright holder is. Sometimes it is the individual making the
contribution, but many times it may be their employer. There is no way
for the proejct maintainer to know.
Eventually, Google changed their policy to no longer recommend trying to
keep the AUTHORS file up to date proactively, and instead to only add to
it when requested: https://opensource.google/docs/releasing/authors.
They are also clear that:
> Adding contributors to the AUTHORS file is entirely within the
> project's discretion and has no implications for copyright ownership.
It was primarily added to appease a small number of large contributors
that insisted that they be recognized as copyright holders (which was
entirely their right to do). But it's not truly necessary, and not even
the most accurate way of identifying contributors and/or copyright
holders.
In practice, we've never added anyone to our AUTHORS file. It only lists
Tailscale, so it's not really serving any purpose. It also causes
confusion because Tailscalars put the "Tailscale Inc & AUTHORS" header
in other open source repos which don't actually have an AUTHORS file, so
it's ambiguous what that means.
Instead, we just acknowledge that the contributors to Tailscale (whoever
they are) are copyright holders for their individual contributions. We
also have the benefit of using the DCO (developercertificate.org) which
provides some additional certification of their right to make the
contribution.
The source file changes were purely mechanical with:
git ls-files | xargs sed -i -e 's/\(Tailscale Inc &\) AUTHORS/\1 contributors/g'
Updates #cleanup
Change-Id: Ia101a4a3005adb9118051b3416f5a64a4a45987d
Signed-off-by: Will Norris <will@tailscale.com>
Adds an observation point that may identify potentially abusive traffic
patterns at outlier values.
Updates tailscale/corp#24681
Signed-off-by: James Tucker <james@tailscale.com>
PR #17258 extracted `derp.Server` into `derp/derpserver.Server`.
This followup patch adds the following cleanups:
1. Rename `derp_server*.go` files to `derpserver*.go` to match
the package name.
2. Rename the `derpserver.NewServer` constructor to `derpserver.New`
to reduce stuttering.
3. Remove the unnecessary `derpserver.Conn` type alias.
Updates #17257
Updates #cleanup
Signed-off-by: Simon Law <sfllaw@tailscale.com>
This exports a number of things from the derp (generic + client) package
to be used by the new derpserver package, as now used by cmd/derper.
And then enough other misc changes to lock in that cmd/tailscaled can
be configured to not bring in tailscale.com/client/local. (The webclient
in particular, even when disabled, was bringing it in, so that's now fixed)
Fixes#17257
Change-Id: I88b6c7958643fb54f386dd900bddf73d2d4d96d5
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Mike O'Driscoll