Add three things to the connected clients debug page:
app=NAME narrows any of the existing filters to connections that
advertised that app name, and may be repeated to match any of several
(app=tailcat-server&app=tailcat-client). On its own it applies to all
connections. An empty app= matches connections that sent no app name.
App names in the table link to their filter, and the next-page and
sort links carry the app filter along.
sort=connected walks by connection time, ascending being longest
connected first, with -connected for newest first. The next-page
links use the connection time in Unix nanoseconds as the cursor; by
hand, after= also accepts a duration such as 30m, meaning connections
that have been up that long, which is the natural way to ask for
"everything older than half an hour".
format=json returns the page as a JSON object with the filter
description, the matching connection and key counts, how many
connections remain after the page, the next page's relative URL, and
the client rows, so the page can be walked from curl or a script the
same way a browser follows the next links. Rows gain a connectedAt
timestamp alongside the rounded connected duration.
Updates tailscale/corp#48933
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib5d2e7c40a9f13e8a6d7c2b5f9e0a4d3c8b17e62
* derp/derphttp: reject invalid DERP node hostname before proxy CONNECT
When a DERP client reaches a node through an HTTP(S) proxy,
dialNodeUsingProxy writes the CONNECT request by hand and puts
net.JoinHostPort(n.HostName, port) into both the request line and the
Host header. n.HostName comes from the control-supplied DERP map and
net.JoinHostPort does no sanitizing, so a hostname carrying CR/LF was
written verbatim into the plaintext request sent to the proxy. That let
whoever populated the DERP map inject extra headers, or a second
pipelined request, into the connection to the operator's proxy.
Validate n.HostName with httpguts.ValidHostHeader at the top of
dialNodeUsingProxy, before the proxy is dialed, and also reject the
empty hostname, which ValidHostHeader accepts. DNS names and IP
literals continue to work.
Add a table-driven test that runs accepted and rejected hostnames
against a fake proxy and checks the CONNECT target that goes out.
Fixestailscale/corp#48122
Signed-off-by: basavaraj-sm05 <basavaraj@digiscrypt.com>
Co-authored-by: Mike Jensen <mikej@tailscale.com>
Signed-off-by: Mike Jensen <mikej@tailscale.com>
The derper debug pages had no way to see which clients were connected.
The expvar gauges only give counts, /debug/check only says whether the
counts agree, and /debug/traffic only reports connections that moved
bytes since its last tick, and only if ss is installed.
Add /debug/clients/, which by default serves an index page with a form
to pick one of four filters: ?all lists every connection, ?ip=1.2.3.4
and ?cidr=1.2.0.0/16 list connections from an address or prefix, and
?key=nodekey:... lists the connection(s) for one node key. Each row
shows the connection number, key, remote address, connection age,
flags (home, mesh, prober, notideal, dup/active/disabled), protocol
version, app name, per-connection rx/tx packet and byte counts, and
the estimated unique sender count.
Big derpers have far too many connections for one page, so results
are paginated with keyset cursors rather than page numbers: sort=key,
ip, conn, rx, tx, rxpkts, or txpkts (with a leading - for descending)
picks the walk order, limit=N the page size, and after=X resumes after
that value of the sort field. The next-page links add afterconn=N so a
page boundary that falls among connections sharing a value (duplicate
keys, one IP with many ports, equal counters) resumes exactly. Column
headers link to the other sort orders.
The walk under Server.mu does only a filter match, a cursor comparison,
and at most a bounded-heap operation per connection, so connections
before the cursor are discarded without being copied and at most limit
entries are ever kept. Only the summary counts (matching connections
and keys) look at every connection. Snapshots are taken and the page
rendered after the lock is released, so a slow debug client can't
stall the server. A benchmark with 100k connections takes about 10ms
per page.
There were no per-connection traffic counters before, only the
server-wide ones, so sclient gains four atomic.Uint64 counters (rx/tx
packets and bytes, counting data packets like the server-wide ones)
bumped alongside them. That's 32 bytes per connection. For the counter
sorts, the value is loaded once per connection during the walk and
used for both the cursor test and the heap order, so the order stays
consistent while the counters keep changing.
The sclient preferred field becomes an atomic.Bool so the page can
report which connections are the client's home DERP; it was previously
only touched by the run goroutine.
Updates tailscale/corp#48933
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I4e9b7c2d5a83f61b0e7d2c94a5f8b3e16d7c0a29
When two connections share a node key, they form a dup client set and
noteClientActivity records each sending connection in the set's
sendHistory. It appended on every frame whenever the sender was not the
immediately previous one, and nothing trimmed the slice while both
connections stayed alive. Two connections taking turns sending therefore
grew sendHistory by one *sclient per frame without bound.
Under the default lastWriterIsActive policy nothing ever stops that
growth, so a malicious client (which controls its own node key) or a
buggy one that keeps two connections alive and both sending could leak
server memory, roughly 8 bytes per frame, for the life of the
connection pair.
Record the sender by moving it to the end of sendHistory and dropping any
earlier occurrence, so each connection appears at most once and the slice
stays bounded by the number of connections in the set. This preserves the
existing behavior: the fighting check under disableFighters still runs
before the move and still disables everyone on the first repeat, and
removeClient still promotes the previous speaker from the slice tail.
Fixestailscale/corp#48884
Change-Id: I06198178e6ab0d7e2f04c1dc4c09eafcb16ace46
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The per-client receive rate limiter charges at most
minRateLimitTokenBucketSize tokens per frame, on the assumption that any
frame larger than that would fail validation and close the connection.
That held for every known frame type, but not for unknown ones:
handleUnknownFrame discarded exactly the declared length, up to 4 GiB,
and returned nil, keeping the connection open.
A client could therefore send unknown frame types with huge declared
lengths and have the server read the bytes off the socket while being
charged only 64 KiB of tokens per frame, bypassing the operator's
configured per-client rate limit by a factor of about 65,000.
Reject unknown frames whose declared length exceeds the largest frame a
regular client can send today, which makes the limiter's assumption true
for all frame types while still tolerating small, reasonably sized
frames from newer clients. Add a test that the server still tolerates
a small unknown frame and closes the connection on a huge one.
Fixestailscale/corp#48889
Updates tailscale/corp#40171
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3f6a2c9d8e1b4a7f5c0d2e9b8a6f4c1d7e3b5a90
ServeDebugTraffic held s.mu while JSON-encoding each record straight
to the ResponseWriter, so a debug client on a slow connection could
block the network write with the server mutex held and stall the
whole DERP server.
Encode into a private bytes.Buffer instead and, once it passes a
threshold size, release s.mu, write the buffer to the network, and
re-take the lock before continuing. That keeps the lock off the
network path without toggling it around every record.
Fixestailscale/corp#48890
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I7c3e91a4d2f58b06e1a9c4f7d3b28e5a61f09c4d
Each client connection ran two goroutines for its lifetime: the reader
in sclient.run and a sendLoop blocked in a select over its send
queues, pong, peer gone, mesh update, and keepalive channels. Almost
all clients are idle at any moment, so the second goroutine mostly
pinned memory: a 4 KiB stack, a g struct, a sudog per select case,
three channels, and a context and errgroup. At 100k idle connections
that was about 8 KB of a client's 22 KB RSS.
Instead, start up the sendLoop only as needed, letting the goroutine
go away otherwise, like Go 1.28-dev's http2 code
(golang/go@5c51011e82) with similar parking to
https://go.dev/cl/834084 but DERP's producers are all non-blocking, so
a kick bit replaces that http2 code's send count.
Measured with 100k idle TLS connections, server RSS per client went
from 22.3 KB to 14.5 KB (22.9 KB to 15.3 KB after each connection had
carried a packet), goroutines dropped from 2 to 1, and with no change
in BenchmarkSendRecv throughput or allocations and no change in the
time to do 100k serial round trips, each of which parks and wakes the
writer. (it's super cheap to start goroutines)
Updates #21064
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I7c3e9a51d4b8f2607a1e5c3d9f8b2a4e6c0d1f3b
Each connected client kept a HyperLogLog sketch of the peers that had
sent it packets, and every relayed packet was inserted into it under a
mutex. That costs memory per client and time on the packet path for a
debug-only estimate that few servers look at.
Keep the accounting but only allocate the sketch when the new
TS_DERP_SENDER_CARDINALITY environment variable is set. When it is
unset, EstimatedUniqueSenders reports 0 and the debug traffic page
omits the field as before.
Updates tailscale/corp#24681
Change-Id: I471bf816e5461069d50b97696e8d91ec3cbe3389
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Every packet the server relayed allocated a fresh []byte for its
payload in recvPacket or recvForwardPacket and dropped it once the
destination's sendLoop had written it. On one busy server, this was
observed allocating about 160 MB/sec of short-lived garbage, and GC
plus malloc were about 5% of the process CPU profile.
Instead, take payload buffers from a size-classed sync.Pool on the
Server, with power-of-two classes from 1 KiB up to derp.MaxPacketSize,
and return them once the packet has been written, forwarded, or
dropped. sync.Pool holds nothing per connection and is trimmed by the
GC, so idle clients pin no memory; only packets actually in flight
hold a buffer. A compile-time assertion ties the largest size class to
derp.MaxPacketSize, and the get and put helpers panic on sizes outside
the pool's classes rather than indexing past it.
Because the memory is now reused, PacketForwarder implementations must
not retain the payload after ForwardPacket returns. Make that explicit
in the signature: the payload is passed as a new derp.LoanedBytes
value, which exposes only Len, WriteTo, and Clone, so an implementation
has to copy to keep it. derp.Client and derphttp.Client, the real
implementations, already wrote it out synchronously; the test-only
channelFwd now clones.
BenchmarkSendRecv shows one fewer allocation per relayed packet and,
for 1000-byte packets, B/op down from 1278 to 263. ns/op on the
loopback benchmarks is dominated by syscalls and is unchanged within
noise.
Updates #21064
Change-Id: Ie40c82388ddb5d22f75fa828749b53fcaba9adde
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
sclient.debugLogf and Server.debugLogf check a debug flag before
logging, but Go evaluates and boxes their arguments before the call.
The per-packet call sites in run, handleFrameSendPacket,
handleFrameForwardPacket, sendPkt, recordDrop, and sendPacket's
deferred stats func were therefore calling key.NodePublic.ShortString
and boxing frame headers on every relayed packet, all for messages
that were then discarded.
On one busy server's heap profile, those discarded arguments were
about half of all objects allocated by the process. Guard each hot
call site with the debug flag so nothing is built unless it will be
logged, and document that requirement on both debugLogf methods.
While here, give the sender cardinality sketch its key bytes from a
stack array rather than an AppendTo(nil) allocation per packet.
BenchmarkSendRecv drops from 10 or 11 allocations per relayed packet
to 3, and BenchmarkConcurrentStreams from 11 to 4.
Updates #21064
Change-Id: Ibb7c4fbff546b6f1b40ce0a21d41ae0976705941
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Each connected client held two buffered channels of 32 pkts each (its
regular and disco send queues), about 5.5 KB per client for the
lifetime of the connection, even though almost all clients are idle at
any given moment.
Replace the channels with pktQueue, a mutex-guarded ring buffer whose
backing array comes from a per-server sync.Pool on first enqueue and
goes back when the queue drains empty, so an idle client holds no
queue memory at all. Enqueuers wake the client's sendLoop through a
one-slot channel.
Updates #21064
Change-Id: I2d7e8f4a1b6c3950e2a7d4b8f1c5e3a9d6b0c2e4
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This change expands our fuzzing coverage in protocol and parsing logic. No issues discovered from this fuzzing. Wiring into oss-fuzz for continual coverage.
Updates https://github.com/tailscale/corp/issues/46608
Change-Id: I6b5218cb1103ccc5b957c512a10d87f637c4b6e5
Signed-off-by: Mike Jensen <mikej@tailscale.com>
Previously the DERP handler served each connection for its lifetime
on its hijacked connection's net/http handler goroutine. That
goroutine's conn.serve stack frame kept the dead HTTP/1 server state
reachable for the whole DERP connection: the http.conn and its 4KB
bufio.Writer (hijack hands over c.bufw but conn.serve still references
it, so derpserver returning it to its flush pool never made it
collectable), the 4KB bufio.Reader, the upgrade *http.Request with its
parsed headers, and the request context chain. The goroutine also kept
the stack growth from the TLS handshake and HTTP request parsing.
Instead, hand the connection off to a new goroutine and return from
the handler (ala tailscale/corp@dc09e27aef), letting all the HTTP
upgrade state be collected. Give Accept a smaller 1KB frame reader,
draining and releasing the hijacked reader if it contains buffered
bytes from a fast-start client, and a nil bufio.Writer so writes go
through pooled buffers held only for the duration of a write instead
of a per-connection buffer.
Because the handler now returns at handoff time, cmd/derper's
gauge_derper_tls_active_version decrement can no longer be deferred
to handler return: intercept Hijack in the TLS metrics wrapper and,
for hijacked connections (DERP, its WebSocket flavor, and CONNECT),
decrement the gauge once when the hijacked connection closes,
restoring the gauge's connection-lifetime semantics. Teach
derpserver's TCP RTT stats to unwrap the close-hook conn so they
still find the underlying *net.TCPConn.
Also soften the UntypedHexString deprecation notices in types/key to
warnings: the untyped hex string format is the DERP wire protocol's
key encoding, so these call sites are legitimate and permanent, and
a Deprecated marker just makes them light up in editors and linters.
The cautionary text about the format's risks remains.
Measured with 100,000 idle TLS DERP connections on linux/amd64:
standing memory drops from 55.6KB to 32.6KB per connection (-41%).
Updates #21064
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: If05a0c6ea79134807e9e8872861db216
Clients can advertise an opaque app name in their ClientInfo but the
server previously did nothing with it.
Constrain app names to at most 32 bytes of printable ASCII, enforced
both in derp.NewClient and by the server when it parses the ClientInfo.
Extend the peerPresent frame, following its existing pattern of
appending optional fields, with a length-prefixed app name after the
flags byte, so trusted mesh watchers (other DERP nodes and stats
tools) can attribute connections by app. Old clients ignore the extra
bytes; old servers send frames without them.
Also add a derper --disallow-app-names flag taking a comma-separated
list of app names whose connections are refused, except for trusted
mesh peers.
Updates tailscale/corp#24454
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I6e721258675145833aafa1355fabf7fc05a5a204
Test that the client parses peerPresent frames from servers of various
eras: old servers that send fewer fields than the client knows about,
and newer servers that send trailing fields the client doesn't know
about, which it must ignore. This matters during rollouts of new DERP
servers, when a region's meshed nodes and watchers run a mix of
versions.
This is in advance of a following commit that extends the frame with a
new trailing field.
Updates tailscale/corp#24454
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I0d46126a6003a9eb07a4da46dc291872149b1a07
Add an AppName field to the DERP ClientInfo so DERP servers can
attribute connections to the application making them, primarily for
best effort stats purposes. The value is plumbed per engine instance
rather than via a process global, so a process hosting multiple stacks
can attribute each one's DERP connections separately:
wgengine.Config.DERPAppName flows through magicsock.Options and
derphttp.Client into the naclbox-sealed ClientInfo JSON. Old servers
ignore the unknown field.
There are no callers in the tree yet setting the name.
Updates tailscale/corp#24454
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ia7d3e9c2b6f8140e5a9d7c3b2e6f1a8d4c0b5e9f
ModifyTLSConfigToAddMetaCert (and its inline copy in cmd/derper) appended
the DERP meta cert directly to the *tls.Certificate returned by the
underlying GetCertificate. autocert returns a certificate sharing a cached
chain slice (and, on the TLS-ALPN token path, the same pointer) across
concurrent handshakes, so the in-place append was a data race and could
grow the served chain unboundedly.
Return a shallow copy with the meta cert appended to a fresh backing
array instead, and have cmd/derper reuse ModifyTLSConfigToAddMetaCert
rather than duplicating the wrapper.
Fixes#20352
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Found with the regex `\b([A-Za-z]+) \1\b`.
Updates #cleanup
Change-Id: I4cc51784d9b6437d3d0c66b531828707f87f7fd5
Signed-off-by: Alex Chan <alexc@tailscale.com>
Adds two tests covering the fix in 0e4c8fc92:
TestDialNodeUsingProxyPort exercises dialNodeUsingProxy directly via a
stub CONNECT proxy, asserting the recorded target across four cases:
HTTPS/HTTP default fallback and explicit DERPPort override for each.
TestConnectThroughProxyHonorsDERPPort drives the full path end-to-end:
a real derpserver on an ephemeral TLS port, a real CONNECT proxy that
tunnels bytes bidirectionally, and a region client routed through it
via feature.HookProxyFromEnvironment. Without the fix, Connect fails
because the proxy is asked to dial :443.
Signed-off-by: Martin Zihlmann <martizih@outlook.com>
dialNode picks the destination port from n.DERPPort when non-zero,
falling back to 443 (or 3340 when useHTTPS is false). The proxy path,
dialNodeUsingProxy, hardcoded "443" in the CONNECT target, so a DERP
server reachable only on a custom port was unreachable through
HTTPS_PROXY: the proxy would faithfully tunnel to :443 at the DERP
hostname, and TLS would either fail cert validation or talk to the
wrong service.
Mirror dialNode's port selection so both paths behave the same.
Fixes#19748
Signed-off-by: Martin Zihlmann <martizih@outlook.com>
Add Go tests that drive a real headless Chromium (via chromedp) against
the built cmd/tsconnect/pkg/ artifact and verify the @tailscale/connect
public API surface end-to-end. The package has not been republished in
three years, in part because no test exercises the produced artifact at
runtime — only tsc --noEmit and a Go build run in CI.
TestCreateIPN loads pkg.js into the browser, calls createIPN with a junk
auth key, and asserts that pkg.createIPN / pkg.runSSHSession are
functions and that createIPN() returns an IPN with the documented
run/login/logout/ssh/fetch methods. No control-plane traffic.
TestFetchTailnetPeer stands up a full local tailnet (testcontrol +
DERP + a tsnet.Server peer) and verifies that the browser-side WASM
client can join over WebSocket-noise to the same control, connect to
DERP over WSS, and then ipn.fetch() an HTTP service hosted on the tsnet
peer through the tailnet. The test asserts the response body matches a
known string. Browser state transitions are logged: NoState -> NeedsLogin
-> Starting -> Running.
Tests are opt-in via --run-headless-browser-tests (matching the existing
--run-vm-tests pattern in tstest/natlab/vmtest) so they never fire in
casual `go test ./...` runs. When the flag is set, a test is skipped if
cmd/tsconnect/pkg/ has not been built, and fails with t.Error if no
chromium binary is found on $PATH (honoring $CHROME_BIN as an override).
findChromium also falls back to /Applications/Google Chrome.app and
/Applications/Chromium.app on darwin, since macOS Chrome's executable
lives inside an .app bundle and is not on $PATH by default. The
.github/workflows/test.yml wasm job is extended to install
google-chrome-stable and run the tests with the flag after build-pkg.
To prevent silently testing a stale pkg/main.wasm (built from an older
checkout than the rest of the test invocation), build-pkg now writes
pkg/build-info.json recording the sha256 of the raw (pre-wasm-opt)
go-build output. The test does its own `go build` of
cmd/tsconnect/wasm with the same -tags/-trimpath/-ldflags (factored
into a new cmd/tsconnect/wasmbuild package shared by both call sites)
and t.Fatalfs with a "rebuild" instruction on mismatch. Cost is
near-zero because the Go build cache from the prior build-pkg makes
the rebuild a cache hit.
The new wasmbuild package also replaces cmd/tsconnect's hardcoded -tags
string with a minimal-feature-set computation. wasmbuild.Keep names the
small set of feature/featuretags entries the browser client actually
needs (netstack, logtail, dns, health, c2n, ipnbus); wasmbuild.Tags()
emits a ts_omit_<f> for every other
omittable feature in feature/featuretags.Features, with transitive deps
expanded via featuretags.Requires. An init() panics if Keep references
a feature unknown to feature/featuretags so a rename there fails
loudly. Net effect on size: 32M raw / 9.4M brotli before this change,
25M raw / 4.4M brotli after — vs the last-published 1.39.98 at 21M /
3.8M. The transitive package-import graph is unchanged (176
tailscale.com/* packages either way): featuretags omits eliminate
dead code via `const HasX = false`, not imports. Trimming the import
graph would require a separate, larger refactor splitting interface
packages by build tag.
Writing TestFetchTailnetPeer surfaced several real issues, all fixed
here:
* cmd/tsconnect built the wasm with the nethttpomithttp2 tag, but
control/ts2021 (since commit 1d93bdce2, "control/controlclient:
remove x/net/http2, use net/http", Oct 2025) requires HTTP/2 from
net/http's bundled implementation. With nethttpomithttp2 set, the
bundle is excluded and the wasm client cannot speak HTTP/2 to any
control plane, including production. Drop the tag. Wasm size grows
~1 MB raw / ~300 KB brotli (more than offset by the feature
pruning above). The last published @tailscale/connect (1.39.98,
early 2023) pre-dates the regression, which is why no consumer has
reported the breakage.
* tstest/integration/testcontrol.Server's /ts2021 noise upgrade
endpoint rejected anything but POST. WebSocket clients (the only
transport available to browser-WASM) come in as GET. Allow both;
the controlhttp AcceptHTTP path dispatches on the Upgrade header,
so the websocket library still enforces GET for WS upgrades.
This matches production, where the same controlhttpserver.AcceptHTTP
routes purely on the Upgrade header without checking method.
* derp/derphttp's urlString built the DERP URL from node.HostName
only, dropping node.DERPPort. Non-WS clients use a separate code
path (connectToHost) that honors DERPPort, but WebSocket-only
clients (browser-WASM) went through urlString and so could not
reach a DERP running on any port other than 443. Include the port
when it differs from the scheme default.
Also move addWebSocketSupport from cmd/derper (where it was main-only)
to derp/derpserver.AddWebSocketSupport so tstest/integration.RunDERPAndSTUN
can wrap its DERP handler with WebSocket support — without that, the
test DERP would not accept the browser's wss connection.
Fixes#9394
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Iff9cdee303e3b239924249b5bffb2fd04e02f391
Server.clientsAtomic was introduced in 6b729795c3 as a lock-free
mirror of Server.clients to skip Server.mu on the packet send hot
path. This drops the non-concurrent map and makes all the existing
callers of the old plain map just use the concurrent map, but still
holding Server.mu.
BenchmarkLookupDestHashTrie is unchanged at ~2ns/op.
Fixes#19726
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I0894e4d86914d152b9b5fef969a3184bcb96f678
Replace the process-global Server.mu lookup in the packet send hot path
with a global hashtriemap mirror of local clientSet entries. The
authoritative clients map remains guarded by Server.mu; clientsAtomic is
only a lock-free fast path for active local clients.
Misses, stale inactive client sets, duplicate accounting, and mesh
forwarding still fall back to lookupDestUncached. This avoids taking
Server.mu for the common local active-client send path, at the cost of
adding one global concurrent map that mirrors Server.clients for local
peers.
The benchmark uses four destination peers. The before run sets
TS_DEBUG_DERP_DISABLE_PEER_HASHTRIE=true to force the old mutex lookup
path; the after run uses the hashtrie fast path.
goos: linux
goarch: amd64
pkg: tailscale.com/derp/derpserver
cpu: Intel(R) Xeon(R) 6975P-C
│ before │ after │
│ sec/op │ sec/op vs base │
LookupDestHashTrie-16 176.050n ± 1% 1.904n ± 6% -98.92% (p=0.000 n=10)
│ before │ after │
│ B/op │ B/op vs base │
LookupDestHashTrie-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
│ before │ after │
│ allocs/op │ allocs/op vs base │
LookupDestHashTrie-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
Updates #3560 (very indirectly, historically)
Updates #19713 (as an alternative to that PR)
Change-Id: Ifb72e5c9854ad00e938cd24c6ab9c27312f297e8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This commit enables the operator to set a global rate limit without any
per-client.
Updates tailscale/corp#40962
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Expvars track count of rate limiters exceeding their threshold.
Covers (1) global rate limiter and (2) total of local rate limiters.
Also publish optional rate-limit metrics during ExpVar() call
if -rate-config is specified. Fixes current rate-limit metrics
being published outside of "derp" in /debug/vars.
Updates tailscale/corp#38509
Change-Id: Ic7f5a1e890d0d7d3d7b679daa4b5f8926a6a6964
Signed-off-by: Alex Valiushko <alexvaliushko@tailscale.com>
By adding a server-global parent bucket. Per-client rate limiting is
subject to the parent bucket if global rate limiting is enabled.
This implementation is experimental, and all related APIs should be
considered unstable.
Updates tailscale/corp#40291
Signed-off-by: Jordan Whited <jordan@tailscale.com>
And cap WaitN calls to prevent token bucket errors. Frame length is
inclusive of DERP key for FrameSendPacket frames.
Updates tailscale/corp#40171
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Add a --rate-config flag pointing to a JSON file for per-client receive
rate limits (bytes/sec and burst bytes). The config is reloaded on SIGHUP,
updating all existing client connections live. The --per-client-rate-limit
and --per-client-rate-burst flags are removed in favor of the config file.
In derpserver, rate limiting uses an atomic.Pointer[xrate.Limiter] per
client: nil when unlimited or mesh (zero overhead), non-nil when
rate-limited.
Document that clientSet.activeClient Store operations require Server.mu.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add server-side per-client bandwidth enforcement using TCP backpressure.
When configured, the server calls WaitN after reading each DERP frame,
which delays the next read, fills the TCP receive buffer, shrinks
the TCP window, and naturally throttles the sender — no packets are dropped.
- Rate limiting is on the receive (inbound) side, which is what an abusive
client controls
- Mesh peers are exempt since they are trusted infrastructure
- The burst size is at least MaxPacketSize (64KB) to ensure a
single max-size frame can always be processed
Also refactors sclient to store a context.Context directly instead of a
done channel, which simplifies the rate limiter's WaitN call.
Flags added to cmd/derper:
--per-client-rate-limit (bytes/sec, default 0 = unlimited)
--per-client-rate-burst (bytes, default 0 = 2x rate limit)
Example for 10Mbps: --per-client-rate-limit=1250000
Updates #38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Add a new vet analyzer that checks t.Run subtest names don't contain
characters requiring quoting when re-running via "go test -run". This
enforces the style guide rule: don't use spaces or punctuation in
subtest names.
The analyzer flags:
- Direct t.Run calls with string literal names containing spaces,
regex metacharacters, quotes, or other problematic characters
- Table-driven t.Run(tt.name, ...) calls where tt ranges over a
slice/map literal with bad name field values
Also fix all 978 existing violations across 81 test files, replacing
spaces with hyphens and shortening long sentence-like names to concise
hyphenated forms.
Updates #19242
Change-Id: Ib0ad96a111bd8e764582d1d4902fe2599454ab65
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Use bufio.Writer.AvailableBuffer to write the frame header directly
into bufio's internal buffer as a single append+Write, avoiding 5
separate WriteByte calls. Fall back to the existing writeUint32
byte-at-a-time path when the buffer has insufficient space.
```
name old ns/op new ns/op speedup
WriteFrameHeader-8 18.8 7.8 ~2.4x
(0 allocs/op in both)
```
Add TestWriteFrameHeader with correctness
checks, allocation assertions, and coverage of both fast and slow
write paths. Move BenchmarkReadFrameHeader from client_test.go to
derp_test.go alongside BenchmarkWriteFrameHeader, co-located with
the functions under test.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Replace byte-at-a-time ReadByte loops with Peek+Discard in the DERP
read path. Peek returns a slice into bufio's internal buffer without
allocating, and Discard advances the read pointer without copying.
Introduce util/bufiox with a BufferedReader interface and ReadFull
helper that uses Peek+copy+Discard as an allocation-free alternative
to io.ReadFull.
- derp.ReadFrameHeader: replace 5× ReadByte with Peek(5)+Discard(5),
reading the frame type and length directly from the peeked slice.
Remove now-unused readUint32 helper.
name old ns/op new ns/op speedup
ReadFrameHeader-8 24.2 12.4 ~2x
(0 allocs/op in both)
- key.NodePublic.ReadRawWithoutAllocating: replace 32× ReadByte with
bufiox.ReadFull. Addresses the "Dear future" comment about switching
away from byte-at-a-time reads once a non-escaping alternative exists.
name old ns/op new ns/op speedup
NodeReadRawWithoutAllocating-8 140 43.6 ~3.2x
(0 allocs/op in both)
- derpserver.handleFramePing: replace io.ReadFull with bufiox.ReadFull.
Updates tailscale/corp#38509
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
I omitted a lot of the min/max modernizers because they didn't
result in more clear code.
Some of it's older "for x := range 123".
Also: errors.AsType, any, fmt.Appendf, etc.
Updates #18682
Change-Id: I83a451577f33877f962766a5b65ce86f7696471c
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This file was never truly necessary and has never actually been used in
the history of Tailscale's open source releases.
A Brief History of AUTHORS files
---
The AUTHORS file was a pattern developed at Google, originally for
Chromium, then adopted by Go and a bunch of other projects. The problem
was that Chromium originally had a copyright line only recognizing
Google as the copyright holder. Because Google (and most open source
projects) do not require copyright assignemnt for contributions, each
contributor maintains their copyright. Some large corporate contributors
then tried to add their own name to the copyright line in the LICENSE
file or in file headers. This quickly becomes unwieldy, and puts a
tremendous burden on anyone building on top of Chromium, since the
license requires that they keep all copyright lines intact.
The compromise was to create an AUTHORS file that would list all of the
copyright holders. The LICENSE file and source file headers would then
include that list by reference, listing the copyright holder as "The
Chromium Authors".
This also become cumbersome to simply keep the file up to date with a
high rate of new contributors. Plus it's not always obvious who the
copyright holder is. Sometimes it is the individual making the
contribution, but many times it may be their employer. There is no way
for the proejct maintainer to know.
Eventually, Google changed their policy to no longer recommend trying to
keep the AUTHORS file up to date proactively, and instead to only add to
it when requested: https://opensource.google/docs/releasing/authors.
They are also clear that:
> Adding contributors to the AUTHORS file is entirely within the
> project's discretion and has no implications for copyright ownership.
It was primarily added to appease a small number of large contributors
that insisted that they be recognized as copyright holders (which was
entirely their right to do). But it's not truly necessary, and not even
the most accurate way of identifying contributors and/or copyright
holders.
In practice, we've never added anyone to our AUTHORS file. It only lists
Tailscale, so it's not really serving any purpose. It also causes
confusion because Tailscalars put the "Tailscale Inc & AUTHORS" header
in other open source repos which don't actually have an AUTHORS file, so
it's ambiguous what that means.
Instead, we just acknowledge that the contributors to Tailscale (whoever
they are) are copyright holders for their individual contributions. We
also have the benefit of using the DCO (developercertificate.org) which
provides some additional certification of their right to make the
contribution.
The source file changes were purely mechanical with:
git ls-files | xargs sed -i -e 's/\(Tailscale Inc &\) AUTHORS/\1 contributors/g'
Updates #cleanup
Change-Id: Ia101a4a3005adb9118051b3416f5a64a4a45987d
Signed-off-by: Will Norris <will@tailscale.com>
Adds an observation point that may identify potentially abusive traffic
patterns at outlier values.
Updates tailscale/corp#24681
Signed-off-by: James Tucker <james@tailscale.com>
Found by staticcheck, the test was calling derphttp.NewClient but not checking
its error result before doing other things to it.
Updates #cleanup
Change-Id: I4ade35a7de7c473571f176e747866bc0ab5774db
Signed-off-by: M. J. Fromberger <fromberger@tailscale.com>
Using memnet and synctest removes flakiness caused by real networking
and subtle timing differences.
Additionally, remove the `t.Logf` call inside the server's shutdown
goroutine that was causing a false positive data race detection.
The race detector is flagging a double write during this `t.Logf` call.
This is a common pattern, noted in golang/go#40343 and elsehwere in
this file, where using `t.Logf` after a test has finished can interact
poorly with the test runner.
This is a long-standing issue which became more common after rewriting
this test to use memnet and synctest.
Fixed#17355
Signed-off-by: Alex Chan <alexc@tailscale.com>
Brad Fitzpatrick