04ae61fe4b
Add Go tests that drive a real headless Chromium (via chromedp) against
the built cmd/tsconnect/pkg/ artifact and verify the @tailscale/connect
public API surface end-to-end. The package has not been republished in
three years, in part because no test exercises the produced artifact at
runtime — only tsc --noEmit and a Go build run in CI.
TestCreateIPN loads pkg.js into the browser, calls createIPN with a junk
auth key, and asserts that pkg.createIPN / pkg.runSSHSession are
functions and that createIPN() returns an IPN with the documented
run/login/logout/ssh/fetch methods. No control-plane traffic.
TestFetchTailnetPeer stands up a full local tailnet (testcontrol +
DERP + a tsnet.Server peer) and verifies that the browser-side WASM
client can join over WebSocket-noise to the same control, connect to
DERP over WSS, and then ipn.fetch() an HTTP service hosted on the tsnet
peer through the tailnet. The test asserts the response body matches a
known string. Browser state transitions are logged: NoState -> NeedsLogin
-> Starting -> Running.
Tests are opt-in via --run-headless-browser-tests (matching the existing
--run-vm-tests pattern in tstest/natlab/vmtest) so they never fire in
casual `go test ./...` runs. When the flag is set, a test is skipped if
cmd/tsconnect/pkg/ has not been built, and fails with t.Error if no
chromium binary is found on $PATH (honoring $CHROME_BIN as an override).
findChromium also falls back to /Applications/Google Chrome.app and
/Applications/Chromium.app on darwin, since macOS Chrome's executable
lives inside an .app bundle and is not on $PATH by default. The
.github/workflows/test.yml wasm job is extended to install
google-chrome-stable and run the tests with the flag after build-pkg.
To prevent silently testing a stale pkg/main.wasm (built from an older
checkout than the rest of the test invocation), build-pkg now writes
pkg/build-info.json recording the sha256 of the raw (pre-wasm-opt)
go-build output. The test does its own `go build` of
cmd/tsconnect/wasm with the same -tags/-trimpath/-ldflags (factored
into a new cmd/tsconnect/wasmbuild package shared by both call sites)
and t.Fatalfs with a "rebuild" instruction on mismatch. Cost is
near-zero because the Go build cache from the prior build-pkg makes
the rebuild a cache hit.
The new wasmbuild package also replaces cmd/tsconnect's hardcoded -tags
string with a minimal-feature-set computation. wasmbuild.Keep names the
small set of feature/featuretags entries the browser client actually
needs (netstack, logtail, dns, health, c2n, ipnbus); wasmbuild.Tags()
emits a ts_omit_<f> for every other
omittable feature in feature/featuretags.Features, with transitive deps
expanded via featuretags.Requires. An init() panics if Keep references
a feature unknown to feature/featuretags so a rename there fails
loudly. Net effect on size: 32M raw / 9.4M brotli before this change,
25M raw / 4.4M brotli after — vs the last-published 1.39.98 at 21M /
3.8M. The transitive package-import graph is unchanged (176
tailscale.com/* packages either way): featuretags omits eliminate
dead code via `const HasX = false`, not imports. Trimming the import
graph would require a separate, larger refactor splitting interface
packages by build tag.
Writing TestFetchTailnetPeer surfaced several real issues, all fixed
here:
* cmd/tsconnect built the wasm with the nethttpomithttp2 tag, but
control/ts2021 (since commit 1d93bdce2, "control/controlclient:
remove x/net/http2, use net/http", Oct 2025) requires HTTP/2 from
net/http's bundled implementation. With nethttpomithttp2 set, the
bundle is excluded and the wasm client cannot speak HTTP/2 to any
control plane, including production. Drop the tag. Wasm size grows
~1 MB raw / ~300 KB brotli (more than offset by the feature
pruning above). The last published @tailscale/connect (1.39.98,
early 2023) pre-dates the regression, which is why no consumer has
reported the breakage.
* tstest/integration/testcontrol.Server's /ts2021 noise upgrade
endpoint rejected anything but POST. WebSocket clients (the only
transport available to browser-WASM) come in as GET. Allow both;
the controlhttp AcceptHTTP path dispatches on the Upgrade header,
so the websocket library still enforces GET for WS upgrades.
This matches production, where the same controlhttpserver.AcceptHTTP
routes purely on the Upgrade header without checking method.
* derp/derphttp's urlString built the DERP URL from node.HostName
only, dropping node.DERPPort. Non-WS clients use a separate code
path (connectToHost) that honors DERPPort, but WebSocket-only
clients (browser-WASM) went through urlString and so could not
reach a DERP running on any port other than 443. Include the port
when it differs from the scheme default.
Also move addWebSocketSupport from cmd/derper (where it was main-only)
to derp/derpserver.AddWebSocketSupport so tstest/integration.RunDERPAndSTUN
can wrap its DERP handler with WebSocket support — without that, the
test DERP would not accept the browser's wss connection.
Fixes #9394
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Iff9cdee303e3b239924249b5bffb2fd04e02f391
DERP
This is the code for the Tailscale DERP server.
In general, you should not need to or want to run this code. The overwhelming majority of Tailscale users (both individuals and companies) do not.
In the happy path, Tailscale establishes direct connections between peers and data plane traffic flows directly between them, without using DERP for more than acting as a low bandwidth side channel to bootstrap the NAT traversal. If you find yourself wanting DERP for more bandwidth, the real problem is usually the network configuration of your Tailscale node(s), making sure that Tailscale can get direction connections via some mechanism.
If you've decided or been advised to run your own derper, then read on.
Caveats
-
Node sharing and other cross-Tailnet features don't work when using custom DERP servers.
-
DERP servers only see encrypted WireGuard packets and thus are not useful for network-level debugging.
-
The Tailscale control plane does certain geo-level steering features and optimizations that are not available when using custom DERP servers.
Guide to running cmd/derper
-
You must build and update the
cmd/derperbinary yourself. There are no packages. Usego install tailscale.com/cmd/derper@latestwith the latest version of Go. You should update this binary approximately as regularly as you update Tailscale nodes. If using--verify-clients, thederperbinary andtailscaledbinary on the machine must be built from the same git revision. (It might work otherwise, but they're developed and only tested together.) -
The DERP protocol does a protocol switch inside TLS from HTTP to a custom bidirectional binary protocol. It is thus incompatible with many HTTP proxies. Do not put
derperbehind another HTTP proxy. -
The
tailscaledclient does its own selection of the fastest/nearest DERP server based on latency measurements. Do not putderperbehind a global load balancer. -
DERP servers should ideally have both a static IPv4 and static IPv6 address. Both of those should be listed in the DERP map so the client doesn't need to rely on its DNS which might be broken and dependent on DERP to get back up.
-
A DERP server should not share an IP address with any other DERP server.
-
Avoid having multiple DERP nodes in a region. If you must, they all need to be meshed with each other and monitored. Having two one-node "regions" in the same datacenter is usually easier and more reliable than meshing, at the cost of more required connections from clients in some cases. If your clients aren't mobile (battery constrained), one node regions are definitely preferred. If you really need multiple nodes in a region for HA reasons, two is sufficient.
-
Monitor your DERP servers with
cmd/derpprobe. -
If using
--verify-clients, atailscaledmust be running alongside thederper, and all clients must be visible to the derper tailscaled in the ACL. -
If using
--verify-clients, atailscaledmust also be running alongside yourderpprobe, andderpprobeneeds to use--derp-map=local. -
The firewall on the
derpershould permit TCP ports 80 and 443 and UDP port 3478. -
Only LetsEncrypt certs are rotated automatically. Other cert updates require a restart.
-
Don't use a firewall in front of
derperthat suppressesRSTs upon receiving traffic to a dead or unknown connection. -
Don't rate-limit UDP STUN packets.
-
Don't rate-limit outbound TCP traffic (only inbound).
Diagnostics
This is not a complete guide on DERP diagnostics.
Running your own DERP services requires exeprtise in multi-layer network and application diagnostics. As the DERP runs multiple protocols at multiple layers and is not a regular HTTP(s) server you will need expertise in correlative analysis to diagnose the most tricky problems. There is no "plain text" or "open" mode of operation for DERP.
-
The debug handler is accessible at URL path
/debug/. It is only accessible over localhost or from a Tailscale IP address. -
Go pprof can be accessed via the debug handler at
/debug/pprof/ -
Prometheus compatible metrics can be gathered from the debug handler at
/debug/varz. -
cmd/stuncin the Tailscale repository provides a basic tool for diagnosing issues with STUN. -
cmd/derpprobeprovides a service for monitoring DERP cluster health. -
tailscale debug derpandtailscale netcheckprovide additional client driven diagnostic information for DERP communications. -
Tailscale logs may provide insight for certain problems, such as if DERPs are unreachable or peers are regularly not reachable in their DERP home regions. There are many possible misconfiguration causes for these problems, but regular log entries are a good first indicator that there is a problem.