James Tucker
f55fd4e9e4
Move the goamd64_capable/goamd64_compiled logic out of tsweb/varz into a new util/cpucaps package so that a future hostinfo extension can share it, and extend it to all supported platforms. The host GOAMD64 level is now computed from CPUID feature bits via x/sys/cpu rather than by parsing /proc/cpuinfo, which removes the linux/amd64 restriction and all file I/O. CPUID is also what the Go runtime checks at startup, so it directly answers the question the metric asks: could this machine run a binary built at a higher level. Four flags required by the runtime checks have no x/sys/cpu field: LAHF/SAHF (v2) and MOVBE, LZCNT and F16C (v3). We report a level when all of its other flags are present. That is exact on real hardware: LAHF/SAHF exists on every 64-bit CPU that also has SSE4.2, and MOVBE/LZCNT/F16C exist on every CPU that also has AVX2, BMI2 and FMA (Haswell and Zen onwards). A linux/amd64 test cross-checks the CPUID result against the exact /proc/cpuinfo flag lists, so a test run on any machine where the assumption fails, or where the kernel's view diverges from CPUID (e.g. clearcpuid=), reports the difference. Add a Caps bitmask of the CPU capabilities that Go's runtime and standard library dispatch on, normalized across amd64 and arm64: AES, CLMUL/PMULL, SHA2, SHA512, CRC32, ADX and LSE, plus the vector capabilities not implied by the GOAMD64 level: AVX, AVX2, AVX512 (the v4 F+BW+CD+DQ+VL subset), VAES, VPCLMULQDQ and GFNI. Detection is lazy and reads only x/sys/cpu globals and build info; there is no init-time work and no new dependency. Coverage includes all key client platforms: linux, windows, darwin, freebsd, android (via the linux auxv/MRS path in x/sys/cpu) and ios (via the darwin sysctl path). x/sys/cpu has no arm64 detection on FreeBSD, so read ID_AA64ISAR0_EL1 directly; the FreeBSD kernel emulates EL0 reads of the ID registers, and Go's runtime already depends on that emulation on this platform. The whole feature is behind a new cpucaps feature tag. Building with ts_omit_cpucaps removes the varz metrics and all detection code following the tka stub pattern: the detection functions compile to stubs returning zero inside the package itself, so future importers cannot reintroduce the code by forgetting a build tag, and the Caps type stays available for decoding recorded values. A tailscaled dependency test enforces the omission, and the generated buildfeatures.HasCPUCaps const is available for the later hostinfo extension. tsweb/varz metric names are unchanged. gauge_goamd64_capable values only change on machines where /proc/cpuinfo disagrees with CPUID. Updates #21002 Signed-off-by: James Tucker <james@tailscale.com>
DERP
This is the code for the Tailscale DERP server.
In general, you should not need to or want to run this code. The overwhelming majority of Tailscale users (both individuals and companies) do not.
In the happy path, Tailscale establishes direct connections between peers and data plane traffic flows directly between them, without using DERP for more than acting as a low bandwidth side channel to bootstrap the NAT traversal. If you find yourself wanting DERP for more bandwidth, the real problem is usually the network configuration of your Tailscale node(s), making sure that Tailscale can get direction connections via some mechanism.
If you've decided or been advised to run your own derper, then read on.
Caveats
-
Node sharing and other cross-Tailnet features don't work when using custom DERP servers.
-
DERP servers only see encrypted WireGuard packets and thus are not useful for network-level debugging.
-
The Tailscale control plane does certain geo-level steering features and optimizations that are not available when using custom DERP servers.
Guide to running cmd/derper
-
You must build and update the
cmd/derperbinary yourself. There are no packages. Usego install tailscale.com/cmd/derper@latestwith the latest version of Go. You should update this binary approximately as regularly as you update Tailscale nodes. If using--verify-clients, thederperbinary andtailscaledbinary on the machine must be built from the same git revision. (It might work otherwise, but they're developed and only tested together.) -
The DERP protocol does a protocol switch inside TLS from HTTP to a custom bidirectional binary protocol. It is thus incompatible with many HTTP proxies. Do not put
derperbehind another HTTP proxy. -
The
tailscaledclient does its own selection of the fastest/nearest DERP server based on latency measurements. Do not putderperbehind a global load balancer. -
DERP servers should ideally have both a static IPv4 and static IPv6 address. Both of those should be listed in the DERP map so the client doesn't need to rely on its DNS which might be broken and dependent on DERP to get back up.
-
A DERP server should not share an IP address with any other DERP server.
-
Avoid having multiple DERP nodes in a region. If you must, they all need to be meshed with each other and monitored. Having two one-node "regions" in the same datacenter is usually easier and more reliable than meshing, at the cost of more required connections from clients in some cases. If your clients aren't mobile (battery constrained), one node regions are definitely preferred. If you really need multiple nodes in a region for HA reasons, two is sufficient.
-
Monitor your DERP servers with
cmd/derpprobe. -
If using
--verify-clients, atailscaledmust be running alongside thederper, and all clients must be visible to the derper tailscaled in the ACL. -
If using
--verify-clients, atailscaledmust also be running alongside yourderpprobe, andderpprobeneeds to use--derp-map=local. -
The firewall on the
derpershould permit TCP ports 80 and 443 and UDP port 3478. -
Only LetsEncrypt certs are rotated automatically. Other cert updates require a restart.
-
Don't use a firewall in front of
derperthat suppressesRSTs upon receiving traffic to a dead or unknown connection. -
Don't rate-limit UDP STUN packets.
-
Don't rate-limit outbound TCP traffic (only inbound).
Diagnostics
This is not a complete guide on DERP diagnostics.
Running your own DERP services requires exeprtise in multi-layer network and application diagnostics. As the DERP runs multiple protocols at multiple layers and is not a regular HTTP(s) server you will need expertise in correlative analysis to diagnose the most tricky problems. There is no "plain text" or "open" mode of operation for DERP.
-
The debug handler is accessible at URL path
/debug/. It is only accessible over localhost or from a Tailscale IP address. -
Go pprof can be accessed via the debug handler at
/debug/pprof/ -
Prometheus compatible metrics can be gathered from the debug handler at
/debug/varz. -
cmd/stuncin the Tailscale repository provides a basic tool for diagnosing issues with STUN. -
cmd/derpprobeprovides a service for monitoring DERP cluster health. -
tailscale debug derpandtailscale netcheckprovide additional client driven diagnostic information for DERP communications. -
Tailscale logs may provide insight for certain problems, such as if DERPs are unreachable or peers are regularly not reachable in their DERP home regions. There are many possible misconfiguration causes for these problems, but regular log entries are a good first indicator that there is a problem.