Brad Fitzpatrick
8e1fbc4740
Previously the DERP handler served each connection for its lifetime on its hijacked connection's net/http handler goroutine. That goroutine's conn.serve stack frame kept the dead HTTP/1 server state reachable for the whole DERP connection: the http.conn and its 4KB bufio.Writer (hijack hands over c.bufw but conn.serve still references it, so derpserver returning it to its flush pool never made it collectable), the 4KB bufio.Reader, the upgrade *http.Request with its parsed headers, and the request context chain. The goroutine also kept the stack growth from the TLS handshake and HTTP request parsing. Instead, hand the connection off to a new goroutine and return from the handler (ala tailscale/corp@dc09e27aef), letting all the HTTP upgrade state be collected. Give Accept a smaller 1KB frame reader, draining and releasing the hijacked reader if it contains buffered bytes from a fast-start client, and a nil bufio.Writer so writes go through pooled buffers held only for the duration of a write instead of a per-connection buffer. Because the handler now returns at handoff time, cmd/derper's gauge_derper_tls_active_version decrement can no longer be deferred to handler return: intercept Hijack in the TLS metrics wrapper and, for hijacked connections (DERP, its WebSocket flavor, and CONNECT), decrement the gauge once when the hijacked connection closes, restoring the gauge's connection-lifetime semantics. Teach derpserver's TCP RTT stats to unwrap the close-hook conn so they still find the underlying *net.TCPConn. Also soften the UntypedHexString deprecation notices in types/key to warnings: the untyped hex string format is the DERP wire protocol's key encoding, so these call sites are legitimate and permanent, and a Deprecated marker just makes them light up in editors and linters. The cautionary text about the format's risks remains. Measured with 100,000 idle TLS DERP connections on linux/amd64: standing memory drops from 55.6KB to 32.6KB per connection (-41%). Updates #21064 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> Change-Id: If05a0c6ea79134807e9e8872861db216
DERP
This directory (and subdirectories) contain the DERP code. The server itself is
in ../cmd/derper.
DERP is a packet relay system (client and servers) where peers are addressed using WireGuard public keys instead of IP addresses.
It relays two types of packets:
-
"Disco" discovery messages (see
../disco) as the a side channel during NAT traversal. -
Encrypted WireGuard packets as the fallback of last resort when UDP is blocked or NAT traversal fails.
DERP Map
Each client receives a "DERP Map" from the coordination server describing the DERP servers the client should try to use.
The client picks its home "DERP home" based on latency. This is done to keep costs low by avoid using cloud load balancers (pricey) or anycast, which would necessarily require server-side routing between DERP regions.
Clients pick their DERP home and report it to the coordination server which shares it to all the peers in the tailnet. When a peer wants to send a packet and it doesn't already have a WireGuard session open, it sends disco messages (some direct, and some over DERP), trying to do the NAT traversal. The client will make connections to multiple DERP regions as needed. Only the DERP home region connection needs to be alive forever.
DERP Regions
Tailscale runs 1 or more DERP nodes (instances of cmd/derper) in various
geographic regions to make sure users have low latency to their DERP home.
Regions generally have multiple nodes per region "meshed" (routing to each
other) together for redundancy: it allows for cloud failures or upgrades without
kicking users out to a higher latency region. Instead, clients will reconnect to
the next node in the region. Each node in the region is required to be meshed
with every other node in the region and forward packets to the other nodes in
the region. Packets are forwarded only one hop within the region. There is no
routing between regions. The assumption is that the mesh TCP connections are
over a VPC that's very fast, low latency, and not charged per byte. The
coordination server assigns the list of nodes in a region as a function of the
tailnet, so all nodes within a tailnet should generally be on the same node and
not require forwarding. Only after a failure do clients of a particular tailnet
get split between nodes in a region and require inter-node forwarding. But over
time it balances back out. There's also an admin-only DERP frame type to force
close the TCP connection of a particular client to force them to reconnect to
their primary if the operator wants to force things to balance out sooner.
(Using the (*derphttp.Client).ClosePeer method, as used by Tailscale's
internal rarely-used cmd/derpprune maintenance tool)
We generally run a minimum of three nodes in a region not for quorum reasons (there's no voting) but just because two is too uncomfortably few for cascading failure reasons: if you're running two nodes at 51% load (CPU, memory, etc) and then one fails, that makes the second one fail. With three or more nodes, you can run each node a bit hotter.