mirror of
https://github.com/tailscale/tailscale.git
synced 2026-07-14 17:43:03 -04:00
125fd88c30
FreeBSD guests downloaded their test binaries from vnet's
files.tailscale VIP at roughly 250 kB/s in CI, and transfers sometimes
wedged outright for many minutes, which is why TestSubnetRouterFreeBSD
timed out in about a third of its runs. Locally the same path moves
data at 100+ MB/s, so the problem was never CPU; it was TCP behavior
under two independent constraints, both diagnosed with a new
throughput harness (TestVnetPerfFreeBSDDownload), a VNET_TCP_DEBUG
endpoint sampler, and pcaps:
First, throughput is capped at receive-window/RTT. FreeBSD starts its
receive window at 64 kB and autoscales it in slow 16 kB steps, and on
an oversubscribed CI runner the effective RTT of the userspace vnet
data path reaches hundreds of milliseconds, giving almost exactly the
observed 250 kB/s. Fix: raise the FreeBSD guest's TCP buffer sysctls
in cloud-init before the downloads, and raise netstack's receive
buffer sizing for the reverse (upload) direction.
Second, the outright wedge: when netstack bursts more data than the
QEMU socket plus the guest's virtio RX ring can absorb, a wide swath
of segments is dropped downstream of vnet, and netstack's loss
recovery then crawls, retransmitting one or two segments per 200 ms
RTO for minutes at a time (a 33 MB transfer was observed taking 526
seconds against an otherwise idle receiver). Rather than depending on
recovery from mass loss, make the path effectively lossless by keeping
the maximum in-flight data (the 1 MB netstack send buffer) below the
downstream buffering: grow the guests' virtio RX rings from 256 to
1024 descriptors, enlarge the vnet-QEMU unix socket buffers, and grow
the netstack link endpoint queue from 512 to 4096 packets so a send
burst can't overflow it.
Also fixed along the way, found while chasing the above:
* pcapWriter fsync'd after every packet, serializing all traffic
behind disk writes when a test enables pcap; a pcap-enabled run
was capped at about 290 kB/s. Keep the per-packet Flush but drop
the per-packet fsync.
* Traffic originating from vnet's own netstack (control plane,
DERP, file servers) bypassed conditionedWrite, so SetLatency and
SetPacketLoss silently didn't apply to it.
* writeEthernetFrameToVM held one global mutex (and a shared
scratch buffer) across writes to all VMs, so one guest slow to
drain its socket stalled traffic to every VM on the server. The
write lock is now per-VM-connection.
TestSubnetRouterFreeBSD now passes locally in 31s (down from 4.5
minutes), still passes with the vnet simulating a 100 ms RTT
(downloads at 2-8 MB/s, previously 250-600 kB/s), and passes in 65s
with KVM disabled while pinned to two host CPUs, a harsher environment
than the CI runners. The benchmark test is opt-in via
--run-perf-tests (in addition to --run-vm-tests) so CI doesn't spend
a matrix job re-measuring it on every run. VMTEST_NO_KVM=1
forces TCG for reproducing slow-host behavior.
Fixes tailscale/corp#44805
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I1a7945a7e9c7d083b0ea2a3530eda0e9757dff18