Ignore malformed D-Bus NameOwnerChanged signals before reading their
body. A short signal previously logged an error but continued indexing
Body[0] and Body[2], which could panic the resolved manager run loop.
Also report the malformed body length correctly and add coverage for
malformed, stopped, and restarted systemd-resolved signals.
Updates #21270
Signed-off-by: alexchang <dragoonchang@gmail.com>
Cover process attribution, loopback filtering, socket churn, transport
fallback and recovery, and namespace consistency. Reuse the existing
kernel-regression skip helper for live tests and leave namespace setup
to the caller. Probe all diagnostic handlers before tests depend on them,
skipping known capability failures such as missing UDP diagnostics.
Add warm and cold backend benchmarks with held TCP connections and
listener scaling. Assert that diagnostic polls do not silently fall back.
Updates #10430
Updates #15492
Signed-off-by: JeremiahM37 <JeremiahM37@users.noreply.github.com>
Query listening TCP and unconnected UDP sockets through inet_diag instead
of scanning the proc socket tables. Preserve process attribution and fall
back to proc on unsupported, interrupted, or invalid diagnostic dumps.
Cache missing protocol handlers as unsupported to avoid retrying every poll.
Read terminal dump flags directly: the pinned netlink library removes
NLMSG_DONE without checking NLM_F_DUMP_INTR. Keep socket creation and
recovery consistent with the proc namespace and close the poller when
its service discovery loop exits.
Include parser, request encoding, and errno classification tests.
Updates #10430
Updates #15492
Signed-off-by: JeremiahM37 <JeremiahM37@users.noreply.github.com>
The map response reader now caps a single message at 256 MiB on the wire
and 1 GiB after zstd decompression. The server-chosen uint32 size prefix
previously let a malicious control server make us allocate up to 4 GiB
before reading any body bytes, and the decoded size was unbounded, so a
small zstd frame could expand into gigabytes of JSON. A 16 MB cap has
been hit by real production traffic before, so both limits sit far above
plausible legitimate sizes. The size-prefixed read moved into a
readMapResponseMessage helper, and the newer control/tsp path already
enforced both kinds of bounds; this brings the long-poll path it
replaces in line, with more generous limits for large tailnets.
ts2021.Client.Do additionally caps every noise response body with
httpbody.LimitSize, so a malicious or buggy control server can't make us
buffer an unbounded response. Client.Do shadows the embedded
http.Client's Do method, so register, set-dns, set-device-attr,
audit-log, all DoNoiseRequest consumers (webclient, tailnet lock, SSH
actions, id-token, feature queries), and the debug CLI get the cap
without per-call-site changes, and future noise endpoints get it for
free.
The cap lives in the new util/httpbody package so other HTTP clients can
adopt the same convention: LimitSize looks the size limit up from
res.Request's context (a Response knows the Request that produced it),
falling back to DefaultMaxSize, 1 MiB, when the context carries no
override. It is like io.LimitReader except that reads past the limit
fail with an error wrapping httpbody.ErrTooLarge instead of silently
truncating, and a body of at most the limit, including one of exactly
the limit, reads back without error: the wrapper probes for EOF once the
limit is exhausted to tell an exactly-at-limit body from an oversize
one. The per-request override, httpbody.WithMaxSize, is a context key,
so transports pick it up with no API changes; LimitSizeTo applies an
explicit limit ignoring any override. Repeated LimitSize or LimitSizeTo
calls replace the previous limit rather than compounding it, so a later
call can raise or remove the limit an earlier one set.
Responses that stream an unbounded number of individually bounded
messages disable the cap with httpbody.WithMaxSize(ctx, 0): the
/machine/map long-poll and control/tsp's map session, whose messages are
already capped per-message (by readMapResponseMessage and decodeMsg, and
by tsp's framedReader and boundedReader). Their non-200 error bodies are
not message streams, so those are capped with LimitSizeTo instead.
The tailnet lock /tka/init/begin, /tka/sync/offer and /tka/affected-sigs
responses can carry per-node key signatures or missing AUMs, which at
100,000 peers reach tens of MB, so they raise the cap to 512 MiB. The
per-response io.LimitedReader decoders that silently truncated those
responses at 1 or 10 MiB are removed: the transport cap is now the
single enforcement point, and it reports oversize bodies instead of
truncating them.
The /key fetch over plain TLS switches from io.LimitReader to
httpbody.LimitSizeTo, so an oversized response reports the problem
instead of producing a confusing truncated-JSON error.
Thanks to Ben Carman for the report!
Updates tailscale/corp#48187
Reported-by: Ben Carman
Change-Id: Ibf95e1ab9e4f0d7ef8866e8c26e62ed2a514455a
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
When a controlclient receives non-keepalive netmap, it updates the controlknobs
based on the capability map of the self node. We added a new knob based on
tailcfg.NodeAttrCacheNetworkMaps in be2f554dd3, and this ensures we correctly
propagate the attribute to the knob when coming up from a cache as well.
Updates #12639
Updates tailscale/projects#27
Change-Id: I40d34053c9743757f9fddca715379fc63a9ae6a0
Signed-off-by: M. J. Fromberger <fromberger@tailscale.com>
A tailnet peer can remotely crash tailscaled by sending a DERP-sealed
disco CallMeMaybeVia message with an all-zero ServerDisco key. The
decoder accepts the zero key, and the relay manager later hands it to
DiscoPrivate.Shared, which panics on zero keys. The sender only needs
to be a relay-capable peer in the victim's netmap.
Auditing the other DiscoPrivate.Shared call sites reachable from
decoded messages turned up the same bug on the relay server side.
AllocateUDPRelayEndpointRequest.ClientDisco is attacker-chosen: one
slot must match the sender's disco key, and the other can be zero. It
flows unchecked into udprelay.Server.AllocateEndpoint, which calls
Shared on both client keys and panics in its eventbus subscriber
goroutine. AllocateEndpoint now rejects zero client keys with an error,
which its only caller already handles by logging.
Thanks to Ben Carman for the report!
Updates tailscale/corp#48187
Reported-by: Ben Carman
Change-Id: Ifc0f64d8f63270100b22c06e9f759dce624ab811
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Tests that exec a fresh CopyTo copy of the tailscale/tailscaled
binaries occasionally failed with "text file busy" on GitHub
Actions, previously worked around with the retry loop in
awaitTailscaledRunnable.
The root cause: CopyTo's Linux hardlink fast path only works while
the built binary is still linked somewhere. The kernel refuses to
hardlink an inode whose link count is zero, even via the still-open
FD, so once the building test's TempDir (and every other test's copy)
has been cleaned up, later tests silently fall through to the
byte-copy path. That path writes the new copy from the test process
itself, and if another parallel test forks a child while the write FD
is open, the child holds the inherited FD (O_CLOEXEC only closes at
exec, not at fork) and a subsequent exec of the fresh copy fails with
ETXTBSY. This is golang.org/issue/22315. On macOS and the BSDs,
CopyTo always takes the byte-copy path, so every copy races there.
Fix it the way the syscall package documents: hold ForkLock for
reading across the copy so that no fork overlaps the lifetime of the
write FD. A standalone stress program reproduced the race in 86 of
200 execs under a fork storm and in 0 of 200 with the lock held.
Updates #15868
Updates #15865
Change-Id: I8a6be72826c9073a9187d7e6c90c8733d2dadb05
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Nothing reads ipn.Notify.NetMap anymore. The previous commit removed
its runtime (non-initial) emission, and every first-party client is
also off the initial one: the Win32 and WinUI GUIs and the Apple
bridge subscribe with InitialStatus or InitialState plus peer deltas,
Android no longer uses it, and the remaining in-tree subscribers that set
NotifyInitialNetMap (sniproxy and the kube helpers) only did so to get
the initial Notify.SelfChange and discarded the netmap that tailscaled
built, encoded, and shipped for them.
Delete the Notify.NetMap field and the NotifyInitialNetMap bit. The
bit value stays reserved under the name ObsoleteNotifyInitialNetMap and
ValidateNotifyWatchOpt rejects subscriptions that set it, like the
NotifyRateLimit bit removed in the previous commit. NotifyNoNetMap
remains accepted as a no-op because shipping GUIs still set it.
The blessed way to seed a watcher's view is NotifyInitialStatus, but
it unconditionally built O(peers) status entries, which is exactly the
waste this series is deleting for watchers that only care about the
self node. Size the initial status to the subscription instead:
Status.Peer is only populated when the watcher also set
NotifyPeerChanges or NotifyPeerPatches, since only peer-delta
subscribers need a peer baseline to apply deltas to. That matches its
existing first-party users (containerboot and the WinUI GUI both pair
InitialStatus with peer bits).
Migrate sniproxy and the kube helpers to NotifyInitialStatus: they
seed from InitialStatus.Self and react to the (ungated) runtime
Notify.SelfChange messages after that, so their initial message now
carries one PeerStatus instead of a full netmap.
Also drop doc comment references to LocalClient.NetMap, a method that
does not exist; on-demand fetches go through other LocalAPI methods
such as LocalClient.Status.
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I242992a744c0ffd0be6f27e8c735aa69d5b23b5e
The WinUI GUI was the last consumer of the legacy Notify.NetMap field
on runtime (non-initial) IPN bus messages. It was converted to peer
deltas in tailscale/corp#47958, and the GUI and tailscaled ship
together as a unit on Windows, so tailscaled no longer needs to build
and emit full netmaps on the bus on any platform. Remove
goosGetsLegacyNetmapNotify and the code it gated. The initial netmap
(NotifyInitialNetMap) is unaffected; NotifyNoNetMap is now a no-op but
remains accepted for compatibility.
With no runtime netmaps left to rate limit, the NotifyRateLimit
subscription bit is meaningless, so remove the rateLimitingBusSender
machinery too. The bit value stays reserved under the name
ObsoleteNotifyRateLimit and ValidateNotifyWatchOpt now rejects
subscriptions that set it; previously it was only rejected in
combination with new-style delta bits.
Updates #12542
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ic6184ea549726de9d9d56f133aae21e53f62cbc5
The tsconnect wasm build still has osusergo and netgo since its early
6f5096fa6, copied from our linux static-linking build flags. They are
no-ops on js/wasm: there is no cgo resolver or cgo os/user
implementation to disable, so the pure Go paths are used regardless.
The omitidna and omitpemdecrypt tags (used by the wasm build and by
gocross for darwin and ios) were binary size reduction patches in our
github.com/tailscale/go fork, not upstream Go, and did not survive the
fork's per-release history reboots: omitpemdecrypt only ever existed
on the tailscale.go1.14 and tailscale.go1.15 branches, and omitidna
made it through tailscale.go1.17. Both have been unrecognized no-ops
since we moved to the go1.18 fork branch (927fc3612, March 2022, first
shipped in Tailscale 1.24.0).
Also stop passing tailscale_go explicitly: the tailscale/go fork's
cmd/go sets that tag itself as of 2026-03-31.
Updates #21250
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I10f1244f02a915943ca84e107aff8683b6e771b3
This change expands our fuzzing coverage in protocol and parsing logic. No issues discovered from this fuzzing. Wiring into oss-fuzz for continual coverage.
Updates https://github.com/tailscale/corp/issues/46608
Change-Id: I6b5218cb1103ccc5b957c512a10d87f637c4b6e5
Signed-off-by: Mike Jensen <mikej@tailscale.com>
DNATNonTailscaleTraffic in the nftables runner installs its exemption rule in the nat PREROUTING chain but matches on meta `oifname`, which routing has not yet selected at that hook. The exemption was therefore always true, so tailnet-arrived packets to the proxy's own address were DNATed to the egress target, forwarded back out tailscale0 and SNATed to the proxy's IP. This let any peer allowed to reach the proxy reach the target on all ports, bypassing tailnet ACLs.
This change matches on meta `iifname` instead, so traffic that arrived on the tun interface is exempt from the DNAT, mirroring the iptables runner's `"!" -i <tun>` semantics. Tailnet-originated packets now fall through to local delivery where the node's own ACL filters apply.
Reported by @KR-Ravindra
Fixestailscale/corp#47962
Change-Id: I1861a348b792ad4ae8e295447078a302f99d7d77
Signed-off-by: Mike Jensen <mikej@tailscale.com>
Two fixes for DNS names from a malicious control server, from Ben
Carman's security review:
dnsname.ToFQDN now rejects names containing whitespace or control
characters. ToFQDN previously checked label lengths only, so a search
domain like "evil.com\nnameserver 6.6.6.6" passed validation and was
written verbatim into /etc/resolv.conf by the direct DNS manager, where
the injected line became a real nameserver. The same shape existed on
Windows, where a CRLF in an ExtraRecords name injected lines into the
hosts file. This is deliberately not RFC 1123 hostname validation (see
the existing comment about issue 2024): labels may still contain any
byte that isn't whitespace or a control character.
dnsConfigForNetmap now drops search domains and split-DNS route suffixes
that ToFQDN rejects. It previously logged the error but appended the
zero FQDN anyway, and FQDN.WithoutTrailingDot panics on the empty FQDN
when OS DNS config is written, so one over-long domain from control
crashed tailscaled on every netmap until control sent a valid one.
Thanks to Ben Carman for the report!
Updates tailscale/corp#48187
Reported-by: Ben Carman
Change-Id: I8de40cafacfb66e2b863f097794c42f3ca5c2da9
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
State keys written by dev-set-state-store are otherwise gated by their own handlers, for example serve-config requires a local admin before storing a _serve/<profile-id> key.
Writing such a key directly through dev-set-state-store skipped that check. Require`IsLocalAdmin` in the handler, the same check serve-config performs.
Credit to @johnnymiranda for reporting this issue.
Fixestailscale/corp#47886
Change-Id: Ie82f961b016f78895793d801929a4fa11ebf7fd8
Signed-off-by: Mike Jensen <mikej@tailscale.com>
Some customers rely on the ability for 4via6 subnet routers to expose
loopback and/or link-local addresses. In situations where the
administrator has deemed these to be safe, accept a list of allowed
addresses in an environment variable named TS_4VIA6_ALLOW_LOCAL.
Fixes#21019
Change-Id: I7b3fe863696ee67aa352c4bd11edcbc8836920e3
Signed-off-by: Adrian Dewhurst <adrian@tailscale.com>
Previously retry-after support was added to register requests in #19403,
in order to have more fine tuned client/control interactions in cases of
rate limiting.
This change expands that support to map sessions as well.
Updates tailscale/corp#47299
Signed-off-by: Evan Lowry <evan@tailscale.com>
Prior to 85bb5f8 mapSession was running directly as part of the
controlClient. That was changed previously to let discoKeys to flow into
the mapSession for TSMP learned keys, using the mapSession for filtering
of keys.
Since moving the TSMP key learning directly into the userspace engine,
and introducing the notion of two active disco keys for an enpoint, we
no longer need mapSession to run in a separate goroutine and have
separate paths of entry for data.
This commit effectively reverts the last changes that was previously
made in the controlClient to support TSMP.
Updates #20590
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
We have visibility into what the gocacheprog is doing via its stats in
the github jobs, and its session on the server side, so this is just a
spammy log line that doesn't print anything very interesting. In
particular, cigocacher is started once for each package when you pass a
list of packages to testwrapper, so it prints many times for jobs where
we filter to a certain set of packages.
Updates #cleanup
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
This is a follow-up to #21029 (aa2681ac5f) to make it a bit stricter
and not cache DNS results until they've passed TLS cert validation,
to weed out DNS servers that are lying (like captive portals).
Because this is done via dnscache.TLSDialer we only catch the control
connection, but that's fine. That's all we need to come back alive
if DNS was down because real system DNS is itself over Tailscale.
The DERP connections should come via IPv4/IPv6 fields in the DERPMap.
And the logging connection isn't important; it'll buffer and catch up
later as needed when DNS is back up.
Updates #21028
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I75f176a0222f04ed52c9de1247deeed9b911f92c
* net/dns: export OSConfigurationReadWarnable
A natlab vmtest checks that a node is not reporting this warning. Exporting
the Warnable lets the test take the warning's text from it, instead of
keeping a copy of the wording that could stop matching without failing.
Updates #20825
Updates tailscale/corp#44793
Signed-off-by: Brendan Creane <bcreane@gmail.com>
* tstest/natlab/vmtest: add VM coverage for the openresolv DNS backend
dnsMode() picks one of five Linux DNS backends, and natlab could provision
only systemd-resolved and direct. Add a DNSOpenresolv mode and a VM test for
it, so the backend behind #20825 is covered.
No cloud image ships openresolv and a guest cannot download it, so its two
source files are vendored under testdata and installed with cloud-init's
write_files. openresolv's build is a set of sed substitutions with nothing to
compile, so resolvconf.go does the substitutions in process.
Updates #20825
Updates tailscale/corp#44793
Signed-off-by: Brendan Creane <bcreane@gmail.com>
---------
Signed-off-by: Brendan Creane <bcreane@gmail.com>
isAddressValid rejected all non-masquerade destination addresses
whenever any masquerade address was set for the peer. With a v6-only
masquerade pair, the peer's client still dials the v4 (native, not
masqueraded) peerapi URL, so every fresh peerapi connection got a 403.
The bug was masked by HTTP keep-alive: peerNode is snapshotted per
connection, so connections established before the masquerade was
configured kept validating against the old node view. It surfaced when
a newer tailscale/go toolchain started closing idle netstack
connections, forcing fresh peerapi connections and failing the
TestNATPing v6=true NAT subtests with 403s.
A masquerade address for one family says nothing about the other
family, so require a masquerade address match only for the address
family it applies to, and fall through to the self-addresses check for
the other family.
Fixes#21194
Change-Id: Idd2ed82b81e805b47b6aa6af4b5938984c6fa208
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Route packets to dedicated WireGuard, host, and loopback queues when
gVisor writes them. Drain each queue in a separate goroutine, preparing
the outbound path for batching and multiqueue WireGuard delivery.
Updates tailscale/corp#37878
Signed-off-by: Jordan Whited <jordan@tailscale.com>
TestUDPConcurrent closed its UDP connection as soon as the send loop
finished, and then required that at least one reply had arrived. With
GOMAXPROCS=1 the send loop runs to completion before the proxy's relay
goroutines are scheduled at all, so the close discarded every response.
The test failed on all 20 runs of "go test -race -count=20 -cpu=1".
Wait for the first reply, with a 10 second timeout, before closing the
connection.
This also makes the test catch the race it was written for more often.
With the fix in 0301c7493 reverted, "-cpu=2" reports the race where it
previously reported none.
Fixes#21182
Signed-off-by: Brendan Creane <bcreane@gmail.com>
Serve copied the raw Server.Logf field into each Conn, so the UDP error
paths called a nil func instead of falling back to log.Printf. Conn
already holds the Server, so drop the duplicated field and log through
the server's method.
Fixes#21047
Signed-off-by: Brendan Creane <bcreane@gmail.com>
handleUDPRequest recorded the client's UDP source address in
Conn.udpClientAddr on the goroutine that reads from the client.
handleUDPResponse read the field to address the responses, and it runs
on a separate goroutine per target. Nothing synchronized the two.
Guard the field with syncs.MutexValue. The most recently written
address still wins, which is what the code did before.
TestUDPConcurrent keeps datagrams in flight to four targets at once.
Without the fix it reports the race on every run.
Fixes#21048
Signed-off-by: Brendan Creane <bcreane@gmail.com>
Android has no /var, so DefaultTailscaledSocket fell through to the
cwd-relative "tailscaled.sock": the daemon created its socket wherever
it was started and the CLI looked for one wherever it was started, and
the two only agreed by coincidence (reproduced on Termux in #21161).
Return os.TempDir()/tailscaled.sock on Android - the shared per-app
temp directory, following the same pattern as the existing QNAP /tmp
default - so daemon and CLI agree on an absolute path.
Fixes#21161
Signed-off-by: breken <hi@breken.ai>
As of Linux 6.2 (torvalds/linux@f1f1f25699),
the stat size of the /proc/self/fd directory is the number of open file
descriptors, so CurrentFDs no longer needs to walk the directory on
modern kernels. The walk is O(n) in the number of open file descriptors,
which adds up on servers like derper that hold many connections and
report this metric on every scrape: at 10k open fds, the walk takes
about 2.2ms while an fstat takes a constant 300ns.
We fstat a directory file descriptor held open for the process lifetime
rather than calling stat with the path, because syscall.Stat copies the
path string to a NUL-terminated buffer on every call and the metric
must remain allocation-free. Kernels older than 6.2 report a size of
zero, in which case we fall back to the directory walk.
Updates #2784
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I1742db2ae1938c77f9b7d137b167ab4a2b6b6167
Adds an opt-in, in-memory aggregator of recent connection-rejection
events (TSMP rejects received from peers, outbound TSMP rejects we emit
on ACL-blocked inbound flows, and pendopen timeouts) keyed by
(direction, proto, peer-address, reason). The aggregated data is exposed
over a new debug-rejects LocalAPI endpoint and a GET /debug/rejects c2n
endpoint, intended for future GUI/CLI consumption when diagnosing why a
connection failed.
Architecture:
- net/connreject holds the data types and a per-LocalBackend
Aggregator (LRU-bounded, default 256 entries on desktop / 32 on
mobile, per direction).
- feature/connreject is a self-registering ipnext.Extension that owns
one Aggregator per LocalBackend, installs note callbacks on the
tundev and engine, subscribes to OnSelfChange to flip the runtime
gate, and serves the LocalAPI/c2n endpoints.
- wgengine.Engine and *tstun.Wrapper each gain a SetConnRejectNote
setter; data-plane sites use a single atomic.Pointer load + nil
check, so the cost when no consumer is installed is one MOV.
Gating:
- Compile-time: ts_omit_connreject build tag (standard
feature/buildfeatures + condregister plumbing). Trims ~41 KB.
- Runtime: nodecap.ConnReject node attribute, off by default
at the control plane. May be removed once the feature is enabled
by default.
Updates CapabilityVersion to 146 (clients understand nodecap.ConnReject
and can serve GET /debug/rejects).
Adds Proto/Src/Dst accessors on flowtrack.Tuple (used by pendopen to
construct events without exposing the tuple's internals to the
aggregator).
Updates #1094
Updates #14802
Change-Id: I83e8f24a7e66fa2d158d128bd25fbe851134941b
Signed-off-by: James Tucker <james@tailscale.com>
Add a new modular dnsresolvecache feature that records every successful
DNS resolution from net/dnscache as a JSON file per hostname in
$statedir/dns-cache/, so a later boot with misconfigured DNS can still
find last-known-good IPs for critical hostnames like the control plane.
Files are rewritten only when their contents change, so a file's
modification time records when the answer last changed.
When regular DNS resolution fails, the disk cache is now consulted
before the DERP-based bootstrap DNS in net/dnsfallback. This is the
first step toward removing the DERP-based mechanism: new clientmetrics
(dnscache_disk_fallback_hit, dnscache_disk_fallback_miss,
dnscache_derp_fallback_ok, dnscache_derp_fallback_dial_ok) will tell us
when the DERP path no longer fires in the fleet and can be deleted.
The feature is linked into tailscaled by default (omittable with
ts_omit_dnsresolvecache) and is not included in tsnet.
Updates #21028
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I59228cbf68e3b48dfb1215cdd12bd8166ab14034
Instead of opening all DNS queries with conn25, only permit clients to
access domains of the apps that they have permission to use. Previous
changes ensure that clients report which app they are making a DNS query
for, simplifying the check.
Updates tailscale/corp#40076
Updates tailscale/corp#47585
Change-Id: I40fee220b3f190b2a0db9b3e6cc79f323ef16b73
Signed-off-by: Adrian Dewhurst <adrian@tailscale.com>
Android denies app UIDs both NETLINK_ROUTE (golang/go#40569, #2293)
and /proc/net, so net.Interfaces always fails and netmon.New errors
out before a standalone binary can do anything. The Android app solves
this from Java via netmon.RegisterInterfaceGetter, but raw binaries
run under Termux or a rooted shell have no Java to lean on. This is
the second half of #21129, following the androiddns feature.
I added a netmon fallback hook, consulted only when no interface
getter was registered and net.Interfaces failed, and a new androidbin
feature (ts_omit_androidbin) that implements it: report a single
synthetic interface whose v4 and v6 addresses come from asking the
kernel to route an outbound UDP socket, which sends no packets and is
permitted in the app sandbox. That's enough for magicsock to discover
local endpoints. The fallback also requires runtime evidence of
Android (GOOS=android, or /dev/__properties__ existing for GOOS=linux
binaries running under an Android kernel), so it's inert on regular
Linux.
The androidbin feature also blank imports androiddns, and fixes a
third gap I found while testing: GOOS=linux binaries on Android have
an empty CA root pool, because Go's unix root loader doesn't know
Android's /system/etc/security/cacerts (the GOOS=android loader
does), so all TLS verification fails. On Android it points
SSL_CERT_DIR there unless the user already set it, as Termux's
ca-certificates package does.
The feature is on by default in tailscaled builds on Linux and
Android via condregister, and deliberately not linked into tsnet by
default; tsnet apps and other programs opt in with a blank import of
tailscale.com/feature/androidbin.
I verified on an Android 13 emulator with SELinux enforcing, running
GOOS=linux static binaries under the app UID (run-as), where
net.Interfaces fails with the exact netlinkrib permission denial from
the issue: netmon.New succeeds with the synthetic interface, and a
tailcat binary importing this feature does DNS via dnsproxyd, fetches
its DERP map over TLS using the Android cert store, completes STUN,
selects a DERP region, and prints its address.
Updates #21129
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: If7dbcbb825ecd24bcf6d9d64b1e334dd00700d51
Previously the DERP handler served each connection for its lifetime
on its hijacked connection's net/http handler goroutine. That
goroutine's conn.serve stack frame kept the dead HTTP/1 server state
reachable for the whole DERP connection: the http.conn and its 4KB
bufio.Writer (hijack hands over c.bufw but conn.serve still references
it, so derpserver returning it to its flush pool never made it
collectable), the 4KB bufio.Reader, the upgrade *http.Request with its
parsed headers, and the request context chain. The goroutine also kept
the stack growth from the TLS handshake and HTTP request parsing.
Instead, hand the connection off to a new goroutine and return from
the handler (ala tailscale/corp@dc09e27aef), letting all the HTTP
upgrade state be collected. Give Accept a smaller 1KB frame reader,
draining and releasing the hijacked reader if it contains buffered
bytes from a fast-start client, and a nil bufio.Writer so writes go
through pooled buffers held only for the duration of a write instead
of a per-connection buffer.
Because the handler now returns at handoff time, cmd/derper's
gauge_derper_tls_active_version decrement can no longer be deferred
to handler return: intercept Hijack in the TLS metrics wrapper and,
for hijacked connections (DERP, its WebSocket flavor, and CONNECT),
decrement the gauge once when the hijacked connection closes,
restoring the gauge's connection-lifetime semantics. Teach
derpserver's TCP RTT stats to unwrap the close-hook conn so they
still find the underlying *net.TCPConn.
Also soften the UntypedHexString deprecation notices in types/key to
warnings: the untyped hex string format is the DERP wire protocol's
key encoding, so these call sites are legitimate and permanent, and
a Deprecated marker just makes them light up in editors and linters.
The cautionary text about the format's risks remains.
Measured with 100,000 idle TLS DERP connections on linux/amd64:
standing memory drops from 55.6KB to 32.6KB per connection (-41%).
Updates #21064
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: If05a0c6ea79134807e9e8872861db216
The PeerRelay resource hardcoded UDP port 41641 in the generated
Service, the tailscaled config, status.endpoints and the advertised
static endpoints. That breaks deployments where the load balancer
address or port is not what peers can actually reach, e.g. behind NAT.
Add spec.service.port to choose the UDP port, and spec.staticEndpoints
to advertise extra address:port pairs alongside the discovered load
balancer endpoints. A replica whose only endpoint is a static one
counts as addressed, and a static entry's port wins when it names an
address the load balancer already provides.
The e2e tests now supply static endpoints, which lets them assert the
PeerRelayReady condition goes True in kind, where no cloud controller
ever gives the LoadBalancer Services an address.
Fixes#20821
Change-Id: I463012cd447c81c2849fa653fa18eb82662aaf4f
Signed-off-by: David Bond <davidsbond93@gmail.com>
Move the container image build logic out of the e2e test setup into
cmd/k8s-operator/e2e/internal/build, with a thin CLI wrapper at
cmd/k8s-operator/e2e/build, so CI can publish images once per commit and
fan out into multiple test jobs that consume them. The command skips
images that already exist in the registry, so a retried run doesn't
fail if the registry is immutable.
Also adds support for the test harness leveraging workload identity
federation credentials so we can use GitHub's token issuer for the test
code itself, and AWS' OIDC provider for the operator, and avoid using
any secrets in CI.
Updates tailscale/corp#46577
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
Android doesn't have /etc/resolv.conf. This causes problems for people
running GOOS=linux binaries (or GOOS=android binaries without cgo, so
they don't use Android's bionic libc) in Termux, adb shell, etc.
(Go binaries built with cgo use bionic on Android: golang/go#10714)
16 years ago when I was on the Android team I added a system-wide DNS
cache (dnsproxyd) and made bionic query that, so each Android app
wasn't doing its own DNS resolution. That interface was never meant to
be stable, and I thought that code would be surely dead by now 16
years later, but apparently it lives on, and is more stable now: both
empirically (time, ossification?), and because of how Android's split system
updates work nowadays, the dnsproxyd lives on the other side of bionic,
so they seem to keep it pretty stable. The old bionic<->dnsproxyd APIs
I added 16 years ago are still there, but 8 years ago it got some additional
APIs to query by a DNS packet instead.
So use it! If we find ourselves on Android and without libc access
(and because we don't want to pull in ebitengine/purego with all its
side effects), just query the DNS server like bionic does.
This can be disabled in Linux binaries with ts_omit_androiddns.
Old links:
LineageOS/android_system_netd@007e987feehttps://android.googlesource.com/platform/system/netd/+/007e987fee7e815e0c4bc820f434a632b7a69a9d
("DNS proxy thread in netd.")
aosp-mirror/platform_bionic@a1dbf0b453https://android.googlesource.com/platform/bionic/+/a1dbf0b453801620565e5911f354f82706b0200d
("DNS proxy: the start. proxies getaddrinfo calls.")
Back then I found it cleaner to proxy at the getaddrinfo level rather
than speak in terms of DNS packets. The raw-packet resnsend command I
use here came eight years later, added in November 2018 for Android
10's android_res_nsend NDK API:
LineageOS/android_system_netd@c0c818f448https://android.googlesource.com/platform/system/netd/+/c0c818f448efa90ab1f9b1733fb86c5e22fb894c
("Add resNetworkSend cmd in DnsProxyListener")
Android 10 (codename Q, API level 29, released September 2019) is
therefore the minimum OS version for this to work.
I verified this against the DnsResolver module on an Android 13
emulator with SELinux enforcing, from the shell UID, with both a pure
Go GOOS=android binary and a static GOOS=linux binary: raw queries,
NXDOMAIN handling, the runtime Android detection, and a tailcat binary
reaching DERP with lookups visible in the daemon's logcat output, some
served from my 2010 DNS cache.
Updates #21129
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I0d63763e255a077e4e5745b3e64ba0d78dab6d69
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
A full netmap carries only the profiles of users with a currently
visible peer (netmapForResponse), and nodeBackend replaces its live
profile set wholesale on every full netmap install. A full netmap that
arrives while a user has no visible peer therefore drops that user's
profile downstream.
When a peer of that user later returns as an incremental upsert,
control does not resend the profile, because MapResponse.UserProfiles
has carried only new or updated profiles since mapver 5. The upsert
indexes the node by address and key, so WireGuard admits its traffic,
but WhoIs then fails one step later at the user profile lookup,
surfacing as "peerapi: unknown peer" until the next full netmap. It is
a second, independent cause of the symptom fixed by the recent index
eviction change.
mapSession.lastUserProfile holds the profile the whole time, so when
handling a response incrementally, also deliver the profiles of
upserted peers' users (and sharers) from that store, before the
mutations that reference them.
Updates tailscale/corp#47435
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I9c4b2f6a8e0d47d3b1a5c2e7f4098d61b23a7c50
Refresh the expiry timer netmap from the live peer state before
reinstalling it, preventing delta updates from being rolled back.
Updates tailscale/corp#47686
Change-Id: Idc738acea82bab5a8ba772084a41e55b38a06bcc
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
When applying netmap deltas, nodeBackend evicted its index entries
(nodeByAddr, nodeByKey, nodeByWGString, nodeByStableID, nodeByName)
derived from a node's last-known value without checking that the entry
still pointed at that node. Control can reassign a churning ephemeral
peer's Tailscale IP (or MagicDNS name) to a newer peer and deliver the
new peer's upsert before the old peer's removal, either in an earlier
MapResponse or reordered within one batch by the NodeID sort in
netmap.MutationsFromMapResponse. The removal then wiped the new
owner's entry.
The peers map itself stayed correct in every ordering, so WireGuard
kept the peer and handshakes succeeded, but WhoIs lookups by IP failed
until the next full netmap rebuilt the indexes. On App Connectors that
surfaced as "peerapi: unknown peer" and refused DNS connections from
affected clients, with a toggle of Tailscale (forcing a full netmap)
as the only recovery.
Make every index eviction conditional on the entry still mapping to
the node being removed or replaced, and add a regression test covering
the cross-batch, intra-batch, and upsert-eviction orderings.
Also add an end-to-end test in tstest/integration showing that a
MapResponse reusing an address is handled incrementally rather than as
a full netmap, and that LocalBackend.WhoIs still resolves the reused
address afterwards, which is the lookup PeerAPI makes before it
accepts a connection.
Updates tailscale/corp#47435
Co-authored-by: Brendan Creane <bcreane@gmail.com>
Signed-off-by: Brendan Creane <bcreane@gmail.com>
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3f8c2a9d41e07b6a5cd2e94f78b013c6ad2f5e91
This commit bumps the wireguard-go dependency to incorporate changes to
the packet memory model and the tun.Device.Read and conn.ReceiveFunc I/O
interfaces. It updates their implementations accordingly.
These changes improve throughput in all measured benchmarks and reduce
peak RSS in six of eight cases. The two regressions will be addressed in
a follow-up commit that reduces peak RSS below the baseline measured at
1e69418. That work is kept separate to simplify review.
The following throughput and peak RSS benchmarks were performed with
iperf3 between two Intel i5-12400 nodes running Ubuntu 24.04 (Linux 6.8).
The UDP benchmarks did not use UDP GSO on the sender, so they were
roughly equivalent to single packet I/O through wireguard-go.
TCP/1 signifies one TCP stream; TCP/128 signifies 128 parallel TCP
streams.
Throughput (Mb/s)
Test 1e69418 After Change
TCP/1 10,371 11,354 +9.5%
TCP/128 7,886 8,404 +6.6%
UDP/1 2,111 2,853 +35.1%
UDP/128 1,747 2,235 +28.0%
Peak memory (VmHWM, kB)
Test Side 1e69418 After Change
TCP/1 TX 98,240 52,596 -46.5%
RX 287,748 73,384 -74.5%
TCP/128 TX 101,196 52,812 -47.8%
RX 290,420 63,620 -78.1%
UDP/1 TX 58,864 160,840 +173.2%
RX 137,516 49,900 -63.7%
UDP/128 TX 66,148 116,096 +75.5%
RX 154,384 56,556 -63.4%
Updates tailscale/corp#46716
Updates tailscale/corp#22467
Updates tailscale/corp#36989
Updates tailscale/corp#37878
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Rename the tailcat job added in 6c06e00ad to
heads-up-hypothetical-tailcat-build-using-this-commit and print an
explanation on failure: the job builds tailcat against tailscale.com at
this commit rather than the version tailcat pins, so a failure doesn't
break tailcat until it bumps its dependency. It only matters if the API
or behavior breakage is unexpected. People were misreading failures as
something more urgent.
Updates tailscale/corp#24454
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3f8a1c92d47e60b5a2e9c14f7d08b361
alexchang