The test resumed a live session, so notifications already
on the wire (notably, spurious health change notifications)
could arrive ahead of the initial status.
In this PR, we add LocalBackend.SuspendWatchNotifications,
which suspends a session as if it had fallen behind,
and update the test to suspend the session and drain
pending notifications before resuming it.
Updates tailscale/corp#51067
Updates tailscale/corp#50256
Updates tailscale/corp#50907
Signed-off-by: Nick Khyl <nickk@tailscale.com>
Since edcc2c94, tailscaled closes the watch when a subscriber falls
behind. That doesn't work well for GUIs where closing the watch has side
effects. On Windows, the last client disconnecting can reset the backend
state unless unattended or always-on mode is enabled. On Android and
Apple platforms, nothing restarts a watch that ended this way, so the
UI silently stops updating.
Add ipn.NotifySuspendWhenBehind. A watcher that sets it stays connected
when it falls behind, but gets a Notify with Suspended set and nothing
else until the client resumes the session via the new resume-ipn-bus
LocalAPI endpoint or LocalBackend.ResumeWatchNotifications. Resuming
replays the initial Notify and new notifications follow from there.
Updates tailscale/corp#51067
Updates tailscale/corp#50256
Updates tailscale/corp#50907
Signed-off-by: Nick Khyl <nickk@tailscale.com>
openresolv before 3.13 exits 1 from "resolvconf -i" when its interface
directory holds no snippets. FreeBSD ships openresolv 3.9.2 in its base
system, so on FreeBSD hosts with static network configuration that status
aborted every DNS reconfiguration. A listing that prints nothing on stdout
or stderr now yields an empty base config whatever its exit status, which
also covers the exit status 2 that openresolv 3.17 returns.
The natlab DNSOpenresolv mode now provisions FreeBSD guests as well, by
deregistering the snippets dhclient added, and the openresolv VM tests run
on FreeBSD too. Without the net/dns change, the new FreeBSD test fails with
the health error reported in #11907.
Updates #11907
Updates #11129
Signed-off-by: Brendan Creane <bcreane@gmail.com>
Give the client a generation counter, bump it in reset, and stamp each
assignment with it as it is queued. insertTransitConnMapping rejects
assignments from an older generation; because that check happens under
c.mu it is exclusive with reset, which is what actually closes the
window. handleAddressAssignment also checks before sending, so an
assignment invalidated while it was still queued doesn't reach a
connector at all.
Fixes https://github.com/tailscale/corp/issues/51125
Change-Id: Ib524324b6e8a196fdcb5b4cd307854f61c7a0f04
Signed-off-by: Mazdak Nasab <mazdak.nasab@gmail.com>
Since aa5da2e5f, a peer that control adds to a running client arrives as a
netmap delta rather than a full netmap. The delta path never re-resolved
the exit node prefs. An exit node selected by IP before it was in the
netmap stayed unresolved, and an auto exit node stayed on the blackhole
route, until tailscaled restarted or a full netmap arrived.
Re-resolve the exit node in UpdateNetmapDelta when the delta can change
the result. Any upserted peer qualifies while an ExitNodeIP is set. An
exit node being upserted or coming online qualifies while an auto exit
node is unresolved. The selected auto exit node going offline, losing its
exit routes, or being removed qualifies once one is selected. The last
case replaces the separate offline check, so the exit node is resolved
once per delta. The auto exit node check walks every peer, so it only
runs in those cases.
On the delta path, the resolved prefs ride in the same Notify as the peer
changes, sent while b.mu is held. resolveExitNodeLocked no longer sends a
Notify; its callers do. The full-netmap path still sends the prefs in a
separate Notify after the netmap.
Fixes#22012
Updates #22206
Signed-off-by: Brendan Creane <bcreane@gmail.com>
client.reset clears the IP pools, assignments and byConnKey, but leaves
addrsCh alone. The channel is buffered, so assignments queued before the
reset are still delivered by sendLoop afterwards: each one POSTs to a
connector from the profile we just left, registering a transit IP this
client no longer knows about, and then fails in addTransitIPForConnector
because the assignment is gone from c.assignments.
Drain addrsCh at the end of reset. The drain is non-blocking, so holding
c.mu can't deadlock against reserveAddresses, which enqueues under the
same lock, or against sendLoop, which receives without it.
Fixes https://github.com/tailscale/tailscale/issues/22185
Change-Id: I579131bc54e82d2c797b4741ccaf6cbe7abfdc4a
Signed-off-by: Mazdak Nasab <mazdak.nasab@gmail.com>
ipn/ipnlocal: resolve serve file redirects against the request path
http.FileServer sends a Location relative to the request URL, such as
"nested/" or "../file". fixLocationHeaderResponseWriter prefixed that
value with the mount point, which dropped every directory between the
mount point and the target. A request for /doc/subdir/nested was
redirected to /doc/nested/.
Resolve the relative Location against the original request path
instead. Collapse leading slashes in the result so a request path like
//host/dir cannot produce a protocol-relative Location.
Fixes#22049
Signed-off-by: Alex Castillo González <alex.castillog33@gmail.com>
This adds a test that walks a PeerRelay through the ProxyClass spec.metrics settings and checks the metrics Service, ServiceMonitor and TS_ENABLE_METRICS at each step, and a case for a PeerRelay waiting on a ProxyClass that is not ready. It also checks that ApplyProxyClass leaves one entry per env name when the ProxyClass sets an env var the operator also sets.
Updates #22034
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
The PeerRelay reconciler ignored a ProxyClass's spec.metrics, so enabling metrics created no metrics Service or ServiceMonitor and the relay Pods never served /metrics. This fixes this by setting TS_ENABLE_METRICS in ApplyProxyClass and calling tailscaled.EnsureMetricsResources for PeerRelay. PeerRelay now also waits for its ProxyClass to be ready before creating anything, as the other proxies do, so the existing ProxyClass test fixture is marked ready.
ApplyProxyClass now replaces an env var that the ProxyClass also sets instead of appending a second one. The PeerRelay StatefulSet is applied with server-side apply, which rejects duplicate env names, so a ProxyClass that sets an env var the operator also sets, such as TS_ENABLE_METRICS, would stop the StatefulSet from being applied.
Updates #22034
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
The metrics Service and ServiceMonitor handling lived in cmd/k8s-operator,
and the ProxyClass reconciler kept its own copy of the ServiceMonitor CRD
check, so reconcilers in k8s-operator/reconciler could not share it. This
moves it into the tailscaled package as EnsureMetricsResources and
EnsureMetricsResourcesDeleted, which take the proxy type and parent, and
switches cmd/k8s-operator and the ProxyClass reconciler over to it. No
behaviour change.
Updates #22034
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Serve looked up its config with the name exactly as the client sent
it. Config keys are lowercase with no trailing dot, so a plain HTTP
request for "svelte.example.ts.net." or "Svelte.Example.TS.net" got a
404. HTTPS had the same gap for case. TLS rejects a trailing dot in
SNI but does not lowercase it, so a client that sent mixed-case SNI
failed the handshake with "no webserver configured". Browsers and curl
lowercase the hostname before connecting, which hid that path.
Lowercase the name and trim the trailing dot in both branches of
getServeHandler, and lowercase the SNI in the cert callback.
Fixes#21538Fixes#21986
Signed-off-by: Brendan Creane <bcreane@gmail.com>
Each send operation (sendPacket, sendPeerGone, sendPong,
sendKeepAlive, etc) calls SetWriteDeadline on the
underlying connection. This re-arms the netpoll timer
on each packet frame write to the intermidiate lazy
buffer for a socket write that happens later. Not the
actual socket write done by runWriter.
Move the SetWriteDeadline down to the buffer that is pulled
by runWriter for each actual socket write.
Updates tailscale/corp#50628
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Adds tests that a conflict while provisioning a Service, a Service on a
ProxyGroup, a ProxyGroup or a Connector proxy results in a timed
requeue, and that the Connector and ProxyGroup status is set to
creating rather than failed. The test for a Service on a ProxyGroup
fails the update to the shared ingress config ConfigMap, as no watch
maps that ConfigMap back to the Service, so without the requeue the
conflict was never retried.
Updates #22074
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Reconcilers logged "optimistic lock error, retrying" with the full
conflict error at Info, and Connector, Recorder and ProxyGroup also put
it in their status. That reads like a failure, but a conflict between a
cached read and an update is expected and is retried. The reconcilers
also returned an empty result, so the retry only happened if a watch
event arrived. This fixes this by logging the conflict at Debug as an
update conflict, using a neutral status message and requeueing after a
short delay.
Updates #22074
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Move the Recorder reconciler and its object specs into
k8s-operator/reconciler/recorder, following the earlier ProxyClass,
DNSConfig and nameserver moves, and switch it to the shared finalizer,
resource tracker and CreateOrUpdate helpers.
Hoist two duplicated helpers into tailscaled.PrefsFromStateSecret and
reconciler.IsOptimisticLockError, and add
k8s-operator/reconciler/reconcilertest for the test helpers each package
had copied, including one fake Tailscale API client in place of four.
No behaviour change; the Recorder's watches and labels are untouched.
Fixestailscale/corp#37086
Signed-off-by: David Bond <davidsbond93@gmail.com>
The PreferredDERP selection loop only considers RegionLatency, which
STUN probes populate for STUN-only regions too. When such a region is
the only one with a latency sample (e.g. self-hosted DERP behind a
TCP-only tunnel while a public STUN server is reachable), it gets
selected as the preferred DERP and all relay traffic blackholes, since
the region has no DERP server to relay through.
Filter STUN-only regions in the selection loop by reusing the existing
regionHasDERPNode predicate (renamed to CanServeDERP), which was
already applied to the HTTPS/ICMP fallback probing but not here.
Regions missing from the map, or present without any nodes, keep the
old behavior, as some tests use latency-only DERP maps.
Fixes#21975
Signed-off-by: Luo Chen <luochen1990@gmail.com>
reserveAddresses inserts the assignment into c.assignments before
enqueueing it, and leaves it there when the enqueue fails. The connector
never learns about such a mapping, and nothing tell it later.
Add addrAssignments.remove and use it to undo the insert and return both
addresses to their pools when enqueueAddressAssignment fails. The caller
still gets a SERVFAIL, but the next DNS query starts from a clean slate.
Fixes https://github.com/tailscale/corp/issues/50917
Change-Id: I5a387bdcf5a5e87e2dca44867e88ed6149678729
Signed-off-by: Mazdak Nasab <mazdak.nasab@gmail.com>
This commit changes the egress ProxyGroup readiness gate to mark a Pod
ready only when, for each egress Service, the Pod is an endpoint in the
EndpointSlice of every IP family that the proxy reports in its egress
status. Before, the gate accepted a Pod that was in either family's
EndpointSlice. On a dual-stack cluster, the StatefulSet rollout could
then move to the next Pod before the other family routed to the new Pod.
The gate reads the families from the proxy's state Secret, because a
proxy older than v1.102 does not report an IPv6 address. Such a proxy
never joins the IPv6 EndpointSlice, so its Pod must not wait for it.
Fixes#21077
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit removes the owner reference from ProxyGroup state Secrets,
also from Secrets that an older operator created. On a Foreground
delete, the garbage collector deleted these Secrets before the
finalizers ran. The finalizers read them to find devices, so cleanup
failed and devices stayed in the tailnet.
The ProxyGroup cleanup now deletes the Secrets after the devices, and
the reconciler watches them by label. The kube-apiserver cleanup finds
its cert resources by label, so it does not need the state Secrets. A
new e2e test covers Foreground deletes.
Fixes#22072
Signed-off-by: David Bond <davidsbond93@gmail.com>
bufioWriterPool was sized at 2KiB, this limited the number
of packets per TLS record during a push, limiting throughput
and increasing syscalls
Increase the buffer to 16KiB to pass extra DERP frames per
flush and TLS record write.
Update the websocket path to also use the pooled buffer
Updates tailscale/corp#50628
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
In singleton egress mode (TS_TAILNET_TARGET_FQDN), a failed resolution
of the target FQDN logged the error but then broke out of the main
runLoop instead of continuing to the next netmap update. From then on
containerboot blocked in wg.Wait() while the pod still looked healthy,
and the egress proxy kept forwarding to stale addresses until restart.
A missing target is typically transient (peer not yet in the netmap,
ACL change in flight), so log and continue to let the next netmap
update retry the resolution and reinstall the forwarding rules.
Fixes#22102
Signed-off-by: Santhi Prakash <b.santhiprakash@gmail.com>
This is a follow-up to #21777, which added this functionality but didn't
test it yet.
Updates tailscale/corp#39984.
Signed-off-by: Naman Sood <mail@nsood.in>
When tailscaled denies a serve config, the CLI advises configuring an
operator to avoid root. That advice is wrong for path and Unix socket
targets, which require root regardless. Replace the access-denied
error for those targets with one that says to use sudo.
Updates #21204
Signed-off-by: Brendan Creane <bcreane@gmail.com>
The operator user could serve a path or Unix socket if tailscaled
found a sudo rule for them, but the check used the GECOS display name
and so failed for most users with a 401. The check cannot be repaired
safely: listing another user's sudo rules never prompts, so it proves
a rule exists, not that the user has authenticated. An operator who
had not entered their sudo password could serve any root-readable file
or root-only socket. Require root instead and say so in the error.
Fixes#21204Fixes#21985Fixes#21988
Signed-off-by: Brendan Creane <bcreane@gmail.com>
portmapper.go was introduced for UPnP in commit
8ed43f1f2a.
This also adds a unit test which exercises the incomplete
natlab server implementation of these portmapping protocols.
Updates #21713
Change-Id: I33b61ccbe8c1d8d99d56aa357ae24777be3f5739
Signed-off-by: Francois Marier <francois@tailscale.com>
This type was previously logically distinct from AppConnectorAttr
despite containing the same fields. It is now logically also the same,
so doesn't need to exist separately.
Fixestailscale/corp#50897.
Signed-off-by: Naman Sood <mail@nsood.in>
Wire up wireguard-go's new NewPeerConfig.Hybrid field. (no current users)
Updates tailscale/corp#50068
Co-authored-by: David Anderson <danderson@tailscale.com>
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I80d7af011b43273e6c949214c6d862f9247a82e1
The funnel command checked Funnel access with a hardcoded port 443, so
any requested port passed and the CLI reported it as available on the
internet. Check the port the user asked for instead. Turning Funnel
off skips the check so existing configs on other ports can still be
removed.
Fixes#10742Fixes#21989
Signed-off-by: Brendan Creane <bcreane@gmail.com>
* cmd/k8s-operator: grant dnsconfigs/finalizers so the nameserver provisions on OpenShift
The nameserver ServiceAccount, Deployment, Service and ConfigMap are created
with an ownerReference to the cluster-scoped DNSConfig and
blockOwnerDeletion: true. Clusters running the
OwnerReferencesPermissionEnforcement admission plugin - which OpenShift enables
by default and upstream Kubernetes does not - require the creating identity to
hold update on the owner finalizers subresource before accepting such a child.
The operator ClusterRole grants dnsconfigs and dnsconfigs/status but no rule on
any finalizers subresource, so every nameserver reconcile is rejected:
serviceaccounts "nameserver" is forbidden: cannot set blockOwnerDeletion if
an ownerReference refers to a resource you cannot set finalizers on
It retries forever, the DNSConfig never gets a status, and dns-records-reconciler
waits indefinitely. Granting the rule lets the reconcile succeed on the next
attempt; it is inert where the admission plugin is not enabled.
Static manifest regenerated with ./tool/go run ./cmd/k8s-operator/generate staticmanifests
Fixes#21275
Signed-off-by: Hamza Khokhar <hamza.khokhar660@gmail.com>
* cmd/k8s-operator: grant proxygroups, recorders and services finalizers
Extend the OwnerReferencesPermissionEnforcement fix to the other owners whose
children are created with blockOwnerDeletion: true: ProxyGroup and Recorder
(tailscale.com) and Service (core). Without update on their finalizers
subresource, creates of the owned resources are rejected on clusters that
enable the admission plugin, such as OpenShift.
Static manifest regenerated with ./tool/go run ./cmd/k8s-operator/generate staticmanifests
Updates #21275
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Hamza Khokhar <hamza.khokhar660@gmail.com>
---------
Signed-off-by: Hamza Khokhar <hamza.khokhar660@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Restructure the L3, L3 HA and L7 ingress e2e tests into IP family subtests
to cover IPv4 and IPv6, plus DualStack for ProxyGroup.
Updates tailscale/corp#38029
Signed-off-by: Becky Pauley <becky@tailscale.com>
The root device fetch timed out after one second, but some router firmware only starts answering after just over a second, so UPnP never worked behind it. Allow three seconds.
Updates #21860
Signed-off-by: Aniket <aniketkomailkro@gmail.com>
Serve deletes client-supplied `Tailscale-User-Login` (and similar) headers
before setting its own. However, `http.Header.Del` only matches the exact
canonical name. A client could send Tailscale_User_Login instead, which
was passed through to the backend.
For Go (and some other) backends, that's not an issue, but backends that
map headers to CGI-style variables (HTTP_TAILSCALE_USER_LOGIN) treat
both names the same. Thus, for Funnel requests (or tagged nodes), where
serve sets no identity headers, the backend could trust the spoofed
value. The same applies to Tailscale-App-Capabilities.
This fixes the issue by also deleting any header whose name matches
case-insensitively after replacing '_' with '-'.
This is similar to how other projects do this:
- Django drops them: https://www.djangoproject.com/weblog/2015/jan/13/security/
- nginx does not allow them by default: https://nginx.org/en/docs/http/ngx_http_core_module.html#underscores_in_headers
- Puma had a CVE and now drops them: https://nvd.nist.gov/vuln/detail/cve-2024-45614Fixestailscale/corp#50821
Signed-off-by: Andrew Dunham <andrew@tailscale.com>
Similar to b6fc02d for private destinations, a destination inside one of
the configured magic or transit IP pools is never somewhere an app can be
reached: the magic range is installed as a route towards Tailscale on
every configured node, and the transit range belongs to conn25's own
peer-to-connector mappings. Forwarding to one routes back into the
mapping layer instead of out to an app.
Updates https://github.com/tailscale/corp/issues/50118
Updates https://github.com/tailscale/corp/issues/46169
Change-Id: I8116543d4b7a02e1f4c3c6b882ee3ddec1ac4f97
Signed-off-by: Mazdak Nasab <mazdak.nasab@gmail.com>
derp.Client.ForwardPacket flushed on every call, so each packet a
derper forwards to a mesh peer cost its own TLS record and write
syscall, serialized behind the client's write mutex with the syscall
inside the critical section.
ForwardPacket now appends the frame to one of two 16 KiB buffers and
returns. A flusher goroutine, started if none is running, swaps the
buffer for the spare under the mutex and writes it with the mutex
released, so callers keep filling the spare while a write is in
progress and frames from concurrent callers share a write. A caller
that finds the spare full waits for the write to complete, preserving
backpressure. The buffers are allocated on first forward, so only mesh
clients carry them.
Forward buffers are written directly to the connection rather than
through the client's bufio.Writer. A new mutex serializes those writes
with the client's other senders, since not every Conn tolerates
concurrent writes and bufio can split one frame over several.
A failed write is logged with the number of frames dropped, closes the
connection, wakes waiting callers with the error, and is returned by
every later call, so derphttp reconnects as before.
Adds BenchmarkForwardPacket, which also reports percentiles of
ForwardPacket's own latency, unit tests for batching, backpressure,
oversized frames, write failure, and write serialization, and an
end-to-end test that concurrently forwarded packets all reach the
destination.
Updates tailscale/corp#50628
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Go's HTTP/1 server closes the request body once the handler starts
writing the response. The proxy's Transport may still be reading that
body, and when its final read fails it closes the backend connection
under the response it is copying. The client gets a truncated response.
Fixes#21894
Signed-off-by: Brendan Creane <bcreane@gmail.com>
Co-authored-by: Sam Jean <samj@tailscale.com>
Add an ipn.Prefs.Tailnet field, passed through controlclient.Options
into tailcfg.RegisterRequest.Tailnet, so a client can recommend or
require a tailnet at registration (e.g. to skip the tailnet picker when
the intent is already known) without impersonating MDM.
Previously this value could only come from the "Tailnet" syspolicy
setting. That setting still takes precedence as an administrative
override; the prefs value is used only when no policy is set.
Updates #22093
Signed-off-by: Adriano Sela Aviles <adriano@tailscale.com>
Adds an e2e test for proxy cleanup when the state Secret has no
device_id, as the unit tests use a fake client and can't show that
control honours the machineKey filter. It fills the tailnet with other
devices, checks the filter returns only the proxy's device, then removes
device_id from the state Secret and unexposes the Service. It checks
that the proxy's device is deleted and the other devices are left alone.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Adds tests that ProxyGroup scale-down and cleanup, Recorder cleanup and
PeerRelay cleanup delete the device matching the machine key in a state
Secret that has no device ID. The Recorder test puts that Secret on
replica 0, so it also checks that cleanup carries on to the other
replicas.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Adds a test that proxy cleanup deletes the device matching the machine
key in a state Secret that has no device_id. It also checks that
device_id is still used when present, that Secrets with a missing or
invalid machine key are skipped, and that Connector status does not
list a proxy before it reports its device ID.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
ProxyGroup, Recorder and PeerRelay cleanup and replica scale-down only
delete a replica's device if its state Secret holds the device's ID. A
replica that is shut down after it registered with control but before it
learnt its ID leaves its device in the tailnet.
This fixes this by deleting the device by the machine key in the state
Secret in each of these places when there is no ID.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Alex Valiushko