* cmd/k8s-operator: grant dnsconfigs/finalizers so the nameserver provisions on OpenShift
The nameserver ServiceAccount, Deployment, Service and ConfigMap are created
with an ownerReference to the cluster-scoped DNSConfig and
blockOwnerDeletion: true. Clusters running the
OwnerReferencesPermissionEnforcement admission plugin - which OpenShift enables
by default and upstream Kubernetes does not - require the creating identity to
hold update on the owner finalizers subresource before accepting such a child.
The operator ClusterRole grants dnsconfigs and dnsconfigs/status but no rule on
any finalizers subresource, so every nameserver reconcile is rejected:
serviceaccounts "nameserver" is forbidden: cannot set blockOwnerDeletion if
an ownerReference refers to a resource you cannot set finalizers on
It retries forever, the DNSConfig never gets a status, and dns-records-reconciler
waits indefinitely. Granting the rule lets the reconcile succeed on the next
attempt; it is inert where the admission plugin is not enabled.
Static manifest regenerated with ./tool/go run ./cmd/k8s-operator/generate staticmanifests
Fixes#21275
Signed-off-by: Hamza Khokhar <hamza.khokhar660@gmail.com>
* cmd/k8s-operator: grant proxygroups, recorders and services finalizers
Extend the OwnerReferencesPermissionEnforcement fix to the other owners whose
children are created with blockOwnerDeletion: true: ProxyGroup and Recorder
(tailscale.com) and Service (core). Without update on their finalizers
subresource, creates of the owned resources are rejected on clusters that
enable the admission plugin, such as OpenShift.
Static manifest regenerated with ./tool/go run ./cmd/k8s-operator/generate staticmanifests
Updates #21275
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Hamza Khokhar <hamza.khokhar660@gmail.com>
---------
Signed-off-by: Hamza Khokhar <hamza.khokhar660@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Restructure the L3, L3 HA and L7 ingress e2e tests into IP family subtests
to cover IPv4 and IPv6, plus DualStack for ProxyGroup.
Updates tailscale/corp#38029
Signed-off-by: Becky Pauley <becky@tailscale.com>
Adds an e2e test for proxy cleanup when the state Secret has no
device_id, as the unit tests use a fake client and can't show that
control honours the machineKey filter. It fills the tailnet with other
devices, checks the filter returns only the proxy's device, then removes
device_id from the state Secret and unexposes the Service. It checks
that the proxy's device is deleted and the other devices are left alone.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Adds tests that ProxyGroup scale-down and cleanup, Recorder cleanup and
PeerRelay cleanup delete the device matching the machine key in a state
Secret that has no device ID. The Recorder test puts that Secret on
replica 0, so it also checks that cleanup carries on to the other
replicas.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Adds a test that proxy cleanup deletes the device matching the machine
key in a state Secret that has no device_id. It also checks that
device_id is still used when present, that Secrets with a missing or
invalid machine key are skipped, and that Connector status does not
list a proxy before it reports its device ID.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
ProxyGroup, Recorder and PeerRelay cleanup and replica scale-down only
delete a replica's device if its state Secret holds the device's ID. A
replica that is shut down after it registered with control but before it
learnt its ID leaves its device in the tailnet.
This fixes this by deleting the device by the machine key in the state
Secret in each of these places when there is no ID.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Recorder cleanup deletes the device of each replica in turn. When a
replica had no state Secret or no device ID in it, cleanup stopped there
and reported itself done, so the devices of later replicas were left in
the tailnet.
This fixes this by moving on to the next replica instead.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Service, Ingress and Connector proxy cleanup finds the devices to delete
through the device_id key in each proxy's state Secret, which
containerboot only writes after the proxy's first netmap. If the proxy is
shut down after it registered with control but before that (for example,
a Service exposed and then unexposed in quick succession), cleanup finds
no device, removes the state Secret and the finalizer, and the device is
left in the tailnet. Connector replica scale-down has the same problem.
This fixes this by deleting the device by the machine key in the state
Secret when there is no device_id. Service, Ingress and Connector status
still only list devices that have an ID.
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Similar to #20866 for 4via6, conn25 needs to block host-scoped
destinations. Due to the similarity between the two situations, the
matching is no longer in a 4via6-specific location. Since there is no
backwards compatibility to satisfy, there is no exception list for
conn25.
ipn/ipnlocal (for 4via6) and feature/conn25 now both use the new
package.
Updates tailscale/corp#50118
Updates tailscale/corp#46169
Change-Id: Ic3f9cccd2f5823a2ca48bd78607792b0fd6d7db9
Signed-off-by: Adrian Dewhurst <adrian@tailscale.com>
Add end-to-end tests for egress to a 4via6 target (an address behind a
subnet router advertising a 4via6 route). Tests include:
- TestConnector4via6Egress/HAEgress: egress to an in-cluster Connector
subnet router. Test requires a dual stack cluster.
- TestHostSubnetRouter4via6Egress/HAEgress: egress to a subnet router that
runs on the test host as a tsnet node. Use RegisterFallbackTCPHandler to
mock the backend response without requiring forwarding on to a backend node
(for test simplicity).
Also update end-to-end test acl for 4via6 routes, and install a dedicated
ProxyClass to accept routes.
Updates tailscale/corp#45732
Signed-off-by: Becky Pauley <becky@tailscale.com>
The operator set each kube-apiserver ProxyGroup's LogLevel from its own
logger's Level(). Unless TS_NO_LOGS_NO_SUPPORT is set, that logger is
teed into the log upload core at debug, so Level() is always debug and
every k8s-proxy logged at debug whatever OPERATOR_LOGGING said. In our
fleet that is millions of debug lines an hour from the proxies.
Read the level before adding the tee and hand it to the ProxyGroup
reconciler. Log uploads are unchanged.
Fixes#21887
Change-Id: I3dd791549c60ba2a4f3c2651da6716623da571a6
Signed-off-by: sugarraysam <sbd@fireworks.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Updates tailscale/corp#49775
Track authoritative NXDOMAINs for at least 60 seconds on macOS and iOS (where our short SOA ttls are not properly respected), and flush when a cached name becomes resolvable. Compile a no-op tracker elsewhere and avoid duplicate VPN reconfiguration on link changes.
The reconciler forced the AWS load balancer address type to ipv4, so on
an IPv6 EKS cluster, where pod targets are IPv6, no Network Load
Balancer was ever provisioned.
Make the address type a default that spec.service.annotations can
override, defaulting IPv6 source NAT on when dualstack is requested.
Resolve load balancer hostnames for both address families and requeue
ready PeerRelays every five minutes, since DNS records for a new load
balancer arrive over several minutes. Add spec.service.ipFamilyPolicy
and ipFamilies so dual-stack GKE and AKS clusters can request a
dual-stack load balancer too.
Shorten the PeerRelay field descriptions and link to the PeerRelay docs
for the detail.
Fixes#21399
Change-Id: Idf22eb02f8ed9a46ff4abea8d4faef4b8c12c111
Signed-off-by: David Bond <davidsbond93@gmail.com>
updates tailscale/corp#33007
When a selected exit node can't carry internet traffic due to a misconfiguration (wrong ID, node
deleted, routes removed, etc), blackhole default routes are installed and all internet traffic is dropped.
That is the correct behavior -- better to drop than leak to the local network -- but it was entirely silent:
the stable-ID lookup in nodeBackend.updateRouteManagerPrefs misses without a log line, and Status
leaves ExitNodeStatus nil. The user sees a healthy Tailscale with no internet. If an admin fat-fingers the
exit node name in an IT policy, for example, it's easy to break every node with zero feedback.
This adds an exit-node-unavailable health warning covering every way the selection can fail to carry
traffic (short of reachability which is a separate concern), reported via exitnodehealth.ArgExitNodeReason.
The warning names the exit node, caching its display name while it is still a peer so the name survives
its departure, and falls back to the stable ID or IP. When the selection is mandated by the ExitNodeID
or ExitNodeIP policy settings, the message tells the user to contact their network administrator instead
of suggesting they pick another exit node.
To test this, set a forced exit node policy with some random ip or node id. The warning has a 5 second
threshold. It should clear as soon as you change to a proper exit node.
Signed-off-by: Jonathan Nobels <jonathan@tailscale.com>
Previously ProxyClass spec.staticEndpoints was only implemented for
ProxyGroups and silently ignored when the ProxyClass was referenced by
a Connector. Extract the NodePort Service provisioning, port
allocation, and node ExternalIP discovery logic shared with ProxyGroup
into a new k8s-operator/reconciler/staticendpoints package, and wire
it into the StatefulSet reconciler so that each Connector replica gets
a per-pod NodePort Service, the discovered endpoints are written to
its tailscaled config as static endpoints, and tailscaled listens on
the Services' target port via the PORT env var. Also reconcile
Connectors on Node changes, clean up NodePort Services on scale down,
Connector deletion, and when static endpoints are removed, and
surface the endpoints in the Connector's device status.
Add e2e tests for static endpoints on both Connectors and ProxyGroups.
They verify the per-replica NodePort Services, the PORT env var, the
endpoints reported in status and advertised to control, and the
cleanup on scale down and on removal of the static endpoints
configuration. The tests skip on clusters whose Nodes have no
ExternalIP addresses, such as kind.
Fixes#18819
Change-Id: Ia134916062471e6c0088692834257aafabd0e713
Signed-off-by: David Bond <davidsbond93@gmail.com>
Previously, ipconfig /flushdns and ipconfig /registerdns were called
from multiple sites without coordination: the DNS manager's SetDNS
(in a fire-and-forget goroutine), the router's winRouter.Set (synchronously
on every route change), the link change handler, and session unlock.
These could fire concurrently or in rapid succession, spawning multiple
ipconfig processes and putting excessive pressure on the Dnscache service.
In this PR, we introduce a new util/coalescedop.CoalescedOp type that provides
"one in-flight, at most one pending" semantics: Do() is non-blocking,
at most one execution runs at a time, and concurrent/rapid successive
requests coalesce into a single follow-up execution.
We then use CoalescedOp for both ipconfig /flushdns and ipconfig /registerdns,
replacing the fire-and-forget goroutine in windowsManager.SetDNS
and in the session unlock handler in tailscaled.
We also remove the redundant dns.Flush() call from winRouter.Set, since every
meaningful router.Set is followed by dns.Set in Reconfig, which already
flushes.
Updates #21001
Signed-off-by: Nick Khyl <nickk@tailscale.com>
The egress Pods readiness reconciler decided whether a Pod could route
traffic by calling each egress Service's health check through its
ClusterIP up to replicas*3 times and looking for a response from that
Pod. That is O(replicas) calls per Service per reconcile, and a Pod
joining a large ProxyGroup is only sampled 3 times on average, so late
Pods often miss and wait for another 5s requeue. The call only goes
through kube-proxy on the operator's own node, so it does not show that
other nodes route to the Pod either.
egress-eps-reconciler already adds a Pod to an egress Service's
EndpointSlice only once the Pod's state Secret shows the proxy has set up
routing for that Service. This sets the readiness condition once the Pod
is an endpoint in an EndpointSlice of every egress Service for the
ProxyGroup, and removes the health check calls.
Updates tailscale/corp#39464
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
The egress Pod readiness reconciler polls each Service's health check
endpoint up to replicas*3 times per Pod. A freshly created Pod is initially
unreachable, so these backoff log lines carry no value and generate
significant noise.
Discard these logs so the per-poll message is no longer logged (the more
useful message "Pod is not yet added as an endpoint for all egress targets,
waiting..." is still logged once per reconcile).
Updates tailscale/tailscale#21079
Signed-off-by: Becky Pauley <becky@tailscale.com>
When the tailscale.com/proxy-group annotation is removed from a Service
that was exposed on a ProxyGroup, the HA Service reconciler returned early
on the empty annotation before reaching its cleanup path. The Tailscale
Service was left advertised and the operator's finalizer was never removed,
so the Tailscale Service leaked and the Kubernetes Service wedged forever in
Terminating once deleted.
The annotation is the only place the ProxyGroup name was recorded, and it is
gone by the time cleanup needs it. This commit encodes the ProxyGroup name in
the finalizer, which survives any modification of the resources annotations.
On reconcile, an empty annotation with our finalizer present now recovers the
ProxyGroup name from the finalizer and runs cleanup. Services carrying the bare
legacy finalizer are migrated to the encoded form while the annotation is still
present.
Updates #19922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
An Ingress annotated with
`tailscale.com/experimental-forward-cluster-traffic-via-ingress` forwards
cluster traffic to the proxy's Pod IP, which is DNATed to the node's Tailscale IP
where serve answers it. On Linux the serve listener is bound to the tunnel
interface and drops that traffic, so set TS_SERVE_ALLOW_ALL_INTERFACES on the
proxy when this annotation is used, which makes serve answer it again.
Document on the annotation how the traffic reaches serve and that it bypasses
tailnet ACLs, and regenerate the CRD and operator manifests.
Updates tailscale/corp#48248
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
The e2e suite covered the operator's in-process API server proxy but
not the ProxyGroup-based one. Add tests for both proxy modes that
drive a ConfigMap through its lifecycle via the proxy, verify a
forbidden request is rejected, and check that deleting the ProxyGroup
cleans up its StatefulSet and Tailscale Service.
Fixestailscale/corp#38009
Change-Id: Ifc0be47ce32dd96f8daa748af6785a8b82ec19a7
Signed-off-by: David Bond <davidsbond93@gmail.com>
Loosens the connector route assertion to allow concurrent tests against
the same tailnet to more reliably pass, while still asserting the
client itself is advertising those routes and they're recognised in the
API.
Also make createOrUpdate more resilient to a test that didn't fully tear
down on the same cluster previously. By respecting the existing resource
version and finalizers, we can update resources that didn't get deleted
from a previous run.
Updates tailscale/corp#45426
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
Make sure the tests select matching-family IP addresses for ingress
tests so they're able to pass on IPv6 clusters. We should probably
follow up with another change for the operator that stops clients from
needing to do this sort of selection, but updating the tests is the easy
option to get them passing on IPv6 clusters in the short term.
Updates tailscale/corp#45426
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
* cmd/k8s-operator: move egress EndpointSlice write back into gated provision
PR #20347 moved the EndpointSlice createOrUpdate outside of provision,
causing it to run on every reconcile. This resulted in racing egress-eps on
the EndpointSlice, sometimes causing the TailscaleEgressSvcConfigured to
become stuck as False with the Service not fully updated. Gate it again so
it only runs when a reprovision is required.
Updates #20916
Signed-off-by: Becky Pauley <becky@tailscale.com>
* cmd/k8s-operator: recover missing egress EndpointSlices
Add a watch for EndpointSlices in the egress-services reconciler so a
deleted slice re-triggers a Service reconcile directly. Treat a Service
whose expected per-family EndpointSlice is missing as not up to date so it
re-enters provision and recreates the slice.
Also sort endpoints by Pod UID before writing them in the egress-eps
reconciler, so an unchanged set of ready Pods cannot result in a different
order and trigger an unnecessary Update.
Updates #20916
Signed-off-by: Becky Pauley <becky@tailscale.com>
---------
Signed-off-by: Becky Pauley <becky@tailscale.com>
The varz handler got its memstats_* metrics from the expvar package's
"memstats" func, which calls runtime.ReadMemStats and so stopped the
world on every Prometheus scrape. Keep the names but compute them from
runtime/metrics, and never call that func, even from
WritePrometheusExpvar.
While there, export the /tailscale/ metrics from our Go fork (stack
size histogram, stack copy counters, timer zombie counts and lifetime
histogram), which nothing could see before, plus a few upstream ones
with no MemStats equivalent: scheduling latency and GC pause
histograms, live heap, GC and total CPU seconds, thread count, and
mutex wait time. The last replaces derper's hand-rolled version.
Everything read is cheap and a scrape allocates nothing after the
first. Names use a go_runtime_ namespace rather than go_ so they can't
collide with the Prometheus Go client's collector in promvarz binaries.
The runtime's 162-bucket time histograms are reduced to one bucket per
factor of four from 256ns to 1s.
Updates #21300
Updates tailscale/go#189
Updates golang/go#75935
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I7e3c9a41f2b85d6e0c4a9b1d3f8e7c2a5b6d4e19
Adds tsweb/compserve: content negotiation for precompressed static
variants, with a transcode-to-identity fallback for clients that do not
accept an encoding (including when the raw file is absent), and
CompressWriter, which live-compresses dynamic responses with zstd in its
fastest mode, streamed incrementally with no buffering. Negotiation is
q-value and wildcard aware (gzip;q=0 previously matched gzip).
client/web serves its prebuilt embedded assets through compserve,
replacing brotli with zstd; the embedded FS is wrapped in
tsweb/vcstime for conditional-request mod times. tsweb/compserve/gzip.go
keeps transitional serving of gzip variants from pre-zstd file systems
(such as the currently published web-client-prebuilt module):
passthrough to gzip-accepting clients, transcoded to identity otherwise;
it becomes inert once a zstd-only module is published.
util/zstdframe gains pooled GetDecoder and GetStreamingEncoder
(concurrency=1). util/precompress is now a build-time tool, generating
zstd variants only. cmd/tsconnect and cmd/build-webclient consume the
new precompress/compserve split. tsweb.AcceptsEncoding and
tsweb/tswebutil are removed; negotiation lives in compserve and the
deprecated shim had no callers. go.mod bumps web-client-prebuilt.
Also fixes a transcoding bug where http.ServeContent's size probe via
the promoted zstd.Decoder.WriteTo could report a zero length, serving
empty bodies.
Updates tailscale/corp#20099
Signed-off-by: James Tucker <james@tailscale.com>
This is the output of the new misc/bumpdeps tool (#21325) run with
--exclude-newer-than-days=7, which asks proxy.golang.org for the newest
version of every direct dependency, ignoring releases younger than a
week in favor of the newest older one, and runs a single go get.
gvisor tracks its "go" branch, wireguard-go its "tailscale" branch,
and golang-x-crypto its "main" branch (the proxy's @latest for it is
a stray v0.91.0 tag from 2024 that predates our acme fork changes).
Indirect deps only moved as far as MVS pulled them.
The week-long cooldown held back gvisor, the gokrazy modules,
chromedp/cdproto, and hashicorp/raft-boltdb/v2, whose only newer
versions are days old; they'll come along next time.
Several upstream changes needed small fixes: nfpm's PrepareForPackager
takes a modification time now (a zero time keeps the old behavior of
using the source file's mtime), esbuild's ServeOptions.Port became an
int while ServeResult.Host became a Hosts slice, client-go's
EventRecorder.Eventf is now recognized by vet as a printf wrapper (so
the k8s-operator calls that passed a preformatted message switch to
Event), google/nftables v0.3.0 reads back the kernel's
NF_NAT_RANGE_PROTO_SPECIFIED flag into a new expr.NAT.Specified field
(so the port map DNAT rule now sets it too or findRule never matches
the rule it just added), and staticcheck v0.8.1 knows encoding/json/v2's
embed tag option, so the two SA5008 suppressions for it are gone.
Two tests assumed old library behavior. client-go's fake clientset now
replays existing objects when a watch starts, as a real apiserver does,
so the k8s-proxy config test must tolerate the loader ignoring that
no-op event before the real reload arrives. fyne.io/systray moved its
dbusmenu object path and answers the first GetLayout with depth 1, so
the systray test now finds the menu via the item's Menu property and
polls until the submenu entries appear.
Then make tidy, make updatedeps, and make kube-generate-all (the
controller-gen bump to v0.22.0 changes doc strings, stops listing
top-level metadata as required, and crd-ref-docs now marks optional
fields).
Updates #8043
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I3f9a2c6e8b1d4705a9e2c7b8d1f4e6a0c2b5d8e3
Update gVisor to include its fix for RACK loss detection with coarse
monotonic clocks. Configure netstack with the 500 microsecond clock
resolution used on Windows so RACK accounts for timestamp quantization.
Remove the TCP recovery override that disabled RACK, enabling gVisor's
default RACK behavior on all platforms.
Switch netstack to cubic congestion control. The int overflow in CUBIC
sender cwnd arithmetic that required pinning reno has since been reworked
upstream into float arithmetic with RFC 9438 target clamping.
Align the natlab vnet stack with netstack: enable cubic, and drop the
now-redundant explicit SACK and receive-buffer moderation sets, both of
which are gVisor defaults.
Fixes#9707
Signed-off-by: James Tucker <jftucker@gmail.com>
The map response reader now caps a single message at 256 MiB on the wire
and 1 GiB after zstd decompression. The server-chosen uint32 size prefix
previously let a malicious control server make us allocate up to 4 GiB
before reading any body bytes, and the decoded size was unbounded, so a
small zstd frame could expand into gigabytes of JSON. A 16 MB cap has
been hit by real production traffic before, so both limits sit far above
plausible legitimate sizes. The size-prefixed read moved into a
readMapResponseMessage helper, and the newer control/tsp path already
enforced both kinds of bounds; this brings the long-poll path it
replaces in line, with more generous limits for large tailnets.
ts2021.Client.Do additionally caps every noise response body with
httpbody.LimitSize, so a malicious or buggy control server can't make us
buffer an unbounded response. Client.Do shadows the embedded
http.Client's Do method, so register, set-dns, set-device-attr,
audit-log, all DoNoiseRequest consumers (webclient, tailnet lock, SSH
actions, id-token, feature queries), and the debug CLI get the cap
without per-call-site changes, and future noise endpoints get it for
free.
The cap lives in the new util/httpbody package so other HTTP clients can
adopt the same convention: LimitSize looks the size limit up from
res.Request's context (a Response knows the Request that produced it),
falling back to DefaultMaxSize, 1 MiB, when the context carries no
override. It is like io.LimitReader except that reads past the limit
fail with an error wrapping httpbody.ErrTooLarge instead of silently
truncating, and a body of at most the limit, including one of exactly
the limit, reads back without error: the wrapper probes for EOF once the
limit is exhausted to tell an exactly-at-limit body from an oversize
one. The per-request override, httpbody.WithMaxSize, is a context key,
so transports pick it up with no API changes; LimitSizeTo applies an
explicit limit ignoring any override. Repeated LimitSize or LimitSizeTo
calls replace the previous limit rather than compounding it, so a later
call can raise or remove the limit an earlier one set.
Responses that stream an unbounded number of individually bounded
messages disable the cap with httpbody.WithMaxSize(ctx, 0): the
/machine/map long-poll and control/tsp's map session, whose messages are
already capped per-message (by readMapResponseMessage and decodeMsg, and
by tsp's framedReader and boundedReader). Their non-200 error bodies are
not message streams, so those are capped with LimitSizeTo instead.
The tailnet lock /tka/init/begin, /tka/sync/offer and /tka/affected-sigs
responses can carry per-node key signatures or missing AUMs, which at
100,000 peers reach tens of MB, so they raise the cap to 512 MiB. The
per-response io.LimitedReader decoders that silently truncated those
responses at 1 or 10 MiB are removed: the transport cap is now the
single enforcement point, and it reports oversize bodies instead of
truncating them.
The /key fetch over plain TLS switches from io.LimitReader to
httpbody.LimitSizeTo, so an oversized response reports the problem
instead of producing a confusing truncated-JSON error.
Thanks to Ben Carman for the report!
Updates tailscale/corp#48187
Reported-by: Ben Carman
Change-Id: Ibf95e1ab9e4f0d7ef8866e8c26e62ed2a514455a
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Adds an opt-in, in-memory aggregator of recent connection-rejection
events (TSMP rejects received from peers, outbound TSMP rejects we emit
on ACL-blocked inbound flows, and pendopen timeouts) keyed by
(direction, proto, peer-address, reason). The aggregated data is exposed
over a new debug-rejects LocalAPI endpoint and a GET /debug/rejects c2n
endpoint, intended for future GUI/CLI consumption when diagnosing why a
connection failed.
Architecture:
- net/connreject holds the data types and a per-LocalBackend
Aggregator (LRU-bounded, default 256 entries on desktop / 32 on
mobile, per direction).
- feature/connreject is a self-registering ipnext.Extension that owns
one Aggregator per LocalBackend, installs note callbacks on the
tundev and engine, subscribes to OnSelfChange to flip the runtime
gate, and serves the LocalAPI/c2n endpoints.
- wgengine.Engine and *tstun.Wrapper each gain a SetConnRejectNote
setter; data-plane sites use a single atomic.Pointer load + nil
check, so the cost when no consumer is installed is one MOV.
Gating:
- Compile-time: ts_omit_connreject build tag (standard
feature/buildfeatures + condregister plumbing). Trims ~41 KB.
- Runtime: nodecap.ConnReject node attribute, off by default
at the control plane. May be removed once the feature is enabled
by default.
Updates CapabilityVersion to 146 (clients understand nodecap.ConnReject
and can serve GET /debug/rejects).
Adds Proto/Src/Dst accessors on flowtrack.Tuple (used by pendopen to
construct events without exposing the tuple's internals to the
aggregator).
Updates #1094
Updates #14802
Change-Id: I83e8f24a7e66fa2d158d128bd25fbe851134941b
Signed-off-by: James Tucker <james@tailscale.com>
The PeerRelay resource hardcoded UDP port 41641 in the generated
Service, the tailscaled config, status.endpoints and the advertised
static endpoints. That breaks deployments where the load balancer
address or port is not what peers can actually reach, e.g. behind NAT.
Add spec.service.port to choose the UDP port, and spec.staticEndpoints
to advertise extra address:port pairs alongside the discovered load
balancer endpoints. A replica whose only endpoint is a static one
counts as addressed, and a static entry's port wins when it names an
address the load balancer already provides.
The e2e tests now supply static endpoints, which lets them assert the
PeerRelayReady condition goes True in kind, where no cloud controller
ever gives the LoadBalancer Services an address.
Fixes#20821
Change-Id: I463012cd447c81c2849fa653fa18eb82662aaf4f
Signed-off-by: David Bond <davidsbond93@gmail.com>
Move the container image build logic out of the e2e test setup into
cmd/k8s-operator/e2e/internal/build, with a thin CLI wrapper at
cmd/k8s-operator/e2e/build, so CI can publish images once per commit and
fan out into multiple test jobs that consume them. The command skips
images that already exist in the registry, so a retried run doesn't
fail if the registry is immutable.
Also adds support for the test harness leveraging workload identity
federation credentials so we can use GitHub's token issuer for the test
code itself, and AWS' OIDC provider for the operator, and avoid using
any secrets in CI.
Updates tailscale/corp#46577
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
This commit bumps the wireguard-go dependency to incorporate changes to
the packet memory model and the tun.Device.Read and conn.ReceiveFunc I/O
interfaces. It updates their implementations accordingly.
These changes improve throughput in all measured benchmarks and reduce
peak RSS in six of eight cases. The two regressions will be addressed in
a follow-up commit that reduces peak RSS below the baseline measured at
1e69418. That work is kept separate to simplify review.
The following throughput and peak RSS benchmarks were performed with
iperf3 between two Intel i5-12400 nodes running Ubuntu 24.04 (Linux 6.8).
The UDP benchmarks did not use UDP GSO on the sender, so they were
roughly equivalent to single packet I/O through wireguard-go.
TCP/1 signifies one TCP stream; TCP/128 signifies 128 parallel TCP
streams.
Throughput (Mb/s)
Test 1e69418 After Change
TCP/1 10,371 11,354 +9.5%
TCP/128 7,886 8,404 +6.6%
UDP/1 2,111 2,853 +35.1%
UDP/128 1,747 2,235 +28.0%
Peak memory (VmHWM, kB)
Test Side 1e69418 After Change
TCP/1 TX 98,240 52,596 -46.5%
RX 287,748 73,384 -74.5%
TCP/128 TX 101,196 52,812 -47.8%
RX 290,420 63,620 -78.1%
UDP/1 TX 58,864 160,840 +173.2%
RX 137,516 49,900 -63.7%
UDP/128 TX 66,148 116,096 +75.5%
RX 154,384 56,556 -63.4%
Updates tailscale/corp#46716
Updates tailscale/corp#22467
Updates tailscale/corp#36989
Updates tailscale/corp#37878
Signed-off-by: Jordan Whited <jordan@tailscale.com>
Replace the allowed-IP-only peer callback result with wgcfg.PeerConfig.
It carries allowed IPs and an optional pre-shared key through lazy peer
creation and active peer synchronization.
Update wireguard-go for the new peer PSK APIs.
Updates tailscale/tailcat#84
Change-Id: Iacd9d2c74b0b64d690f3cbdf93918686ac6076d7
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Previously, the new disco keys entering the client via TSMP was routed
into controlClient to allow for deduplication and filtering of disco
keys and avoid control overwriting an active TSMP learned key with a
stale key.
A system supporting multiple disco keys was introduced to allow these
keys living side by side, along side a system for selecting an active
egress key with a bias towards keys learned via TSMP. On ingress any
known key is accepted.
This PR makes the switch to key routing, by sending new TSMP learned
disco keys directly into the userspace engine and in turn magicsock,
removing the need for a full layer of deduplication in the
controlClient.
Additionally, the system that previously fully reconfigured clients on
disco key updates coming from control is now using an optimistic
handshake througha newly introduced wireguard-go method,
ScheduleHandshakeOnUserSend implemented in:
https://github.com/tailscale/wireguard-go/pull/81
What this PR does not do is revert the mapSession back to being single
writer. Bringing the mapSession back to this state is desired, however
to ensure the plumbing is done right and easier to reason about, that
will be done in a separate PR.
Updates #20590
Signed-off-by: Claus Lensbøl <claus@tailscale.com>
Promote the toolchain from Go 1.26.6 to Go 1.27.0, matching what
go.toolchain.next.rev has been testing. Besides the toolchain files
themselves (updated by pull-toolchain.sh), this bumps the go.mod go
directive, the Dockerfile golang base image, and the README, and
regenerates the depaware.txt files and the gzip assets in
tempfork/spf13/cobra and util/eventbus, whose bytes change with
Go 1.27's rewritten compress/flate.
Also bump golangci-lint to v2.13.1, the first release line built
with Go 1.27; the prebuilt v2.10.1 binary refuses to target a Go
version newer than the one it was built with.
Also bump golang.org/x/net to v0.58.0 (plus the sibling x/ module
upgrades it requires) to pick up upstream commit 8d10596d2624
(http2: avoid deadlocks in wrapped ClientConn state callback),
which we hit during Go 1.27 rc testing.
Also add docs/go-bump-checklist.md for next time.
Updates #20220
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ia3e4c9effafbc91227eed39efb52f1fba1b8d89c
Extend e2e test setup to work with a remote registry and real cluster.
Previously only kind was supported.
Detect cluster node architecture and build images only for the given
architecture to reduce build time. Build for all platforms as the fallback
option. This assumes only one node architecture per cluster, which is
reasonable for our test cases.
By default, --build loads the built images into a kind cluster. Use with
--registry to instead specify and push to a remote registry.
Fixestailscale/corp#46577
Signed-off-by: Becky Pauley <becky@tailscale.com>
The service-pg-reconciler reconciles Services annotated for an ingress
ProxyGroup, but its ProxyGroup watch reused ingressProxyGroupFilter,
which is ingressesFromIngressProxyGroup. That handler lists Ingresses
and returns Ingress keys, so when a ProxyGroup became Available the
requests it produced never matched a Service and the reconciler's Get
just came back NotFound.
This fixes this by adding servicesFromIngressProxyGroup, which lists the
Services indexed for the ProxyGroup and returns their keys, matching what
the egress path already does with egressSvcsFromEgressProxyGroup.
Fixes#20944
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Guard optional annotations maps in generated CEL expressions before
looking up expose or proxy-group annotations. This lets ordinary
annotationless Services and Ingresses pass admission while preserving
deny-all and allowlist behavior for present proxy-group annotations.
RELNOTE: Kubernetes ProxyGroupPolicy now permits annotationless resources.
Fixes#20906
Change-Id: I8b9475d34c003ca6d233246c021ac656e0530fb1
Signed-off-by: joshrzemien <joshua.k.rzemien@gmail.com>
Let clients pin favorite devices, exit nodes, and services so GUIs can
surface & change them. Pins are stored per login profile in the new
favorites feature module, keyed per category; devices and exit nodes by
StableNodeID, services by ServiceName.
The item types live in a leaf package feature/favorites/pintype,
keeping them out of the core ipn hierarchy. Each category has
its own type (pintype.Device, pintype.ExitNode, pintype.Service).
Exposed over LocalAPI at GET/POST /localapi/v0/pins, where POST replaces
only the categories named in the request so a client can update one
category without clobbering the others. Pins are local to the device and
are not synced across a user's devices.
updates tailscale/corp#44836
Signed-off-by: Will Hannah <willh@tailscale.com>
Co-authored-by: Will Hannah <wph@Wills-Virtual-Machine.local>
Deploy k8s nameserver during e2e test setup, and point the cluster resolver
(CoreDNS or kube-dns) to it so tests can resolve MagicDNS names inside the
cluster.
Add a test to verify singleton L7 Ingress is reachable from inside the
cluster using its MagicDNS Name.
Update existing egress tests to use a dedicated tailnet target per test (to
avoid conflicts). Egress tests now also verify that an egress target is
reachable from within the cluster using both its Service and MagicDNS name.
To successfully curl using the target's MagicDNS name, publish test CAs as a
ConfigMap to the cluster, and mount these for each curl pod.
Fixestailscale/corp#38027
Signed-off-by: Becky Pauley <becky@tailscale.com>
Static size checks (iossize) catch binary dirty-page growth but nothing
covered runtime heap cost, which is what actually consumes the iOS
Network Extension's 50 MiB jetsam budget. Bring up a tsnet backend
(with the full condregister feature set, matching shipping clients)
against an in-process testcontrol server and assert live post-GC heap
budgets for (a) backend startup with zero peers and (b) marginal cost
per netmap peer.
The startup test measures 1.3 MiB today and fails loudly on the
conn25 flow-table pre-allocation regression (17 MiB) that jetsam-killed
the iOS extension on large tailnets.
Budgets are deliberately generous (6-12x current measurements) to stay
flake-free while still catching the multi-MiB regressions that matter
for mobile.
A new debugknob enables us to constrain the GSO/GRO batch size to 1 for
these tests so as to avoid the memory allocation associated with those
buffers, which are a known issue with their own work stream.
Updates tailscale/corp#46408
Updates tailscale/corp#18514
Signed-off-by: James Tucker <james@tailscale.com>
This commit modifies the "generate" tool we use on the kubernetes operator
that produces helm chart and static manifest assets for CRDs.
Previously, this required always remembering to add new constants to
a `main.go` and did not have any mechanism to fail in CI if you forgot
to. Now this tool will iterate over all the CRDs and ensures that they're
in the places they're expected to be, with a test that will fail if they
are not.
This removes the requirement for remembering to add these constants
every time you have a new CRD.
Closes: #20594
Signed-off-by: David Bond <davidsbond93@gmail.com>
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
This commit changes how PeerRelay services are exposed on AWS. A Network
Load Balancer only forwards to targets in an availability zone enabled
on it, and spec.aws.elasticIPs pins each service to a single subnet,
which enables just one zone. A replica scheduled anywhere else silently
receives nothing while still reporting PeerRelayReady with an address in
status.endpoints.
Without spec.aws we now leave the subnet unpinned, so the AWS Load
Balancer Controller spreads the load balancer over every zone it finds,
and cross-zone load balancing is on by default so any of its addresses
reach the pod. Hostname resolution is no longer gated on the
eip-allocations annotation, which had left these unpinned services in
EndpointsPending forever, and a failure to resolve now logs at debug
since it is expected while a load balancer provisions.
Such a load balancer has an address per zone, and AWS bills for each, so
every one of them is now advertised rather than only the lowest sorted.
That also lets a peer reach the relay when one zone is unreachable.
status.endpoints gains address as a second list map key so a replica can
hold an entry per address; no field changes, so existing readers of
endpoints[].address keep working. Readiness counts the replicas that
have an endpoint rather than the entries, so a replica with several
addresses cannot mask one that has none.
The pods now serve containerboot's health check endpoint and the load
balancer is pointed at it over HTTP. A peer relay listens only on UDP,
so the default TCP check against the port the load balancer forwards
could never succeed and every target sat unhealthy while relaying
perfectly well. /healthz reports 200 once the device has tailnet
addresses, which is the condition that actually matters.
The CRD docs now describe spec.aws as the exception, note that it also
needs a ProxyClass pinning pods to the zone of the subnets it names, and
drop the claim that an Elastic IP has an availability zone of its own.
Fixes: https://github.com/tailscale/tailscale/issues/20833
Signed-off-by: David Bond <davidsbond93@gmail.com>
The egressPodsReconciler (added in #14792) only sets the
tailscale.com/egress-services readiness condition on egress ProxyGroup
replica Pods that declare the corresponding readiness gate. However, the
gate was never actually added to the egress Pod template, so the
reconciler always hit its early-return and the readiness condition was
never set. This commit adds said readiness gate to the egress Pod template.
Updates #14326
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
ProxyGroup, Recorder, and PeerRelay each carried a near-identical copy of
the auth key re-issuance state machine (in-flight tracking, per-parent rate
limiting, stale-device cleanup). A bug fix had to land in three places and
could silently drift.
Extract it into a shared tailscaled.Reissuer, alongside the other tailscaled
workload helpers (NewAuthKey, AuthKeyFromConfigSecret, DeviceIDFromStateSecret)
that the callers already use. It owns its own mutex, tracks in-flight reissues
per replica keyed by parent, and rate-limits re-issuance per parent; the three
reconcilers drive it via EnsureState/RemoveState/ShouldReissue. The device
deletion helper is shared too, so the reissue state machine and its tests now
live in one place.
Updates #20544
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Package tailcfg defines the types and constants used by the Tailscale
protocol, but since everything is all in one package, it’s difficult
to sift through the docs: https://pkg.go.dev/tailscale.com/tailcfg
We define and enumerate capabilities as string constants for
tailcfg.NodeCapability and tailcfg.PeerCapability. This PR extracts
them into their own packages:
- tailcfg.CapabilityFileSharing becomes nodecap.FileSharing
- tailcfg.NodeAttrOnlyTCP443 becomes nodecap.OnlyTCP443
- tailcfg.PeerCapabilityTaildrive becomes peercap.Taildrive
We originally intended for CapabilityFoo to grant an entitlement or
permission for Foo, and for NodeAttrBar to configure Bar in the
nodeAttrs section of the policy file. However, there was no technical
enforcement of this convention, so new capabilities have used the
NodeAttr prefix regardless of meaning. Therefore, this PR unifies
tailcfg.CapabilityFoo and tailcfg.NodeAttrBar into a single package as
nodecap.Foo and nodecap.Bar.
Ran `go fix -inline ./...` and committed the changes that replaced
uses of the tailcfg aliases with the authoritative ones.
Updates #20259
Change-Id: Ieb7e7e6c8247c39faf42fdf15c68cdc7c621c730
Signed-off-by: Simon Law <sfllaw@tailscale.com>
