We had an internal Google doc about this (Tailscalars: http://go/clientmod) but that doesn't help open source contributors or agents. So move the docs to git. Updates #12614 Change-Id: I0b0e9f0286b23b4fb1b51ff3d41eba75edf62cdf Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
17 KiB
Modular Tailscale features
This directory contains Tailscale's modular feature system. The Tailscale
client has grown large; not every user wants every feature (an IoT device on
a few-dollar chip does not need Taildrop, WebDAV, ACME, or SSH). The
feature/ tree is how we make individual features conditionally linkable so
that resource-constrained builds can omit them, and so that new code lives
in small self-contained packages instead of being dumped into
LocalBackend.
New code lives here
The preferred home for new functionality is a feature/<name> package.
If there is any plausible user who might not want your feature in their
build (which is almost always the case), it should be modular from the
start. A feature package should install itself into the rest of Tailscale
via hooks, extensions, and registrations from its own init (see
"Hooks and registration" below). It should not export API used by callers
elsewhere in the tree; callers reach the feature through the hook, not by
importing it.
Think your functionality is so core that it must always be linked in?
Imagine any plausible user who might not want it. If you cannot,
ask before dumping code into ipn/ipnlocal or
similar.
The half-migrated reality
Much of the tree is in a half-migrated state. A given feature may:
- Have a
ts_omit_<name>build tag declared infeaturetags/featuretags.go, and - Have some of its code moved to
feature/<name>/, but - Still have significant code living in
ipn/ipnlocal,cmd/tailscaled, or elsewhere, either:- Behind a build-tag-conditional file (a whole
.gofile gated with//go:build !ts_omit_<name>), or - Guarded at runtime by an
if buildfeatures.HasFoo { ... }block that the compiler and linker dead-code-eliminate when the tag is set.
- Behind a build-tag-conditional file (a whole
When you touch one of these features, prefer to keep migrating code into
its feature/<name> package rather than adding more to the old location.
But do not feel obligated to finish the migration in one PR.
The registry: feature/featuretags
featuretags/featuretags.go is the single
source of truth. It declares the Features
map: for each feature tag (a
lowercase string like acme or taildrop), it records the exported symbol
name used by generated constants, a human-readable description, and any
other features it depends on (forming a DAG).
Adding a new feature starts with adding an entry there. See the top of the
file for the tag naming convention: feature foo corresponds to build tag
ts_omit_foo (opt-out). The one exception is cli, which is opt-in via
ts_include_cli; see FeatureTag.IsOmittable.
Features can be marked ImplementationDetail: true
when they are internal
plumbing (e.g. dbus, c2n) that users would not select directly; they
exist only so that user-visible features can depend on them.
Generated constants: feature/buildfeatures
After editing featuretags.go, run:
./tool/go generate ./feature/buildfeatures
(or the meta make generate). That regenerates two files per feature in
buildfeatures/: feature_<name>_enabled.go and
feature_<name>_disabled.go, gated with //go:build !ts_omit_<name> and
//go:build ts_omit_<name> respectively. Each pair exports a single
boolean constant, e.g. HasACME. Because these are Go constants, code of
the form:
if buildfeatures.HasACME {
// ...
}
is dead-code-eliminated by the compiler and linker when the feature is omitted. Constants can eliminate code but they cannot remove imports; for that you still need a build-tag-gated file.
Use constants when moving code to feature/<name> is impractical (e.g.
the code has to sit inside LocalBackend), or when a small conditional
inside otherwise-shared logic is clearer than splitting into two files.
Prefer separate packages when you can.
feature/condregister: opt-out registration
condregister/ is the one central package that
tailscaled (and the macOS/iOS closed-source client) empty-imports so
that all "on by default" features get registered. Every feature that
should ship in tailscaled by default has a maybe_<name>.go file here
of the form:
//go:build !ts_omit_<name>
package condregister
import _ "tailscale.com/feature/<name>"
That is the whole file. The build tag is the only mechanism that decides
whether the feature is compiled in. Because tailscaled imports
condregister, all these maybe_* files pull in their respective
feature/<name> packages by default; adding ts_omit_<name> to the
build removes the file, and with it the import, and with it the
package's init and everything it transitively depends on.
Some maybe_* files carry additional build tags (e.g.
//go:build !ios && !ts_omit_capture) when a feature should be excluded
on specific platforms.
The feature/<name> package should not carry ts_omit_* tags
The ts_omit_<name> tag lives only on the condregister/maybe_<name>.go
(and analogous tsnet/maybe_<name>.go) import shim, not on the
feature/<name>/*.go files themselves. Keeping the feature package
free of its own omit tag lets any program directly opt in with a blank
import regardless of what ts_omit_* tags are set on the top-level
build:
import _ "tailscale.com/feature/foo"
That is how a tsnet-using application pulls a feature in on its own
terms without having to reason about which ts_omit_* tags are in
effect.
OS-specific build tags inside feature/<name>/ are fine, and often
desirable: if the feature only makes sense on some platforms, gating
its files with //go:build linux (or !ios, etc.) both keeps
go test ./... passing on every GOOS and prevents a tsnet user on an
unsupported platform from accidentally blank-importing a package that
would fail to compile or run there.
tsnet does NOT depend on condregister
tsnet is a library, not a daemon, and it links in a different (and
generally smaller) default feature set than tailscaled. tsnet has its
own top-level maybe_*.go files that decide which features it opts in
to, and its own depaware.txt.
Consequence: you cannot assume buildfeatures.HasFoo == true means
feature foo was actually linked. In a tsnet build, ts_omit_foo may
not be set (so HasFoo is true), yet tsnet may have never imported
feature/foo, so its init never ran and no hooks were installed.
buildfeatures.HasFoo really means "not explicitly omitted by build
tag." Whether the feature is actually present is
feature.IsRegistered("foo").
The idiom is:
buildfeatures.HasFoo && feature.IsRegistered("foo")
HasFoo lets the compiler DCE the whole expression in ts_omit_foo
builds; IsRegistered guards against the tsnet case where the tag is
unset but the package was never imported.
The feature API
Package feature itself
exposes the small runtime API used by feature packages and their callers:
feature.Register(name): a feature package calls this from itsinitto record that it was linked in. Callers usefeature.IsRegistered(name)to check.feature.Hook[Func]: a single-writer, single-reader hook. The extension point declares avar HookX feature.Hook[func(...)], the feature packageSets it once frominit, and callers useGetOk/GetOrNil/IsSetto invoke it.Setpanics if called twice.feature.Hooks[Func]: a slice of hooks for extension points that may legitimately have multiple registrants.- Common cross-feature hooks (auto-update, proxy, TPM, SSH host keys,
hardware attestation) live in
hooks.goalongside their small dispatcher functions likefeature.CanAutoUpdate()andfeature.TPMAvailable(). Add hooks here only sparingly: every hook in this package is loaded by every consumer offeature, so its function signature must not reference types from "heavy" packages. By heavy we mean any package whose presence in this signature would unnecessarily growcmd/tailscaled/depaware-min.txt: packages with a large API/dependency footprint (e.g.net/http,crypto/tls,golang.org/x/crypto/ssh,k8s.io/...), or packages that trigger expensive runtime features likereflect-based registration. The minimal build's depaware file is sacred and must not grow without good reason. Prefer primitives, smalltypes/...packages, or interfaces. If your hook needs a wide type, keep the hook (and its dispatcher) in the caller's package instead of promoting it here.
ipnext.RegisterExtension
(in ipn/ipnext) is the corresponding
mechanism for features that need to hook into LocalBackend; see the
next section.
IPN extensions: ipn/ipnext
For features that need to attach state and behavior to LocalBackend,
the plain feature.Hook mechanism is not enough; you also need
per-LocalBackend state and a well-defined lifecycle. That is what
ipn/ipnext provides.
Per-LocalBackend state matters because a single process can have more
than one LocalBackend alive at once: a tsnet program can host many
tsnet.Server nodes concurrently, each with its own LocalBackend.
Package-global variables in a feature package would be shared across all
of them, which should be avoided unless it is otherwise infeasible. An
ipnext.Extension
is instantiated once per LocalBackend, so each node gets its own
copy of the extension's state.
(A related, not-yet-realized goal is letting different tsnet nodes in
the same process opt in to different sets of hooks. Today the hook
registration is still process-global (the init runs once and the hook
is on for every LocalBackend), but the extension design leaves room to
make this per-node in the future. Don't rely on the process-global
behavior; keep state on the extension instance, not in globals.)
An ipnext.Extension is created once per LocalBackend. From init, a
feature package registers a factory:
func init() {
feature.Register("taildrop")
ipnext.RegisterExtension("taildrop", newExtension)
}
When LocalBackend.Start runs, it instantiates each registered
extension and calls its
Init(host ipnext.Host) error
method. Init is where the extension wires up its per-backend hooks
against host.Hooks():
func (e *extension) Init(h ipnext.Host) error {
h.Hooks().ProfileStateChange.Add(e.onChangeProfile)
h.Hooks().OnSelfChange.Add(e.onSelfChange)
h.Hooks().MutateNotifyLocked.Add(e.setNotifyFilesWaiting)
h.Hooks().SetPeerStatus.Add(e.setPeerStatus)
h.Hooks().BackendStateChange.Add(e.onBackendStateChange)
return nil
}
ipnext.Hooks
is a struct of feature.Hook/feature.Hooks fields
covering LocalBackend's extension points: backend and profile state
changes, netmap toggles, self-node changes, notify mutation, peer
status, packet-filter hooks, audit logging, and more. Read the type in
ipn/ipnext/ipnext.go for the current list; new hooks are added there
as needed.
Extension hooks run synchronously with the triggering LocalBackend
operation and can influence its outcome. Because multiple extensions may
touch the same shared state (prefs, active profile, exit node, etc.),
new hooks should be designed with a clear conflict-resolution story
rather than assuming a single writer.
To get back to your extension from a LocalBackend inside a LocalAPI or
C2N handler, use
ipnlocal.GetExt[*yourExtType](b).
Canonical examples:
feature/taildrop/ext.go: full extension with per-backend state.feature/acme/acme.go: extension that alsoSets a handful ofipnlocal.Hook*singletons.
The ipnext mechanism is a work in progress and not every part of
LocalBackend has been converted to hooks yet. If you need a new
extension point, ask.
Tests: featuretags and depaware / depchecker
Tests come in three flavors:
-
feature/featuretagsself-tests (featuretags_test.go) validate the registry itself: that every declared dependency exists, that there are no cycles, thatRequires/RequiredByreturn the expected sets, and that everyts_omit_*string that appears anywhere in the tree (viagit grep) is declared in the map. -
depaware(github.com/tailscale/depaware) snapshots the full set of Go packages linked into each binary into a checked-indepaware.txtfile. Its job is right there in the name: to force us to be aware of our deps. Every change to the dependency graph shows up as a diff todepaware.txtin the same PR that introduced it, so reviewers can see, prominently, exactly what got pulled in or dropped. We use this to have an auditable history of the dependency footprint of key programs and libraries. We track several flavors:- full
tailscaledon all GOOSes - full
tailscaleCLI on all GOOSes cmd/derper- minimal
tailscaled+CLI (depaware-min.txt,depaware-minbox.txt) tsnet(has its owndepaware.txt)k8s-operator, which usestsnet
When reviewing a PR, look at the depaware diff to see what got pulled in or dropped, and make sure the changes make sense.
- full
-
deptest.DepCheckerlets you lock down omissions once you've achieved them. Rather than hoping nobody accidentally adds ataildropimport back into ats_omit_taildropbuild, you write:func TestOmitACME(t *testing.T) { deptest.DepChecker{ GOOS: "linux", GOARCH: "amd64", Tags: "ts_omit_acme,ts_include_cli", OnDep: func(dep string) { if strings.Contains(dep, "/acme") { t.Errorf("unexpected dep with ts_omit_acme: %q", dep) } }, }.Check(t) }cmd/tailscaled/deps_test.gohas many examples; add new ones there or wherever they fit.tsnethas its ownTestDepsintsnet_test.gowith aBadDepsmap asserting that things likefeature/remoteconfig,feature/syspolicy, andx/crypto/sshnever sneak intotsnet.
Tooling
cmd/featuretagsconstructs valid sets of Go omit build tags. Start from a minimal build and add features with--min --add=..., or start from a full build and remove with--remove=...; either way it respects the declared dependency DAG.build_dist.shusescmd/featuretagsunder the hood, so you do not need to hand-maintain its build-tag lists as new features are added.
Summary of what to do for a new feature
- Add an entry to
Featuresinfeature/featuretags/featuretags.go, including anyDeps. - Run
./tool/go generate ./feature/buildfeatures(ormake generate). - Create
feature/<name>/with the code, and register hooks and anyipnext.Extensionfrom itsinit. Callfeature.Register("<name>")frominittoo. - If the feature should be on by default in
tailscaled, addfeature/condregister/maybe_<name>.gowith//go:build !ts_omit_<name>and a blank import oftailscale.com/feature/<name>. - If
tsnetshould also link it, add an equivalentmaybe_<name>.goundertsnet/. Otherwise rememberbuildfeatures.HasFooalone is not enough; pair it withfeature.IsRegistered("foo"). - Add a
deptest.DepCheckertest to lock down what does not get linked when the feature is omitted. - Regenerate depaware and review the diff.
Asking questions
If something in this document is unclear, if you are not sure whether your work should be modular, or if you need a new extension point:
- Tailscale employees: ask in the
#clientSlack channel. - Open source contributors: file a public GitHub issue with your proposal or question at github.com/tailscale/tailscale/issues.