mirror of
https://github.com/tailscale/tailscale.git
synced 2026-07-29 08:46:33 -04:00
f6fa294635
Previously there was a mismatch between how nodes store AUMs and what the control plane would offer during sync: - Client compaction: Nodes aggressively compact their TKA state -- they keep the last 24 AUMs, every AUM received in the last two weeks, and then everything from there back to the last checkpoint. Depending on when it compacts, a node may only have ~50 AUMs. - Exponential sampling: To save bandwidth, the control plane would send a SyncOffer containing ancestors at exponentially increasing intervals (4th, 16th, 64th, 256th...). If a node has been offline for too long, the exponential sampling skips the node's smaller window. When the SyncOffer and local state are disjoint, the node cannot find a common ancestor to use for synchronisation. It enters a failure loop where it keeps polling for new TKA state, but it cannot catch up and has an increasingly-outdated view of the tailnet. This patch replaces the exponential sampling with a SyncOffer that sends every checkpoint ancestor of the current HEAD. Since every node is guaranteed to keep at least one checkpoint after compaction, we're more likely to have an intersection for the sync process. This patch also increases `maxSyncHeadIntersectionIter`, which in practice means the control plane will send every checkpoint in the current chain. This means all affected nodes will be able to find an intersection and catch up immediately, without requiring a client update. It's still possible for a node to be unable to sync, but these edge cases become less likely with this change. (For example, if a node is 1000+ AUMs behind, or if it creates a local branch and then compacts away the intersection with the main chain.) This patch includes a regression test with synthetic data, and I verified the fix with customer data. Updates https://github.com/tailscale/corp/issues/40404 Change-Id: I2174011bb23a2b5972f6d1591aadcc016e3cba35 Signed-off-by: Alex Chan <alexc@tailscale.com>
36 lines
930 B
Go
36 lines
930 B
Go
// Copyright (c) Tailscale Inc & contributors
|
|
// SPDX-License-Identifier: BSD-3-Clause
|
|
|
|
package tka
|
|
|
|
import (
|
|
"time"
|
|
)
|
|
|
|
const (
|
|
// Upper bound on checkpoint elements, chosen arbitrarily. Intended
|
|
// to cap the size of large AUMs.
|
|
maxDisablementValues = 32
|
|
maxKeys = 512
|
|
|
|
// Max amount of metadata that can be associated with a key, chosen arbitrarily.
|
|
// Intended to avoid people abusing TKA as a key-value score.
|
|
maxMetaBytes = 512
|
|
|
|
// Max iterations searching for any intersection during the sync process.
|
|
maxSyncIter = 2000
|
|
|
|
// Max iterations searching for a head intersection during the sync process.
|
|
maxSyncHeadIntersectionIter = 1000
|
|
|
|
// Limit on scanning AUM trees, chosen arbitrarily.
|
|
maxScanIterations = 2000
|
|
)
|
|
|
|
var (
|
|
CompactionDefaults = CompactionOptions{
|
|
MinChain: 24, // Keep at minimum 24 AUMs since head.
|
|
MinAge: 14 * 24 * time.Hour, // Keep 2 weeks of AUMs.
|
|
}
|
|
)
|