Reconnect paired peers after an in-process restart (#3106)

Close the iroh endpoint on networking shutdown, bound every reconnection
dial, give the deferring side of the EndpointId rule a grace period after
which it dials anyway, and re-arm the periodic sweep for any device still
unconnected.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
This commit is contained in:
Jamie Pineandcapy-ai-staging[bot] authored and GitHub committed 2026-10-04 23:53:45 +00:00
1 parent 46a54caf3b
commit 09d7ea2ca9
7 files changed
+371 -149

No files matched your search

+1
View File
@@ -199,6 +199,7 @@ Recent landed.
| CI green on `sources` | landed 2026-10-04 (Fable, SPAC-4); Rust Formatting, Clippy and TypeScript green on #3102 | Fable | The three `ci.yml` jobs were red on `sources` before any PR: the self-hosted runner's preinstalled toolchain had no rustfmt or clippy (`setup-rust` now installs both every run, and `rust-toolchain.toml` names them), `maximize-build-space` left the Blacksmith runner's root disk with 100 MB so no toolchain could install (step removed), the clippy job ran on GitHub's ubuntu-22.04 whose FFmpeg 4.4 cannot build `sd-ffmpeg` (now `blacksmith-4vcpu-ubuntu-2404`), and the server and desktop crates failed to compile without a built web UI (debug builds now get an empty dist with a warning; release builds still refuse), `cargo clippy --workspace -D warnings` failed in seven crates plus the Tauri sidecar check and the two platform-only file-opening crates, and `bun run typecheck` in `apps/tauri` had 30 errors in `packages/interface`. All fixed with no product change: sqlx row aliases in `sd-store`, exact casts and raw FFI borrows in `sd-ffmpeg`, `DaemonRequest.device` in `sd-mobile-core`, target-gated `file-opening-macos`/`-windows`, a Tauri build script that drops `externalBin` when no daemon sidecar exists, unused imports and `SourceInfo.adapter_id` nullability in the interface, the `/redundancy/duplicates` route the dashboard already linked. `core/tests/delete_strategy_test.rs` compiles again (calls `ops::files::trash::trash` directly), the 13 stale doctests compile, `scripts/setup.sh` installs the libav dev packages, bun is pinned to 1.3.4 with `--frozen-lockfile`, workspace `rust-version` is 1.95 (gpui's floor). Still red and out of this scope: 20 integration-test failures on a bare Linux VM (see SPAC-2); the three unit-test bugs have their own row below. Next: confirm the three jobs green on the PR, then decide whether `core_tests.yml` should run on pull requests. |
| Unit-test bugs on `sources` | fixed 2026-10-04 (Fable, SPAC-5); `cargo test -p sd-core --lib` green five runs in a row with default parallelism | Fable | Three bugs the SPAC-2 baseline found. The btrfs parser read the filesystem UUID from a line starting with `uuid:`, but `btrfs filesystem show` prints it on the `Label:` line, so every Btrfs volume failed with "Could not parse Btrfs UUID"; the header line now yields both label and UUID. The plus-code parser rejected its own encoder's output and Google's `WR2C+2C Bibra Lake` form (its alphabet check demanded every character equal every alphabet digit); `PlusCode::is_valid` now follows the Open Location Code validation rules, the encoder is unchanged. The `ops::files` fixture flakes were not an FTS5 or temp-path problem: `LibraryManager`'s directory watcher auto-opens a new `.sdlibrary` folder 500 ms after it appears, and under load `create_library` was still initializing it, so two openers migrated one database at once (`seaql_migrations` UNIQUE failures, `vtable constructor failed: search_index`, `AlreadyInUse`). Paths a create call is building are now registered and the watcher leaves them alone. Next: the integration-test triage (needs a Mac for the pairing suites). |
| Linux integration-test triage | in progress 2026-10-04 (Fable, SPAC-6); 7 of the 13 red `core/tests` targets green on a bare Linux VM | Fable | Each of the 20 failures from SPAC-2 was reproduced on 61dea60 and classed. Stale assertions fixed with no product change: `library_test` expected a `sidecars` dir that ba8efcd stopped creating (now checks `previews` and `exports`), `event_filtering_test` listed the subfolder itself as a "subdirectory only" path (exact mode matches direct children since cfa31cc), `sync_event_log_test` read the newest-first query as oldest-first and raced the startup pruner, `sync_metrics_test` assumed no change arrives before the first explicit action (library open broadcasts the device record), the two volume capacity tests compared ext4 usable bytes to the image size. Environment assumptions made hermetic: `resource_events_test` indexes a seeded temp dir instead of `~/Desktop` and asserts on the file batch, `volume_tracking_test` builds a loop-backed volume when the machine has no user-visible mount (Linux hides `/`), skipping only without sudo. Still red, product bugs rather than tests: pull transfers compare raw blake3 against the sampled content hash (`copy/strategy.rs:705` vs `file_transfer.rs:1252`, every pull fails checksum), the Linux fs-watcher turns a move out of the watched root into a Modify instead of a Remove (`platform/linux.rs:83`, `watcher_test` is right), two cores sharing a hostname collide on `devices.slug` during backfill (`sync_backfill_test`), and after an in-process restart neither peer reconnects (`file_transfer_with_restart_test`, which also needs `core.shutdown()` rather than `drop` to release `secrets.redb`). Next: brief fixes for the four bugs; decide whether `core_tests.yml` runs on PRs (see `reports/SPAC-6.md` in the captain drive). |
| Peer reconnect after restart | in progress 2026-10-04 (Fable, SPAC-8); `file_transfer_with_restart_test` green on a Linux VM | Fable | Bug 4 from the triage. Two causes. `NetworkingService::shutdown` never closed the iroh endpoint, so an in-process restart left the old socket alive with the same node id: the peer never saw a disconnect and the restarted core's dials by node id timed out against its own ghost. Shutdown now aborts the periodic reconnection task and closes the endpoint, which sends CONNECTION_CLOSE to every peer. The reconnection loop was bounded (10 dials, 5 s apart) with no bound on a single dial, the device with the higher EndpointId never dialed at all, and the periodic sweep ignored any device seen within five minutes. `service/network/core/reconnect.rs` now decides the dial role (lower EndpointId initiates, the other side waits a 15 s grace period and then dials anyway) and the backoff (2 s doubling to 30 s, 12 attempts), each dial is bounded by a 30 s timeout, the loop stops once the registry shows the device connected, one loop per device is enforced, and the 30 s sweep re-arms any device still unconnected. Next: live proof between a Mac and the NAS; the suite now runs in `cargo xtask test-core`. |
| Pairing bugs from the integration triage | fixed 2026-10-04 (Fable, SPAC-7); `file_copy_pull_test`, `watcher_test`, `sync_backfill_test` green on Linux | Fable | Three product bugs SPAC-6 located. Pull transfers always failed verification: the receiver hashed every byte with blake3 while the sender sent the 16-hex sampled content hash, and the push receiver never verified at all; `TransferComplete.final_checksum` is now the full blake3 of the file bytes in both directions (`ContentHashGenerator::generate_integrity_hash`, computed while streaming on the sender), compared on both receivers, with a corrupted file deleted and reported as `TransferError`. The Linux watcher turned a move out of the watched root into a Modify of a path that no longer existed: `RawEventKind::Rename` now carries inotify's From/To/Both, a lone MOVED_FROM becomes Remove and a lone MOVED_TO becomes Create after a 200 ms pairing window, and a buffered Modify is dropped when its path is removed. Two devices with the same hostname collided on `devices.slug` during sync apply (collision handling existed on insert only, not on update), so backfill failed every 5 s forever; the slug now gets a suffix from the device id on both paths, and three consecutive backfill failures log an error and pause sync instead of looping. `test_volume_resource_events_on_sync` now accepts the coarse `Refresh` that #3061 made backfill emit. Still red in `sd-fs-watcher --lib` and out of scope: `test_file_deletion_events` reads the write's buffered Modify before the Remove, and `a_shallow_watch_delivers_events_for_immediate_children` is filtered because `tempfile` dirs contain `.tmp`. Next: bug 4 (no reconnect after in-process restart) in its own thread; decide whether `core_tests.yml` runs on pull requests. |
## Product sequence
+16 -8
View File
@@ -87,6 +87,9 @@ pub struct NetworkingEventLoop {
/// Nodes that already have connection watchers spawned (to prevent duplicates)
watched_nodes: Arc<RwLock<std::collections::HashSet<EndpointId>>>,
/// Devices with a reconnection loop in flight (shared with the service)
reconnecting: Arc<RwLock<std::collections::HashSet<Uuid>>>,
/// Logger for event loop operations
logger: Arc<dyn NetworkLogger>,
}
@@ -102,6 +105,7 @@ impl NetworkingEventLoop {
active_connections: Arc<
RwLock<std::collections::HashMap<(EndpointId, Vec<u8>), Connection>>,
>,
reconnecting: Arc<RwLock<std::collections::HashSet<Uuid>>>,
logger: Arc<dyn NetworkLogger>,
) -> Self {
let (command_tx, command_rx) = mpsc::unbounded_channel();
@@ -118,6 +122,7 @@ impl NetworkingEventLoop {
shutdown_tx,
identity,
active_connections,
reconnecting,
watched_nodes: Arc::new(RwLock::new(std::collections::HashSet::new())),
logger,
}
@@ -653,18 +658,21 @@ impl NetworkingEventLoop {
{
let command_sender = Some(self.command_tx.clone());
let endpoint = Some(self.endpoint.clone());
let device_registry = self.device_registry.clone();
let reconnecting = self.reconnecting.clone();
let logger = self.logger.clone();
// Spawn reconnection with a small delay to prevent immediate retry loops
tokio::spawn(async move {
crate::service::network::core::NetworkingService::attempt_device_reconnection(
device_id,
persisted_device,
command_sender,
endpoint,
logger,
)
.await;
device_id,
persisted_device,
command_sender,
endpoint,
device_registry,
reconnecting,
logger,
)
.await;
});
}
}
+180 -140
View File
@@ -1,6 +1,7 @@
//! Core networking engine with Iroh P2P
pub mod event_loop;
pub mod reconnect;
use crate::device::DeviceManager;
use crate::service::network::{
@@ -17,6 +18,7 @@ use tokio::sync::{broadcast, mpsc, RwLock};
use uuid::Uuid;
pub use event_loop::{EventLoopCommand, NetworkingEventLoop};
pub use reconnect::{dial_role, DialRole, ReconnectSchedule};
/// Protocol ALPN identifiers
pub const PAIRING_ALPN: &[u8] = b"spacedrive/pairing/1";
@@ -112,6 +114,14 @@ pub struct NetworkingService {
/// Nodes that already have connection watchers spawned (to prevent duplicates)
watched_nodes: Arc<RwLock<std::collections::HashSet<EndpointId>>>,
/// Devices with a reconnection loop in flight, so the periodic sweep and
/// connection-lost handling never run two dial loops for one device
reconnecting: Arc<RwLock<std::collections::HashSet<Uuid>>>,
/// Long-running tasks (periodic reconnection) aborted on shutdown so they
/// release their endpoint clones
background_tasks: Arc<RwLock<Vec<tokio::task::JoinHandle<()>>>>,
/// Sync multiplexer for routing sync messages to correct library
sync_multiplexer: Arc<SyncMultiplexer>,
@@ -165,6 +175,8 @@ impl NetworkingService {
event_sender,
active_connections: Arc::new(RwLock::new(std::collections::HashMap::new())),
watched_nodes: Arc::new(RwLock::new(std::collections::HashSet::new())),
reconnecting: Arc::new(RwLock::new(std::collections::HashSet::new())),
background_tasks: Arc::new(RwLock::new(Vec::new())),
sync_multiplexer,
logger,
})
@@ -297,6 +309,7 @@ impl NetworkingService {
self.event_sender.clone(),
self.identity.clone(),
self.active_connections.clone(),
self.reconnecting.clone(),
self.logger.clone(),
);
@@ -362,15 +375,18 @@ impl NetworkingService {
for (device_id, persisted_device) in auto_reconnect_devices {
let command_sender = self.command_sender.clone();
let endpoint = self.endpoint.clone();
let device_registry = self.device_registry.clone();
let reconnecting = self.reconnecting.clone();
let logger = self.logger.clone();
// Spawn a background task for each device reconnection
tokio::spawn(async move {
Self::attempt_device_reconnection(
device_id,
persisted_device,
command_sender,
endpoint,
device_registry,
reconnecting,
logger,
)
.await;
@@ -378,25 +394,31 @@ impl NetworkingService {
}
}
/// Attempt to reconnect to a specific device
async fn attempt_device_reconnection(
/// Run one bounded dial loop for a paired device.
///
/// The lower EndpointId dials first and the other side waits a grace
/// period, which avoids both sides racing into duplicate connections while
/// still guaranteeing that one side dials even when the initiator's retry
/// state was lost (see `reconnect`). Each dial is bounded by a timeout so a
/// hung discovery never stalls the schedule, and the loop stops as soon as
/// the registry shows the device connected, whichever side made it so.
pub(crate) async fn attempt_device_reconnection(
device_id: Uuid,
persisted_device: crate::service::network::device::PersistedPairedDevice,
command_sender: Option<tokio::sync::mpsc::UnboundedSender<EventLoopCommand>>,
endpoint: Option<Endpoint>,
device_registry: Arc<RwLock<DeviceRegistry>>,
reconnecting: Arc<RwLock<std::collections::HashSet<Uuid>>>,
logger: Arc<dyn NetworkLogger>,
) {
// Deterministic reconnection: only the device with the lower EndpointId initiates
// This prevents both sides from simultaneously trying to connect
let endpoint_ref = match &endpoint {
Some(ep) => ep,
None => {
let (endpoint, sender) = match (endpoint, command_sender) {
(Some(endpoint), Some(sender)) => (endpoint, sender),
_ => {
logger.warn("No endpoint available for reconnection").await;
return;
}
};
let my_node_id = endpoint_ref.id();
let remote_node_id = match persisted_device
.device_info
.network_fingerprint
@@ -412,174 +434,178 @@ impl NetworkingService {
}
};
// Deterministic rule: only device with lower EndpointId initiates outbound connections
// This prevents both sides from creating competing connections
if my_node_id > remote_node_id {
if !reconnecting.write().await.insert(device_id) {
logger
.debug(&format!(
"Skipping outbound reconnection to {} - waiting for them to connect to us (EndpointId rule: {} > {})",
persisted_device.device_info.device_name,
my_node_id,
remote_node_id
"Reconnection to {} already in progress, not starting another",
persisted_device.device_info.device_name
))
.await;
return;
}
let mut schedule = ReconnectSchedule::new(dial_role(&endpoint.id(), &remote_node_id));
logger
.info(&format!(
"EndpointId rule: {} < {} - we should initiate connection",
my_node_id, remote_node_id
"Reconnecting to {} as {:?} (EndpointId rule: {} vs {}), first dial in {:?}",
persisted_device.device_info.device_name,
schedule.role(),
endpoint.id(),
remote_node_id,
schedule.initial_delay()
))
.await;
tokio::time::sleep(schedule.initial_delay()).await;
logger
.info(&format!(
"Starting reconnection attempts for device: {}",
device_id
))
loop {
let already_connected = matches!(
device_registry.read().await.get_device_state(device_id),
Some(crate::service::network::device::DeviceState::Connected { .. })
);
if already_connected || endpoint.is_closed() {
logger
.debug(&format!(
"Stopping reconnection loop for device {} (connected or endpoint closed)",
device_id
))
.await;
break;
}
let node_addr = EndpointAddr::new(remote_node_id);
let dial = tokio::time::timeout(
reconnect::CONNECT_TIMEOUT,
endpoint.connect(node_addr, MESSAGING_ALPN),
)
.await;
if let (Some(endpoint), Some(sender)) = (endpoint, command_sender) {
// Try to parse node ID from the persisted device
if let Ok(node_id) = persisted_device
.device_info
.network_fingerprint
.node_id
.parse::<EndpointId>()
{
// Build EndpointAddr - Iroh will discover addresses automatically
let node_addr = EndpointAddr::new(node_id);
match dial {
Ok(Ok(conn)) => {
logger
.info(&format!("Successfully connected to device {}", device_id))
.await;
let _ = sender.send(EventLoopCommand::TrackOutboundConnection {
node_id: remote_node_id,
conn,
});
let _ = sender.send(EventLoopCommand::ConnectionEstablished {
device_id,
node_id: remote_node_id,
});
break;
}
Ok(Err(e)) => {
logger
.info(&format!(
"Connection attempt {} failed for device {}: {}",
schedule.attempts() + 1,
device_id,
e
))
.await;
}
Err(_) => {
logger
.info(&format!(
"Connection attempt {} timed out for device {}",
schedule.attempts() + 1,
device_id
))
.await;
}
}
// Attempt connection with retries to give discovery time to work
let mut retry_count = 0;
let max_retries = 10;
let retry_delay = tokio::time::Duration::from_secs(5);
loop {
// Use MESSAGING_ALPN for reconnection to paired devices
match endpoint.connect(node_addr.clone(), MESSAGING_ALPN).await {
Ok(conn) => {
logger
.info(&format!("Successfully connected to device {}", device_id))
.await;
// Track this outbound connection so it persists
let _ = sender.send(EventLoopCommand::TrackOutboundConnection {
node_id,
conn: conn.clone(),
});
logger
.info(&format!("Connection established to device {}", device_id))
.await;
// Send connection established command
let _ = sender.send(EventLoopCommand::ConnectionEstablished {
device_id,
node_id,
});
break;
}
Err(e) => {
retry_count += 1;
if retry_count >= max_retries {
logger
.error(&format!(
"Failed to connect to device {} after {} attempts: {}",
device_id, max_retries, e
))
.await;
break;
} else {
logger
.info(&format!(
"Connection attempt {} of {} failed for device {}, retrying in {:?}...",
retry_count, max_retries, device_id, retry_delay
))
.await;
tokio::time::sleep(retry_delay).await;
}
}
}
match schedule.next_delay() {
Some(delay) => tokio::time::sleep(delay).await,
None => {
logger
.warn(&format!(
"Giving up on device {} after {} attempts; the periodic sweep will retry",
device_id,
schedule.attempts()
))
.await;
break;
}
}
}
reconnecting.write().await.remove(&device_id);
}
/// Start periodic reconnection attempts for disconnected devices
///
/// This is the safety net behind the event-driven loop: a device whose
/// dial loop exhausted, or whose peer vanished without a clean close, gets
/// a fresh schedule on every tick while it stays unconnected.
async fn start_periodic_reconnection(&self) {
let device_registry = self.device_registry.clone();
let command_sender = self.command_sender.clone();
let endpoint = self.endpoint.clone();
let reconnecting = self.reconnecting.clone();
let logger = self.logger.clone();
tokio::spawn(async move {
let task = tokio::spawn(async move {
let mut interval = tokio::time::interval(tokio::time::Duration::from_secs(30));
loop {
interval.tick().await;
// Get disconnected devices that should be reconnected
if let Ok(auto_reconnect_devices) = {
let auto_reconnect_devices = {
let registry = device_registry.read().await;
registry.get_auto_reconnect_devices().await
} {
// Only attempt reconnection for devices we haven't seen recently
let now = chrono::Utc::now();
for (device_id, persisted_device) in auto_reconnect_devices {
// Skip if device was seen recently (within last 5 minutes)
if let Some(last_connected) = persisted_device.last_connected_at {
if now.signed_duration_since(last_connected)
< chrono::Duration::minutes(5)
{
continue;
}
}
// Every paired device loads as Paired and only becomes
// Disconnected after a session drops, so a peer that was
// unreachable when this device started is still Paired and
// needs dialing just the same.
let is_unconnected = {
let registry = device_registry.read().await;
if let Some(device_state) = registry.get_device_state(device_id) {
matches!(
device_state,
crate::service::network::device::DeviceState::Paired { .. }
| crate::service::network::device::DeviceState::Disconnected { .. }
)
} else {
true // Not in registry, try to reconnect
}
};
if is_unconnected {
logger
.info(&format!(
"Attempting periodic reconnection to device: {}",
device_id
))
.await;
let cmd_sender = command_sender.clone();
let ep = endpoint.clone();
let logger_clone = logger.clone();
tokio::spawn(async move {
Self::attempt_device_reconnection(
device_id,
persisted_device,
cmd_sender,
ep,
logger_clone,
)
.await;
});
}
match registry.get_auto_reconnect_devices().await {
Ok(devices) => devices,
Err(_) => continue,
}
};
for (device_id, persisted_device) in auto_reconnect_devices {
// Every paired device loads as Paired and only becomes
// Disconnected after a session drops, so a peer that was
// unreachable when this device started is still Paired and
// needs dialing just the same.
let is_unconnected = {
let registry = device_registry.read().await;
match registry.get_device_state(device_id) {
Some(state) => matches!(
state,
crate::service::network::device::DeviceState::Paired { .. }
| crate::service::network::device::DeviceState::Disconnected { .. }
),
None => true,
}
};
if !is_unconnected || reconnecting.read().await.contains(&device_id) {
continue;
}
logger
.info(&format!(
"Attempting periodic reconnection to device: {}",
device_id
))
.await;
let cmd_sender = command_sender.clone();
let ep = endpoint.clone();
let registry = device_registry.clone();
let in_flight = reconnecting.clone();
let logger_clone = logger.clone();
tokio::spawn(async move {
Self::attempt_device_reconnection(
device_id,
persisted_device,
cmd_sender,
ep,
registry,
in_flight,
logger_clone,
)
.await;
});
}
}
});
self.background_tasks.write().await.push(task);
}
/// Start periodic health checks for connected devices
@@ -846,6 +872,20 @@ impl NetworkingService {
// Wait a bit for graceful shutdown
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
}
for task in self.background_tasks.write().await.drain(..) {
task.abort();
}
// Closing the endpoint sends CONNECTION_CLOSE to every peer so they
// see the drop at once instead of at QUIC idle timeout, and stops the
// old socket answering for our node id once a new endpoint takes over
// (an in-process restart with the same identity).
if let Some(endpoint) = &self.endpoint {
endpoint.close().await;
self.active_connections.write().await.clear();
self.logger.info("Endpoint closed").await;
}
Ok(())
}
+146
View File
@@ -0,0 +1,146 @@
//! # Reconnection policy
//!
//! Decides which side of a paired pair dials after a connection drops and how
//! the dial attempts are spaced. The device with the lower EndpointId is the
//! initiator and dials at once, so the two sides do not race each other into
//! duplicate connections. The other side defers, but only for a grace period:
//! if the initiator lost its retry state (it restarted too, or its dial cannot
//! reach us) the deferrer dials anyway rather than waiting forever.
//!
//! Retries back off exponentially up to a cap and stop after a bounded number
//! of attempts; the periodic sweep in the networking service re-arms a fresh
//! schedule for any device that is still disconnected.
use std::time::Duration;
use iroh::EndpointId;
/// How long the deferring side waits for the initiator before dialing itself.
pub const DEFER_GRACE: Duration = Duration::from_secs(15);
/// Upper bound on a single dial, so a hung discovery cannot stall the schedule.
pub const CONNECT_TIMEOUT: Duration = Duration::from_secs(30);
const BASE_DELAY: Duration = Duration::from_secs(2);
const MAX_DELAY: Duration = Duration::from_secs(30);
const MAX_ATTEMPTS: u32 = 12;
/// Which side of a pair is expected to dial first.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum DialRole {
Initiator,
Deferrer,
}
/// The lower EndpointId initiates; exactly one side of any pair gets this role.
pub fn dial_role(me: &EndpointId, remote: &EndpointId) -> DialRole {
if me < remote {
DialRole::Initiator
} else {
DialRole::Deferrer
}
}
/// Retry schedule for one device's reconnection loop.
#[derive(Debug, Clone)]
pub struct ReconnectSchedule {
role: DialRole,
attempts: u32,
}
impl ReconnectSchedule {
pub fn new(role: DialRole) -> Self {
Self { role, attempts: 0 }
}
pub fn role(&self) -> DialRole {
self.role
}
/// Delay before the first dial: none for the initiator, the grace period
/// for the deferrer.
pub fn initial_delay(&self) -> Duration {
match self.role {
DialRole::Initiator => Duration::ZERO,
DialRole::Deferrer => DEFER_GRACE,
}
}
/// Records a failed dial and returns how long to wait before the next one,
/// or `None` once the schedule is exhausted.
pub fn next_delay(&mut self) -> Option<Duration> {
self.attempts += 1;
if self.attempts >= MAX_ATTEMPTS {
return None;
}
let exponent = (self.attempts - 1).min(16);
let delay = BASE_DELAY.saturating_mul(1u32 << exponent);
Some(delay.min(MAX_DELAY))
}
pub fn attempts(&self) -> u32 {
self.attempts
}
}
#[cfg(test)]
mod tests {
use super::*;
use iroh::SecretKey;
fn ids() -> (EndpointId, EndpointId) {
let a = SecretKey::from_bytes(&[1u8; 32]).public();
let b = SecretKey::from_bytes(&[2u8; 32]).public();
if a < b {
(a, b)
} else {
(b, a)
}
}
#[test]
fn exactly_one_side_initiates() {
let (low, high) = ids();
assert_eq!(dial_role(&low, &high), DialRole::Initiator);
assert_eq!(dial_role(&high, &low), DialRole::Deferrer);
}
#[test]
fn initiator_dials_at_once_and_deferrer_waits_for_grace() {
assert_eq!(
ReconnectSchedule::new(DialRole::Initiator).initial_delay(),
Duration::ZERO
);
assert_eq!(
ReconnectSchedule::new(DialRole::Deferrer).initial_delay(),
DEFER_GRACE
);
}
#[test]
fn backoff_grows_then_caps_then_exhausts() {
let mut schedule = ReconnectSchedule::new(DialRole::Initiator);
let mut delays = Vec::new();
while let Some(delay) = schedule.next_delay() {
delays.push(delay);
}
assert_eq!(delays.len() as u32, MAX_ATTEMPTS - 1);
assert_eq!(delays[0], Duration::from_secs(2));
assert_eq!(delays[1], Duration::from_secs(4));
assert_eq!(delays[2], Duration::from_secs(8));
assert_eq!(delays[3], Duration::from_secs(16));
assert!(delays[4..].iter().all(|d| *d == MAX_DELAY));
assert_eq!(schedule.attempts(), MAX_ATTEMPTS);
let total: Duration = delays.iter().sum();
assert!(total <= Duration::from_secs(5 * 60));
}
#[test]
fn deferrer_eventually_dials_within_bounded_time() {
let mut schedule = ReconnectSchedule::new(DialRole::Deferrer);
let first_dial_at = schedule.initial_delay();
assert!(first_dial_at <= Duration::from_secs(30));
assert!(schedule.next_delay().is_some());
}
}
+15 -1
View File
@@ -384,9 +384,23 @@ async fn bob_restart_scenario() {
println!("Bob: Waiting for Alice to restart and send files...");
// Create directory for received files
// Create directory for received files and allow it as a transfer destination
let received_dir = std::path::Path::new("/tmp/received_files_restart");
std::fs::create_dir_all(received_dir).unwrap();
if let Some(networking) = core.networking() {
let protocol_registry = networking.protocol_registry();
let registry = protocol_registry.read().await;
if let Some(handler) = registry.get_handler("file_transfer") {
if let Some(ft_handler) =
handler
.as_any()
.downcast_ref::<sd_core::service::network::protocol::FileTransferProtocolHandler>(
) {
ft_handler.add_allowed_path(received_dir.to_path_buf());
println!("Bob: Added {} as allowed path", received_dir.display());
}
}
}
// Wait for expected files list
let expected_files = loop {
+9
View File
@@ -133,6 +133,15 @@ unknown connection state until it can query the job again.
Remote reads fail with explicit availability. Cached metadata can remain
browsable, but the interface does not pretend that unavailable bytes are local.
Paired devices reconnect on their own. The device with the lower endpoint ID
dials as soon as a connection drops or the daemon starts; the other side waits
15 seconds for that dial and then dials itself, so a peer whose retry state was
lost (its own restart, a dial that cannot reach us) does not leave the pair
stuck. Each dial has a 30 second bound, retries back off from 2 seconds to 30
seconds for 12 attempts, one loop runs per device, and a 30 second sweep
re-arms any device still unconnected. Shutdown closes the endpoint so peers see
the drop at once rather than at the QUIC idle timeout.
## Observability
Status and logs expose:
+4
View File
@@ -88,6 +88,10 @@ pub const CORE_TESTS: &[TestSuite] = &[
name: "File transfer test",
test_args: &["--test", "file_transfer_test"],
},
TestSuite {
name: "File transfer with restart test",
test_args: &["--test", "file_transfer_with_restart_test"],
},
TestSuite {
name: "Cross device copy test",
test_args: &["--test", "cross_device_copy_test"],