spacedrive/docs/core/testing.mdx

---
title: Testing
sidebarTitle: Testing
---

Testing in Spacedrive Core ensures reliability across single-device operations and multi-device networking scenarios. This guide covers the available frameworks, patterns, and best practices.

## Testing Infrastructure

Spacedrive Core provides two primary testing approaches:

1. **Standard Tests** - For unit and single-core integration testing
2. **Subprocess Framework** - For multi-device networking and distributed scenarios

### Test Organization

Tests live in two locations:
- `core/tests/` - Integration tests that verify complete workflows
- `core/src/testing/` - Test framework utilities and helpers

## Standard Testing

For single-device tests, use Tokio's async test framework:

```rust
#[tokio::test]
async fn test_library_creation() {
    let setup = IntegrationTestSetup::new("library_test").await.unwrap();
    let core = setup.create_core().await.unwrap();
    
    let library = core.libraries
        .create_library("Test Library", None)
        .await
        .unwrap();
    
    assert!(!library.id.is_empty());
}
```

### Integration Test Setup

The `IntegrationTestSetup` utility provides isolated test environments:

```rust
// Basic setup
let setup = IntegrationTestSetup::new("test_name").await?;

// Custom configuration
let setup = IntegrationTestSetup::with_config("test_name", |builder| {
    builder
        .log_level("debug")
        .networking_enabled(true)
        .volume_monitoring_enabled(false)
}).await?;
```

Key features:
- Isolated temporary directories per test
- Structured logging to `test_data/{test_name}/library/logs/`
- Automatic cleanup on drop
- Configurable app settings

## Multi-Device Testing

Spacedrive provides two approaches for testing multi-device scenarios:

### When to Use Subprocess Framework

**Use `CargoTestRunner` subprocess framework when:**
- Testing **real networking** with actual network discovery, NAT traversal, and connections
- Testing **device pairing** workflows that require independent network stacks
- Scenarios need **true process isolation** (separate memory spaces, different ports)
- You want to test network reconnection, timeout, and failure handling
- Testing cross-platform network behavior

**Examples:** Device pairing, network discovery, connection management

```rust
// Uses real networking, separate processes
let mut runner = CargoTestRunner::new()
    .add_subprocess("alice", "alice_pairing_scenario")
    .add_subprocess("bob", "bob_pairing_scenario");
```

### When to Use Custom Transport/Harness

**Use custom harness with mock transport when:**
- Testing **sync logic** without network overhead
- Fast iteration on **data synchronization** algorithms
- Testing **deterministic scenarios** without network timing issues
- Verifying **database state** and **conflict resolution**
- Need precise control over sync event ordering

**Examples:** Real-time sync, backfill, content identity linking, conflict resolution

```rust
// Uses mock transport, single process, fast and deterministic
let harness = TwoDeviceHarnessBuilder::new("sync_test")
    .collect_events(true)
    .build()
    .await?;
```

### Comparison

| Aspect | Subprocess Framework | Custom Harness |
|--------|---------------------|----------------|
| **Speed** | Slower (real networking) | Fast (in-memory) |
| **Networking** | Real (discovery, NAT) | Mock transport |
| **Isolation** | True process isolation | Shared process |
| **Debugging** | Harder (multiple processes) | Easier (single process) |
| **Determinism** | Network timing varies | Fully deterministic |
| **Use Case** | Network features | Sync/data logic |

## Subprocess Testing Framework

The subprocess framework spawns separate `cargo test` processes for each device role:

```rust
let mut runner = CargoTestRunner::new()
    .with_timeout(Duration::from_secs(90))
    .add_subprocess("alice", "alice_scenario")
    .add_subprocess("bob", "bob_scenario");

runner.run_until_success(|outputs| {
    outputs.values().all(|output| output.contains("SUCCESS"))
}).await?;
```

### Writing Multi-Device Tests

Create separate test functions for each device role:

```rust
#[tokio::test]
async fn test_device_pairing() {
    let mut runner = CargoTestRunner::new()
        .add_subprocess("alice", "alice_pairing")
        .add_subprocess("bob", "bob_pairing");
        
    runner.run_until_success(|outputs| {
        outputs.values().all(|o| o.contains("PAIRING_SUCCESS"))
    }).await.unwrap();
}

#[tokio::test]
#[ignore]
async fn alice_pairing() {
    if env::var("TEST_ROLE").unwrap_or_default() != "alice" {
        return;
    }
    
    let data_dir = PathBuf::from(env::var("TEST_DATA_DIR").unwrap());
    let core = create_test_core(data_dir).await.unwrap();
    
    // Alice initiates pairing
    let (code, _) = core.start_pairing_as_initiator().await.unwrap();
    fs::write("/tmp/pairing_code.txt", &code).unwrap();
    
    // Wait for connection
    wait_for_connection(&core).await;
    println!("PAIRING_SUCCESS");
}
```

<Note>
Device scenario functions must be marked with `#[ignore]` to prevent direct execution. They only run when called by the subprocess framework.
</Note>

### Process Coordination

Processes coordinate through:
- **Environment variables**: `TEST_ROLE` and `TEST_DATA_DIR`
- **Temporary files**: Share data like pairing codes
- **Output patterns**: Success markers for the runner to detect

## Common Test Patterns

### Filesystem Watcher Testing

When testing filesystem watcher functionality, several critical setup steps are required:

#### Enable Watcher in Test Config

The default `TestConfigBuilder` **disables the filesystem watcher** (for performance in sync tests). Tests that verify watcher events must explicitly enable it:

```rust
let mut config = TestConfigBuilder::new(test_root.clone())
    .build()?;

// CRITICAL: Enable watcher for change detection tests
config.services.fs_watcher_enabled = true;
config.save()?;

let core = Core::new(config.data_dir.clone()).await?;
```

#### Use Home Directory Paths on macOS

macOS temp directories (`/var/folders/...`) don't reliably deliver filesystem events. Use home directory paths instead:

```rust
// ❌ Don't use TempDir for watcher tests
let temp_dir = TempDir::new()?;

// ✅ Use home directory
let home = std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string());
let test_root = PathBuf::from(home).join(".spacedrive_test_my_test");

// Clean up before
let _ = tokio::fs::remove_dir_all(&test_root).await;
tokio::fs::create_dir_all(&test_root).await?;

// ... run test ...

// Clean up after
tokio::fs::remove_dir_all(&test_root).await?;
```

#### Ephemeral Watching Requirements

Ephemeral paths must be **indexed before watching**:

```rust
// 1. Index the directory (ephemeral mode)
let config = IndexerJobConfig::ephemeral_browse(
    SdPath::local(dest_dir.clone()),
    IndexScope::Current
);
let job = IndexerJob::new(config);
library.jobs().dispatch(job).await?.wait().await?;

// 2. Mark indexing complete (indexer job does this automatically)
context.ephemeral_cache().mark_indexing_complete(&dest_dir);

// 3. Register for watching (indexer job does this automatically)
watcher.watch_ephemeral(dest_dir.clone()).await?;

// Now filesystem events will be detected
```

<Note>
The `IndexerJob` automatically calls `watch_ephemeral()` after successful indexing, so manual registration is only needed when bypassing the indexer.
</Note>

#### Persistent Location Watching

For persistent locations, the watcher auto-loads locations at startup. New locations created during tests must be manually registered:

```rust
// After creating and indexing a location
let location_meta = LocationMeta {
    id: location_uuid,
    library_id: library.id(),
    root_path: location_path.clone(),
    rule_toggles: RuleToggles::default(),
};

watcher.watch_location(location_meta).await?;
```

The `IndexingHarness` handles this automatically.

#### Event Collection Best Practices

Start collecting events **after** initialization to avoid library statistics noise:

```rust
// Complete all setup first
let harness = IndexingHarnessBuilder::new("test").build().await?;
let location = harness.add_and_index_location(...).await?;

// Wait for setup to settle
tokio::time::sleep(Duration::from_millis(500)).await;

// Start collecting BEFORE the operation you're testing
let mut collector = EventCollector::new(&harness.core.events);
let handle = tokio::spawn(async move {
    collector.collect_events(Duration::from_secs(5)).await;
    collector
});

// Perform operation
perform_copy_operation().await?;

// Collect and verify
let collector = handle.await.unwrap();
let stats = collector.analyze().await;
assert!(stats.resource_changed.get("file").copied().unwrap_or(0) >= 2);
```

The `EventCollector` automatically filters out:
- Library statistics updates (`LibraryStatisticsUpdated`)
- Library resource events (non-file/entry events)

#### Expected Event Types

Different handlers emit different event types:

- **Ephemeral handler**: Individual `ResourceChanged` events per file (CREATE + MODIFY)
- **Persistent handler**: Batched `ResourceChangedBatch` events

```rust
// Ephemeral assertion
let file_events = stats.resource_changed.get("file").copied().unwrap_or(0);
assert!(file_events >= 2, "Expected file ResourceChanged events");

// Persistent assertion
let batch_count = stats.resource_changed_batch.get("file").copied().unwrap_or(0);
assert!(batch_count >= 2, "Expected file ResourceChangedBatch events");
```

### Event Monitoring

#### Waiting for Specific Events

Wait for specific Core events with timeouts:

```rust
let mut events = core.events.subscribe();

let event = wait_for_event(
    &mut events,
    |e| matches!(e, Event::JobCompleted { .. }),
    Duration::from_secs(30)
).await?;
```

#### Collecting All Events for Analysis

For tests that need to verify event emission patterns (e.g., ResourceChanged events during operations), use the shared `EventCollector` helper:

```rust
use helpers::EventCollector;

// Create collector with full event capture for debugging
let mut collector = EventCollector::with_capture(&harness.core.events);

// Spawn collection task
let collection_handle = tokio::spawn(async move {
    collector.collect_events(Duration::from_secs(10)).await;
    collector
});

// Perform operations that emit events
perform_copy_operation().await?;
location.reindex().await?;

// Retrieve collector and analyze
let collector = collection_handle.await.unwrap();

// Print statistics summary
let stats = collector.analyze().await;
stats.print();

// Print full event details for debugging (when using with_capture)
collector.print_events().await;

// Write events to JSON file for later inspection
collector.write_to_file(&snapshot_dir.join("events.json")).await?;

// Filter specific events
let file_events = collector.get_resource_batch_events("file").await;
let indexing_events = collector.get_events_by_type("IndexingCompleted").await;
```

The `EventCollector` tracks:
- **ResourceChanged/ResourceChangedBatch** events by resource type
- **Indexing** start/completion events
- **Job** lifecycle events (started/completed)
- **Entry** events (created/modified/deleted/moved)

**Statistics Output:**
```
Event Statistics:
==================

ResourceChangedBatch events:
  file → 45 resources

Indexing events:
  Started: 1
  Completed: 1

Entry events:
  Created: 3
  Modified: 0

Job events:
  Started:
    indexer → 1
  Completed:
    indexer → 1
```

**Detailed Event Output (with `with_capture()`):**
```
=== Collected Events (8) ===

[1] IndexingStarted
  Location: 550e8400-e29b-41d4-a716-446655440000

[2] JobStarted
  Job: indexer (job_123)

[3] ResourceChangedBatch
  Type: file
  Resources: 45 items
  Paths: 1 affected

[4] IndexingCompleted
  Location: 550e8400-e29b-41d4-a716-446655440000
  Files: 42, Dirs: 3

[5] JobCompleted
  Job: indexer (job_123)
  Output: Success
```

**Use Cases:**
- Verifying watcher events during file operations
- Testing normalized cache updates
- Debugging event emission patterns
- Creating test fixtures with real event data
- Inspecting actual resource payloads in events

### Database Verification

Query the database directly to verify state:

```rust
use sd_core::entities;

let entries = entities::entry::Entity::find()
    .filter(entities::entry::Column::Name.contains("test"))
    .all(db.conn())
    .await?;

assert_eq!(entries.len(), expected_count);
```

### Job Testing

Test job execution and resumption:

```rust
// Start a job
let job_id = core.jobs.dispatch(IndexingJob::new(...)).await?;

// Monitor progress
wait_for_event(&mut events, |e| matches!(
    e, 
    Event::JobProgress { id, .. } if *id == job_id
), timeout).await?;

// Verify completion
let job = core.jobs.get_job(job_id).await?;
assert_eq!(job.status, JobStatus::Completed);
```

### Mock Transport for Sync Testing

Test synchronization without real networking:

```rust
let transport = Arc::new(Mutex::new(Vec::new()));

let mut core_a = create_test_core().await?;
let mut core_b = create_test_core().await?;

// Connect cores with mock transport
connect_with_mock_transport(&mut core_a, &mut core_b, transport).await?;

// Verify sync
perform_operation_on_a(&core_a).await?;
wait_for_sync(&core_b).await?;
```

## Test Helpers

### Common Utilities

The framework provides comprehensive test helpers in `core/tests/helpers/`:

**Event Collection:**
- `EventCollector` - Collect and analyze all events from the event bus
- `EventStats` - Statistics about collected events with formatted output

**Indexing Tests:**
- `IndexingHarnessBuilder` - Create isolated test environments with indexing support
- `TestLocation` - Builder for test locations with files
- `LocationHandle` - Handle to indexed locations with verification methods

**Sync Tests:**
- `TwoDeviceHarnessBuilder` - Pre-configured two-device sync test environments
- `MockTransport` - Mock network transport for deterministic sync testing
- `wait_for_sync()` - Sophisticated sync completion detection
- `TestConfigBuilder` - Custom test configurations

**Database & Jobs:**
- `wait_for_event()` - Wait for specific events with timeout
- `wait_for_indexing()` - Wait for indexing job completion
- `register_device()` - Register a device in a library

<Tip>
See `core/tests/helpers/README.md` for detailed documentation on all available helpers including usage examples and migration guides.
</Tip>

### Test Volumes

For volume-related tests, use the test volume utilities:

```rust
use helpers::test_volumes;

let volume = test_volumes::create_test_volume().await?;
// Test volume operations
test_volumes::cleanup_test_volume(volume).await?;
```

## Running Tests

### All Tests
```bash
cargo test --workspace
```

### Specific Test
```bash
cargo test test_device_pairing --nocapture
```

### Debug Subprocess Tests
```bash
# Run individual scenario
TEST_ROLE=alice TEST_DATA_DIR=/tmp/test cargo test alice_scenario -- --ignored --nocapture
```

### With Logging
```bash
RUST_LOG=debug cargo test test_name --nocapture
```

## Best Practices

### Test Structure

1. **Use descriptive names**: `test_cross_device_file_transfer` over `test_transfer`
2. **One concern per test**: Focus on a single feature or workflow
3. **Clean up resources**: Use RAII patterns or explicit cleanup

### Subprocess Tests

1. **Always use `#[ignore]`** on scenario functions
2. **Check TEST_ROLE early**: Return immediately if role doesn't match
3. **Use clear success patterns**: Print distinct markers for the runner
4. **Set appropriate timeouts**: Balance between test speed and reliability

### Debugging

<Tip>
When tests fail, check the logs in `test_data/{test_name}/library/logs/` for detailed information about what went wrong.
</Tip>

Common debugging approaches:
- Run with `--nocapture` to see all output
- Check job logs in `test_data/{test_name}/library/job_logs/`
- Run scenarios individually with manual environment variables
- Use `RUST_LOG=trace` for maximum verbosity

### Performance

1. **Run tests in parallel**: Use `cargo test` default parallelism
2. **Minimize sleeps**: Use event waiting instead of fixed delays
3. **Share setup code**: Extract common initialization into helpers

## Writing New Tests

### Single-Device Test Checklist

- [ ] Create test with `#[tokio::test]`
- [ ] Use `IntegrationTestSetup` for isolation
- [ ] Wait for events instead of sleeping
- [ ] Verify both positive and negative cases
- [ ] Clean up temporary files

### Multi-Device Test Checklist

- [ ] Create orchestrator function with `CargoTestRunner`
- [ ] Create scenario functions with `#[ignore]`
- [ ] Add TEST_ROLE guards to scenarios
- [ ] Define clear success patterns
- [ ] Handle process coordination properly
- [ ] Set reasonable timeouts

## Examples

For complete examples, refer to:

**Single Device Tests:**
- `tests/copy_action_test.rs` - Event collection during file operations (persistent + ephemeral)
- `tests/job_resumption_integration_test.rs` - Job interruption handling

**Subprocess Framework (Real Networking):**
- `tests/device_pairing_test.rs` - Device pairing with real network discovery

**Custom Harness (Mock Transport):**
- `tests/sync_realtime_test.rs` - Real-time sync testing with deterministic transport
- `tests/sync_integration_test.rs` - Complex sync scenarios with mock networking
- `tests/file_transfer_test.rs` - Cross-device file operations