spacedrive/docs/react/ui/normalized-cache.mdx

---
title: Normalized Query
sidebarTitle: Normalized Query
---

# Real-Time Normalized Cache with TanStack Query

The `useNormalizedQuery` hook provides instant, event-driven cache updates with server-side filtering. Built with 2025 best practices including runtime validation (Valibot), type-safe merging (ts-deepmerge), and proper subscription cleanup.

## Overview

`useNormalizedQuery` wraps TanStack Query to add real-time capabilities:

- **Instant updates** across all devices via WebSocket events
- **Server-side filtering** reduces network traffic by 90%+
- **Client-side safety** ensures correctness even with unrelated events
- **Proper cleanup** prevents connection leaks
- **Runtime validation** catches malformed events
- **Type-safe merging** preserves data integrity

## Architecture

<FlowDiagram steps={[
  {
    title: "Device A: Create file",
    description: "User creates or modifies a file on their device"
  },
  {
    title: "Backend: Emit event",
    description: "Backend detects change and emits events to all connected clients",
    metrics: { "During indexing": "10,000 events" }
  },
  {
    title: "Server Filter: Per subscription",
    description: "Events are filtered server-side based on subscription criteria",
    items: [
      "Desktop: 100 events (1%)",
      "Movies: 500 events (5%)",
      "Inspector: 1-5 events (0.05%)"
    ]
  },
  {
    title: "Subscription Manager",
    description: "Multiplexes identical filters to optimize connections",
    items: [
      "1 backend sub → N hooks",
      "Auto deduplication",
      "Reference counting"
    ]
  },
  {
    title: "Client: Validate & filter",
    description: "Final validation and cache updates trigger React re-renders",
    items: [
      "Valibot validation",
      "Client-side filtering",
      "Atomic cache updates"
    ]
  }
]} />

## Basic Usage

### Directory Listing

```tsx
import { useNormalizedQuery } from "@sd/ts-client";

function DirectoryView({ path }: { path: SdPath }) {
	const { data, isLoading } = useNormalizedQuery({
		query: "files.directory_listing",
		input: { path },
		resourceType: "file",
		pathScope: path,
		includeDescendants: false, // Only direct children
	});

	if (isLoading) return <Spinner />;

	return (
		<div>
			{data?.files?.map((file) => <FileCard key={file.id} file={file} />)}
		</div>
	);
}
```

**What happens:**

1. Initial query fetches directory listing
2. Hook subscribes to file events for this path (exact mode)
3. When files are created/updated, events arrive instantly
4. Cache updates atomically
5. UI re-renders with new data

### Media View (Recursive)

```tsx
function MediaGallery({ path }: { path: SdPath }) {
	const { data } = useNormalizedQuery({
		query: "files.media_listing",
		input: { path, include_descendants: true },
		resourceType: "file",
		pathScope: path,
		includeDescendants: true, // All media in subtree
	});

	return (
		<Grid>
			{data?.files?.map((file) => <MediaThumbnail key={file.id} file={file} />)}
		</Grid>
	);
}
```

### Global Resources

```tsx
function SpacesList() {
	const { data } = useNormalizedQuery({
		query: "spaces.list",
		input: null,
		resourceType: "space",
		// No pathScope - spaces are global resources
	});

	return (
		<ul>{data?.spaces?.map((space) => <li key={space.id}>{space.name}</li>)}</ul>
	);
}
```

### Single Resource Queries

```tsx
function FileInspector({ fileId }: { fileId: string }) {
	const { data: file } = useNormalizedQuery({
		query: "files.by_id",
		input: { file_id: fileId },
		resourceType: "file",
		resourceId: fileId, // Only events for this file
	});

	return (
		<div>
			<h1>{file?.name}</h1>
			{/* Updates instantly when thumbnails generate */}
			{file?.sidecars?.map((sidecar) => (
				<Thumbnail key={sidecar.id} src={sidecar.url} />
			))}
		</div>
	);
}
```

## API Reference

### Options

```tsx
interface UseNormalizedQueryOptions<I> {
	// Query method to call (e.g., "files.directory_listing")
	query: string;

	// Input for the query
	input: I;

	// Resource type for event filtering (e.g., "file", "space")
	resourceType: string;

	// Whether query is enabled (default: true)
	enabled?: boolean;

	// Optional path scope for server-side filtering
	pathScope?: SdPath;

	// Whether to include descendants (recursive) or only direct children (exact)
	// Default: false (exact matching)
	includeDescendants?: boolean;

	// Resource ID for single-resource queries
	resourceId?: string;
}
```

### Path Filtering Modes

#### Exact Mode (Default)

Only events for files **directly in** the specified directory:

```tsx
pathScope: { Physical: { device_slug: 'my-mac', path: '/Photos' } },
includeDescendants: false // or omit (default)
```

**Behavior:**

- File in `/Photos/image.jpg` → ✓ Included
- File in `/Photos/Vacation/beach.jpg` → ✗ Excluded
- Directory `/Photos/Vacation` → ✗ Excluded

#### Recursive Mode

All events for files **anywhere under** the specified directory:

```tsx
pathScope: { Physical: { device_slug: 'my-mac', path: '/Photos' } },
includeDescendants: true
```

**Behavior:**

- File in `/Photos/image.jpg` → ✓ Included
- File in `/Photos/Vacation/beach.jpg` → ✓ Included
- File in `/Photos/Vacation/Cruise/pic.jpg` → ✓ Included

## Server-Side Filtering

### How It Works

Each hook creates a filtered subscription on the backend:

```tsx
client.subscribeFiltered({
	resource_type: "file", // Only file events
	path_scope: "/Desktop", // Only this path
	include_descendants: false, // Exact mode
	library_id: "abc-123", // Current library
});
```

Backend applies filters **before** sending events:

1. ✓ `resource_type` matches?
2. ✓ `library_id` matches?
3. ✓ `path_scope` matches? (with `include_descendants` mode)
4. ✓ `resourceId` matches? (if specified)

**Result:** Only matching events are transmitted over the network.

### Filter Logic

**Exact Mode:**

```
Event has affected_paths: [
  "/Desktop/file.txt",           // File path
  "/Desktop"                     // Parent directory
]

Subscription path_scope: "/Desktop"
include_descendants: false

Check: Does affected_paths contain "/Desktop" exactly?
Result: YES → Forward event
```

**Recursive Mode:**

```
Event has affected_paths: [
  "/Desktop/Subfolder/file.txt",
  "/Desktop/Subfolder"
]

Subscription path_scope: "/Desktop"
include_descendants: true

Check: Does "/Desktop/Subfolder" start with "/Desktop"?
Result: YES → Forward event
```

## Client-Side Safety Filtering

Even with server-side filtering, the client applies a safety filter to batch events:

```tsx
// Server forwards batch if ANY file matches
// Client filters to ONLY files that match

Batch has 100 files:
- 10 in /Desktop/ (direct children)
- 90 in /Desktop/Subfolder/ (subdirectories)

Server: Has 1 direct child → forward entire batch
Client: Filter batch → keep only 10 direct children
Cache: Contains only 10 files ✓
```

This ensures correctness even if server-side filtering has edge cases.

## Event Types

### ResourceChanged (Single)

```tsx
{
  ResourceChanged: {
    resource_type: "space",
    resource: {
      id: "uuid",
      name: "Photos",
      path: "/Users/me/Photos",
      // ... full resource data
    },
    metadata: {
      no_merge_fields: ["sd_path"],
      affected_paths: [],
      alternate_ids: []
    }
  }
}
```

### ResourceChangedBatch (Multiple)

```tsx
{
  ResourceChangedBatch: {
    resource_type: "file",
    resources: [
      { id: "1", name: "photo1.jpg", ... },
      { id: "2", name: "photo2.jpg", ... }
    ],
    metadata: {
      no_merge_fields: ["sd_path"],
      affected_paths: [
        { Physical: { device_slug: "mac", path: "/Desktop/photo1.jpg" } },
        { Physical: { device_slug: "mac", path: "/Desktop" } },
        { Content: { content_id: "uuid" } }
      ],
      alternate_ids: []
    }
  }
}
```

### ResourceDeleted

```tsx
{
  ResourceDeleted: {
    resource_type: "space",
    resource_id: "uuid"
  }
}
```

### Refresh (Invalidate All)

```tsx
"Refresh";
```

Triggers `queryClient.invalidateQueries()` to refetch all data.

## Deep Merge Behavior

Uses `ts-deepmerge` for type-safe, configurable merging:

```tsx
// Existing cache
{
  id: "1",
  name: "Photos",
  metadata: { size: 1024, created_at: "2024-01-01" }
}

// Incoming event (partial update)
{
  id: "1",
  name: "My Photos",
  metadata: { size: 2048 }
}

// Result after merge
{
  id: "1",
  name: "My Photos",      // Updated
  metadata: {
    size: 2048,           // Updated
    created_at: "2024-01-01"  // Preserved ✓
  }
}
```

### No-Merge Fields

Some fields should be replaced entirely, not merged:

```tsx
metadata: {
	no_merge_fields: ["sd_path"];
}

// sd_path is replaced entirely, not deep merged
// This prevents incorrect path combinations
```

## Runtime Validation

All events are validated with Valibot before processing:

```tsx
const ResourceChangedSchema = v.object({
  ResourceChanged: v.object({
    resource_type: v.string(),
    resource: v.any(),
    metadata: v.nullish(v.object({ ... }))
  })
});

// Invalid events are logged and ignored
// Prevents crashes from malformed backend data
```

## Subscription Multiplexing

Multiple hooks with identical filters automatically share a single backend subscription:

```tsx
// Component A
function SpacesList() {
  useNormalizedQuery({
    query: 'spaces.list',
    resourceType: 'space',
  });
}

// Component B (mounted at same time)
function SpacesDropdown() {
  useNormalizedQuery({
    query: 'spaces.list',
    resourceType: 'space',
  });
}

// Result: Only 1 backend subscription created!
// Both hooks receive events from the same connection.
```

**How it works:**

1. First hook creates subscription with filter `{resource_type: "space", library_id: "abc"}`
2. Subscription manager generates key from filter: `{"resource_type":"space","library_id":"abc"}`
3. Second hook with same filter reuses existing subscription
4. Events broadcast to all listeners
5. When both unmount, subscription cleaned up automatically

**Benefits:**

- Eliminates duplicate subscriptions during render cycles
- Reduces backend load (fewer Unix socket connections)
- Faster subscription setup (reuses existing connection)
- Automatic reference counting prevents premature cleanup

## Subscription Cleanup

Subscriptions are properly cleaned up when components unmount:

```tsx
useEffect(() => {
	let unsubscribe: (() => void) | undefined;

	client.subscribeFiltered(filter, handleEvent).then((unsub) => {
		unsubscribe = unsub;
	});

	return () => {
		unsubscribe?.(); // Closes WebSocket subscription
	};
}, [dependencies]);
```

**Cleanup process:**

1. React calls cleanup function
2. Frontend stops listening to events
3. Tauri sends `Unsubscribe` request to daemon
4. Daemon closes subscription
5. Unix socket connection closed

**Result:** No connection leaks, no memory leaks.

## Performance

### Event Reduction

```
Indexing 10,000 files:

Without filtering:
- Each hook receives: 10,000 events
- Total transmitted: 50,000 events (5 hooks × 10,000)
- Result: UI lag, slow

With filtering:
- Desktop hook: 100 events (1%)
- Movies hook: 500 events (5%)
- Inspector: 1-5 events (0.05%)
- Total transmitted: ~600 events
- Result: Zero lag
```

### Connection Management

- **Multiplexing:** Multiple hooks with identical filters share one backend subscription
- **Reference counting:** Subscriptions cleaned up when last hook unmounts
- **Deduplication:** Eliminates duplicate subscriptions during render cycles
- **Monitoring:** Check `client.getSubscriptionStats()` for active subscriptions

## Testing

### Test Coverage

**Rust (Backend):**

- 9/9 event filtering tests passing
- Validates exact vs recursive modes
- Tests all path types (Physical, Content, Cloud, Sidecar)

**TypeScript (Frontend):**

- 5/5 integration tests passing
- Uses real backend event fixtures
- Validates filtering and cache updates
- Proves correctness with actual production code

### Run Tests

```bash
# Rust tests
cargo test --test event_filtering_test

# TypeScript tests
cd packages/ts-client && bun test

# Generate new fixtures from backend
cargo test --test normalized_cache_fixtures_test
```

## Best Practices

### Always Scope File Queries

```tsx
// Good
const { data } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	pathScope: path, // Server filters efficiently
});

// Bad - will skip subscription
const { data } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	// Missing pathScope! Subscription skipped to prevent overload
});
```

### Use Correct Mode for View Type

```tsx
// Directory view - exact mode
includeDescendants: false; // Only direct children

// Media gallery - recursive mode
includeDescendants: true; // All media in subtree

// Search results - recursive mode
includeDescendants: true; // All matching files
```

### Combine with TanStack Query Options

```tsx
const { data } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	pathScope: path,
	// TanStack Query options
	enabled: !!path,
	staleTime: 5 * 60 * 1000,
	refetchOnWindowFocus: true,
});
```

## Advanced Usage

### Content-Addressed Files

Files use Content-based `sd_path` but have Physical paths in `alternate_paths`:

```tsx
// File structure
{
  sd_path: { Content: { content_id: "uuid" } },
  alternate_paths: [
    { Physical: { device_slug: "mac", path: "/Desktop/file.txt" } }
  ]
}

// Client-side filtering uses alternate_paths for path matching
// This enables deduplication while maintaining path filtering
```

### Multiple Instances

Multiple files with same content have different IDs:

```tsx
// file1.txt (original)
{ id: "1", content_identity: { uuid: "abc" } }

// file2.txt (duplicate)
{ id: "2", content_identity: { uuid: "abc" } }

// Both update when content is processed
```

## Debugging

### Enable Logging

```tsx
// Check console for:
// "[useNormalizedQuery] Invalid event: ..." - Validation failures
// "[TauriTransport] Unsubscribing: ..." - Cleanup events
```

### Monitor Subscriptions

```bash
# Backend logs show subscription lifecycle
RUST_LOG=sd_core::infra::daemon::rpc=debug cargo run -p spacedrive-tauri

# Look for:
# "New subscription created: ..." - Subscription started
# "Subscription cancelled: ..." - Cleanup triggered
# "Unsubscribe sent successfully" - Connection closed
```

**Frontend subscription stats:**

```tsx
import { useSpacedriveClient } from '@sd/ts-client';

function DebugPanel() {
  const client = useSpacedriveClient();
  const stats = client.getSubscriptionStats();

  console.log(`Active subscriptions: ${stats.activeSubscriptions}`);
  stats.subscriptions.forEach(sub => {
    console.log(`  ${sub.key}: ${sub.refCount} hooks, ${sub.listenerCount} listeners`);
  });
}
```

### Inspect Cache

```tsx
import { useQueryClient } from "@tanstack/react-query";

const queryClient = useQueryClient();

// View all cached queries
console.log(queryClient.getQueryCache().getAll());

// View specific query
const queryKey = ["query:files.directory_listing", libraryId, { path }];
console.log(queryClient.getQueryData(queryKey));
```

## Migration

### From useLibraryQuery

```tsx
// Before (no real-time updates)
const { data } = useLibraryQuery({
	type: "spaces.list",
	input: {},
});

// After (instant updates)
const { data } = useNormalizedQuery({
	query: "spaces.list",
	input: null,
	resourceType: "space",
});
```

### Backward Compatibility

The old `useNormalizedCache` name is aliased:

```tsx
// Both work identically
import { useNormalizedQuery } from "@sd/ts-client";
import { useNormalizedCache } from "@sd/ts-client"; // Alias

// Prefer useNormalizedQuery for new code
```

## Technical Details

### Exported Functions

Core logic is exported for testing:

```tsx
import {
	filterBatchResources, // Filter resources by pathScope
	updateBatchResources, // Update cache with batch
	updateSingleResource, // Update single resource
	deleteResource, // Remove from cache
	safeMerge, // Deep merge utility
	handleResourceEvent, // Event dispatcher
} from "@sd/ts-client/hooks/useNormalizedQuery";
```

### Runtime Dependencies

- **ts-deepmerge** - Type-safe deep merging
- **valibot** - Runtime event validation
- **tiny-invariant** - Assertion helpers
- **type-fest** - TypeScript utilities
- **@tanstack/react-query** - Core caching

### Subscription Lifecycle

```
1. Component mounts
   ↓
2. useNormalizedQuery creates subscription
   ↓
3. Backend creates filtered event stream
   ↓
4. Events flow: Backend → Tauri → Frontend → Hook → Cache
   ↓
5. Component unmounts
   ↓
6. Cleanup function called
   ↓
7. Tauri cancels background task
   ↓
8. Backend receives Unsubscribe
   ↓
9. Unix socket closed
   ↓
10. Connection freed
```

## Common Patterns

### List with Real-Time Updates

```tsx
const { data: items } = useNormalizedQuery({
	query: "items.list",
	input: filters,
	resourceType: "item",
});

// Items list updates instantly when:
// - New items created
// - Existing items modified
// - Items deleted
```

### Directory with Instant File Appearance

```tsx
const { data: files } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	pathScope: path,
});

// New files appear instantly:
// - Screenshot taken → appears immediately
// - File copied → shows up without refresh
// - File renamed → updates in real-time
```

### Inspector with Sidecar Updates

```tsx
const { data: file } = useNormalizedQuery({
	query: "files.by_id",
	input: { file_id },
	resourceType: "file",
	resourceId: file_id,
});

// Sidecars update as they're generated:
// - Thumbnail generated → appears instantly
// - Thumbstrip created → shows immediately
// - OCR extracted → updates in real-time
```

## Summary

`useNormalizedQuery` provides production-grade real-time caching:

- Server-side filtering (90%+ event reduction)
- Client-side safety (validates and filters)
- Proper cleanup (no connection leaks)
- Runtime validation (catches bad events)
- Type-safe merging (preserves data)
- Comprehensive tests (9 Rust + 5 TypeScript)
- TanStack Query compatible (all features work)
- Cross-device sync (instant updates everywhere)

Use it for any query where data can change and you want instant updates without manual refetching.