diff --git a/.github/logo.png b/.github/logo.png new file mode 100644 index 000000000..085d001da Binary files /dev/null and b/.github/logo.png differ diff --git a/README.md b/README.md index 11c854e45..a17e98a44 100644 --- a/README.md +++ b/README.md @@ -1,408 +1,126 @@

- Spacedrive Logo -

Spacedrive

-

- A file manager built on a virtual distributed filesystem -
- spacedrive.com - · - v2 Documentation - · - Discord -

-

- - - - - - - - - - - - -

+ Spacedrive

-Spacedrive is an open source cross-platform file manager, powered by a virtual distributed filesystem (VDFS) written in Rust. +

Spacedrive

-Organize files across multiple devices, clouds, and platforms from a single interface. Tag once, access everywhere. Never lose track of where your files are. +

+ An open source cross-platform file manager.
+ Powered by a virtual distributed filesystem written in Rust. +

-> [!IMPORTANT] -> **v2.0.0-alpha.1 Released: December 26, 2025** -> -> This is Spacedrive v2—a complete ground-up rewrite. After development of the original alpha version stopped in January this year, I rebuilt Spacedrive from scratch with the hard lessons learned. -> -> **Current status:** Alpha release for macOS and Linux. Windows support coming in alpha.2. Mobile apps (iOS/Android) coming soon. -> -> **[Download Release](https://github.com/spacedriveapp/spacedrive/releases/tag/v2.0.0-alpha.1)** · Visit [v2.spacedrive.com](https://v2.spacedrive.com) for complete documentation and guides. -> -> If you're looking for the previous version, see the [v1 branch](https://github.com/spacedriveapp/spacedrive/tree/v1). +

+ + + + + + + + + +

-## The Problem +

+ spacedrive.com • + Discord • + Getting Started +

-Computing was designed for a single-device world. The file managers we use today—Finder, Explorer, Files—were built when your data lived in one place: the computer in front of you. +--- -The shift to multi-device computing forced us into cloud ecosystems. Want your files everywhere? Upload them to someone else's servers. The convenience came at a cost: **data ownership**. This wasn't accidental—centralization was the path of least resistance for solving multi-device sync. +## What is Spacedrive? -Now AI is accelerating this trend. Cloud services offer intelligent file analysis and semantic search, but only if you upload your data to their infrastructure. As we generate more data and AI becomes more capable, we're giving away more and more to access basic computing conveniences. +Spacedrive is a file manager that treats files as first-class objects with content identity, not paths. A photo on your laptop and the same photo on your NAS are recognized as one piece of content. Organize files across multiple devices, clouds, and platforms from a single interface. -**The current system isn't built for a world where:** +- **Content identity** — every file gets a BLAKE3 content hash. Same file on two devices produces the same hash. Spacedrive tracks redundancy and deduplication across all your machines. +- **Cross-device** — see all your files across all your devices in one place. Files on disconnected devices stay in the index and appear as offline. +- **P2P sync** — devices connect directly via Iroh/QUIC. No servers, no cloud, no single point of failure. Metadata syncs between devices. Files stay where they are. +- **Cloud volumes** — index S3, Google Drive, Dropbox, OneDrive, Azure, and GCS as first-class volumes alongside local storage. +- **Nine views** — grid, list, columns, media, size, recents, search, knowledge, and splat. QuickPreview for video, audio, code, documents, 3D, and images. +- **Local-first** — everything runs on your machine. No data leaves your device unless you choose to sync between your own devices. -- You own multiple devices with underutilized compute and storage -- Local AI models are becoming competitive with cloud alternatives -- Privacy and data sovereignty matter -- You shouldn't have to choose between convenience and control +### Data Archival -## The Vision +Beyond files, Spacedrive can index and archive data from external sources via script-based adapters. Gmail, Apple Notes, Chrome bookmarks, Obsidian, Slack, GitHub, calendar events, contacts. Each data source becomes a searchable repository. Search fans out across files and archived data together. -Spacedrive is infrastructure for the next era of computing. It's an architecture designed for multi-device environments from the ground up—not cloud services retrofitted with offline support, but local-first sync that scales to the cloud when you want it. +Adapters are simple: a folder with an `adapter.toml` manifest and a sync script in any language. If it can read stdin and print lines, it can be an adapter. -As local AI models improve, Spacedrive becomes the fabric that enables the same insights cloud services offer today, but running on hardware you already own, on data that never leaves your control. This is a long-term project correcting computing's trajectory toward centralization. +Shipped adapters: Gmail, Apple Notes, Chrome Bookmarks, Chrome History, Safari History, Obsidian, OpenCode, Slack, macOS Contacts, macOS Calendar, GitHub. -The file explorer interface is deliberate. Everyone understands it. It's seen the least innovation in decades. And it has the most potential when you bake distributed computing, content awareness, and local AI into something universally familiar. +### Spacebot -## How It Works - -Spacedrive treats files as **first-class objects with content identity**, not paths. A photo on your laptop and the same photo on your NAS are recognized as one piece of content. This enables: - -- **Content-aware deduplication** - Track redundancy across all devices -- **Semantic search** - Find files in under 100ms across millions of entries -- **Transactional operations** - Preview conflicts, space savings, and outcomes before execution -- **Peer-to-peer sync** - No servers, no consensus protocols, no single point of failure -- **Offline-first** - Full functionality without internet, syncs when devices reconnect - -Files stay where they are. Spacedrive just makes them universally addressable with rich metadata and cross-device intelligence. +Spacedrive integrates with [Spacebot](https://github.com/spacedriveapp/spacebot), an open source AI agent runtime. Spacebot runs as a separate process alongside Spacedrive, communicating over APIs. Spacedrive provides the data layer. Spacebot provides the intelligence layer. Neither depends on the other. Together, they form an operating surface where humans and agents work side by side. --- ## Architecture -Spacedrive is built on four core principles: +The core is a single Rust crate with CQRS/DDD architecture. Every operation (file copy, tag create, search query) is a registered action or query with type-safe input/output that auto-generates TypeScript types for the frontend. -### 1. Virtual Distributed Filesystem (VDFS) - -Files and folders become first-class objects with rich metadata, independent of their physical location. Every file gets a universal address (`SdPath`) that works across devices. Content-aware addressing means you can reference files by what they contain, not just where they live. - -### 2. Content Identity System - -Adaptive hashing (BLAKE3 with strategic sampling for large files) creates a unique fingerprint for every piece of content. This enables: - -- **Deduplication**: Recognize identical files across devices -- **Redundancy tracking**: Know where your backups are -- **Content-based operations**: "Copy this file from wherever it's available" - -### 3. Transactional Actions - -Every file operation can be previewed before execution. See exactly what will happen—space savings, conflicts, estimated time—then approve or cancel. Operations become durable jobs that survive network interruptions and device restarts. - -### 4. Leaderless Sync - -Peer-to-peer synchronization without central coordinators. Device-specific data (your filesystem index) uses state replication. Shared metadata (tags, ratings) uses a lightweight HLC-ordered log with deterministic conflict resolution. No leader election, no single point of failure. - ---- - -## Core Features - -| Feature | Description | -| ----------------------- | ---------------------------------------------------------------------------- | -| **Cross-Platform** | macOS, Windows, Linux, iOS, Android | -| **Multi-Device Index** | Unified view of files across all your devices | -| **Content Addressing** | Find optimal file copies automatically (local-first, then LAN, then cloud) | -| **Smart Deduplication** | Identify identical files regardless of name or location | -| **Cloud Integration** | Index S3, Google Drive, Dropbox as first-class volumes | -| **P2P Networking** | Direct device connections with automatic NAT traversal (Iroh + QUIC) | -| **Semantic Tags** | Graph-based tagging with hierarchies, aliases, and contextual disambiguation | -| **Action Preview** | Simulate any operation before execution | -| **Offline-First** | Full functionality without internet, syncs when devices reconnect | -| **Local Backup** | P2P backup between your own devices (iOS photo backup available now) | -| **Extension System** | WASM-based plugins for domain-specific functionality | - ---- - -## Tech Stack - -**Core** - -- **Rust** - Entire VDFS implementation (~183k lines) -- **Tokio** - Async runtime -- **SQLite + SeaORM** - Local-first database with type-safe ORM queries -- **Iroh** - P2P networking with QUIC transport, hole-punching, and local discovery -- **BLAKE3** - Fast cryptographic hashing for content identity -- **Wasmer** - Sandboxed WASM extension runtime -- **Axum** - HTTP/GraphQL server for web and API access -- **OpenDAL** - Unified cloud storage abstraction (S3, Google Drive, OneDrive, Dropbox, Azure Blob, GCS) -- **Specta** - Auto-generated TypeScript and Swift types from Rust - -**Cryptography & Security** - -- **Ed25519 / X25519** - Signatures and key exchange -- **ChaCha20-Poly1305 / AES-GCM** - Authenticated encryption -- **Argon2** - Password hashing -- **BIP39** - Mnemonic phrase support for key backup -- **redb** - Encrypted key-value store for credentials - -**Media Processing** - -- **FFmpeg** (via custom `sd-ffmpeg` crate) - Video thumbnails, audio extraction -- **libheif** - HEIF/HEIC image support -- **Pdfium** - PDF rendering -- **Whisper** - On-device speech recognition (Metal-accelerated on Apple platforms) -- **Blurhash** - Compact image placeholders - -**Interface** (shared across web and desktop) - -- **React 19** - UI framework -- **Vite** - Build tooling -- **TypeScript** - Type-safe frontend code -- **TanStack Query** - Server state management -- **Zustand** - Client state management -- **Radix UI** - Accessible headless components -- **Tailwind CSS** - Utility-first styling -- **Framer Motion** - Animations -- **React Hook Form + Zod** - Form management and validation -- **Three.js / React Three Fiber** - 3D visualization -- **dnd-kit** - Drag and drop -- **TanStack Virtual / TanStack Table** - Virtualized lists and tables - -**Desktop** - -- **Tauri 2** - Cross-platform desktop shell (macOS, Linux, Windows) - -**Mobile (React Native)** - -- **React Native** 0.81 + **Expo** - Cross-platform mobile framework -- **Expo Router** - File-based routing -- **NativeWind** - Tailwind CSS for React Native -- **React Navigation** - Native navigation stack -- **Reanimated** - Native-thread animations -- **sd-mobile-core** - Rust core bridge via FFI - -**Architecture Patterns** - -- Event-driven design with centralized EventBus -- CQRS: Actions (mutations) and Queries (reads) with preview-commit-verify -- Durable jobs with MessagePack serialization and checkpointing -- Domain-separated sync with clear data ownership boundaries -- Compile-time operation registration via `inventory` crate - ---- - -## Project Structure +| Component | Technology | +| ------------------- | ----------------------------------------------------------------- | +| Language | Rust | +| Async runtime | Tokio | +| Database | SQLite (SeaORM + sqlx) | +| P2P | Iroh (QUIC, hole-punching, local discovery) | +| Content hashing | BLAKE3 | +| Vector search | LanceDB + FastEmbed | +| Cloud storage | OpenDAL | +| Cryptography | Ed25519, X25519, ChaCha20-Poly1305, AES-GCM | +| Media | FFmpeg, libheif, Pdfium, Whisper | +| Desktop | Tauri 2 | +| Mobile | React Native + Expo | +| Frontend | React 19, Vite, TanStack Query, Tailwind CSS | +| Type generation | Specta | ``` spacedrive/ -├── core/ # Rust VDFS implementation -│ ├── src/ -│ │ ├── domain/ # Core models (Entry, Library, Device, Tag, Volume) -│ │ ├── ops/ # CQRS operations (actions & queries) -│ │ ├── infra/ # Infrastructure (DB, events, jobs, sync) -│ │ ├── service/ # High-level services (network, file sharing, sync) -│ │ ├── crypto/ # Key management and encryption -│ │ ├── device/ # Device identity and configuration -│ │ ├── filetype/ # File type detection and registry -│ │ ├── location/ # Location management and indexing -│ │ ├── library/ # Library lifecycle and operations -│ │ └── volume/ # Volume detection and fingerprinting -│ └── tests/ # Integration tests (pairing, sync, file transfer) +├── core/ # Rust engine (CQRS/DDD) ├── apps/ -│ ├── cli/ # CLI and daemon entry point -│ ├── server/ # Headless server for Docker/self-hosting -│ ├── tauri/ # Desktop app shell (macOS, Windows, Linux) -│ ├── web/ # Web app (Vite, connects to daemon via WebSocket) -│ ├── mobile/ # React Native mobile app (Expo) -│ ├── api/ # Cloud API server (Bun + Elysia) -│ ├── landing/ # Marketing site and docs (Next.js) -│ ├── ios/ # Native iOS prototype (Swift) -│ ├── macos/ # Native macOS prototype (Swift) -│ └── gpui-photo-grid/ # GPUI media viewer prototype +│ ├── tauri/ # Desktop app (macOS, Windows, Linux) +│ ├── mobile/ # React Native (iOS, Android) +│ ├── cli/ # CLI and daemon +│ ├── server/ # Headless server +│ └── web/ # Browser client ├── packages/ -│ ├── interface/ # Shared React UI (used by web and desktop) -│ ├── ts-client/ # Auto-generated TypeScript client and hooks -│ ├── swift-client/ # Auto-generated Swift client -│ ├── ui/ # Shared component library -│ └── assets/ # Icons and images -├── crates/ -│ ├── crypto/ # Cryptographic primitives -│ ├── ffmpeg/ # FFmpeg bindings for video/audio -│ ├── images/ # Image processing (HEIF, PDF, SVG) -│ ├── media-metadata/ # EXIF/media metadata extraction -│ ├── fs-watcher/ # Cross-platform file system watcher -│ ├── sdk/ # WASM extension SDK -│ ├── sdk-macros/ # Extension procedural macros -│ ├── task-system/ # Durable job execution engine -│ ├── sd-client/ # Rust client library -│ └── ... # actors, fda, log-analyzer, utils -├── extensions/ # WASM extensions (photos, test-extension) -└── docs/ # Architecture documentation +│ ├── interface/ # Shared React UI +│ ├── ts-client/ # Auto-generated TypeScript client +│ ├── ui/ # Component library +│ └── assets/ # Icons, images, SVGs +├── crates/ # Standalone Rust crates (ffmpeg, crypto, etc.) +├── adapters/ # Script-based data source adapters +└── schemas/ # TOML data type schemas ``` --- -## Extensions - -Spacedrive's WASM-based extension system enables specialized functionality while maintaining security and portability. - -> [!NOTE] -> The extension system is under active development. A stable SDK API will be available in a future release. - -### Professional Extensions - -| Extension | Purpose | Key Features | Status | -| ------------- | ------------------------------- | --------------------------------------------------------------------------- | ----------- | -| **Photos** | AI-powered photo management | Face recognition, place identification, moments, scene classification | In Progress | -| **Chronicle** | Research & knowledge management | Document analysis, knowledge graphs, AI summaries | In Progress | -| **Atlas** | Dynamic CRM & team knowledge | Runtime schemas, contact tracking, deal pipelines | In Progress | -| **Studio** | Digital asset management | Scene detection, transcription, proxy generation | Planned | -| **Ledger** | Financial intelligence | Receipt OCR, expense tracking, tax preparation | Planned | -| **Guardian** | Backup & redundancy monitoring | Content identity tracking, zero-redundancy alerts, smart backup suggestions | Planned | -| **Cipher** | Security & encryption | Password manager, file encryption, breach alerts | Planned | - -### Open Source Archive Extensions - -| Extension | Purpose | Provides Data For | Status | -| ------------------- | ----------------------- | ------------------------ | ------- | -| **Email Archive** | Gmail/Outlook backup | Atlas, Ledger, Chronicle | Planned | -| **Chrome History** | Browsing history backup | Chronicle | Planned | -| **Spotify Archive** | Listening history | Analytics | Planned | -| **GPS Tracker** | Location timeline | Photos, Analytics | Planned | -| **Tweet Archive** | Twitter backup | Chronicle, Analytics | Planned | -| **GitHub Tracker** | Repository tracking | Chronicle | Planned | - ---- - ## Getting Started -### Prerequisites - -- **Rust** 1.81+ ([rustup](https://rustup.rs/)) -- **Bun** 1.3+ ([bun.sh](https://bun.sh)) - For Tauri desktop app - -### Quick Start with Desktop App (Tauri) - -Spacedrive runs as a daemon (`sd-daemon`) that manages your libraries and P2P connections. The Tauri desktop app can launch its own daemon instance, or connect to a daemon started by the CLI. +Requires [Rust](https://rustup.rs/) 1.81+, [Bun](https://bun.sh) 1.3+, [just](https://github.com/casey/just), and Python 3.9+ (for adapters). ```bash -# Clone the repository git clone https://github.com/spacedriveapp/spacedrive cd spacedrive -# Install dependencies -bun install -cargo run -p xtask -- setup # generates .cargo/config.toml with aliases -cargo build # builds all core and apps (including the daemon and cli) - -# Copy dependencies into the debug Folder ( probably windows only ) -Copy-Item -Path "apps\.deps\lib\*.dll" -Destination "target\debug" -ErrorAction SilentlyContinue -Copy-Item -Path "apps\.deps\bin\*.dll" -Destination "target\debug" -ErrorAction SilentlyContinue - -# Run the desktop app (automatically starts daemon) -cd apps/tauri -bun run tauri:dev -``` - -### Quick Start with CLI - -The CLI can manage libraries and run a persistent daemon that other apps connect to: - -```bash -# Build and run the CLI -cargo run -p sd-cli -- --help - -# Start the daemon (runs in background) -cargo run -p sd-cli -- daemon start - -# Create a library -cargo run -p sd-cli -- library create "My Library" - -# Add a location to index -cargo run -p sd-cli -- location add ~/Documents - -# Search indexed files -cargo run -p sd-cli -- search . - -# Now launch Tauri app - it will connect to the running daemon -``` - -### Running Tests - -Spacedrive has a comprehensive test suite covering single-device operations and multi-device networking scenarios. - -```bash -# Run all tests -cargo test --workspace - -# Run specific test -cargo test test_device_pairing --nocapture - -# Run with detailed logging -RUST_LOG=debug cargo test test_name --nocapture - -# Run core tests only -cargo test -p sd-core -``` - -See the [Testing Guide](https://v2.spacedrive.com/core/testing) for detailed documentation on: - -- Integration test framework -- Multi-device subprocess testing -- Event monitoring patterns -- Test helpers and utilities - -All integration tests are in `core/tests/` including device pairing, sync, file transfer, and job execution tests. - -### Development Commands - -```bash -# Run all tests -cargo test - -# Run tests for specific package -cargo test -p sd-core - -# Build CLI in release mode -cargo build -p sd-cli --release - -# Format code -cargo fmt - -# Run lints -cargo clippy +just setup # bun install + native deps + cargo config +just dev-daemon # start the daemon +just dev-desktop # launch the desktop app (connects to daemon) +just dev-server # headless server (alternative to desktop) +just test # run all workspace tests +just cli -- help # run the CLI ``` --- -## Privacy & Security +## Contributing -Spacedrive is **local-first**. Your data stays on your devices. - -- **End-to-End Encryption**: All P2P traffic encrypted via QUIC/TLS -- **At-Rest Encryption**: Libraries can be encrypted on disk (SQLCipher) -- **No Telemetry**: Zero tracking or analytics in the open source version -- **Self-Hostable**: Run your own relay servers and cloud cores -- **Data Sovereignty**: You control where your data lives - -Optional cloud integration (Spacedrive Cloud) is available for backup and remote access, but it's never required. The cloud service runs unmodified Spacedrive core as a standard P2P device—no special privileges, no custom APIs. - ---- - -## Documentation - -- **[v2 Documentation](https://v2.spacedrive.com)** - Complete guides and API reference -- **[Self-Hosting Guide](https://v2.spacedrive.com/overview/self-hosting)** - Deploy Spacedrive server -- **[Whitepaper](whitepaper/spacedrive.pdf)** - Technical architecture (work in progress) -- **[Contributing Guide](CONTRIBUTING.md)** - How to contribute -- **[Architecture Docs](docs/core/architecture.md)** - Detailed system design -- **[Extension SDK](docs/sdk.md)** - Build your own extensions - ---- - -## Get Involved - -- **Star the repo** to support the project - **Join [Discord](https://discord.gg/gTaF2Z44f5)** to chat with developers and community -- **Read the [v2 Documentation](https://v2.spacedrive.com)** for guides and API reference -- **Read the [Whitepaper](whitepaper/spacedrive.pdf)** for the full technical vision -- **Build an Extension** - Check out the [SDK docs](docs/sdk.md) +- **[Contributing Guide](CONTRIBUTING.md)** +- **[Adapter Guide](docs/ADAPTERS.md)** — write a data source adapter + +--- + +## License + +FSL-1.1-ALv2 — [Functional Source License](https://fsl.software/), converting to Apache 2.0 after two years. diff --git a/apps/tauri/Spacedrive.icon/Assets/Ball.png b/apps/tauri/Spacedrive.icon/Assets/Ball.png deleted file mode 100644 index db641ae5c..000000000 Binary files a/apps/tauri/Spacedrive.icon/Assets/Ball.png and /dev/null differ diff --git a/apps/tauri/Spacedrive.icon/Assets/spacedrive.png b/apps/tauri/Spacedrive.icon/Assets/spacedrive.png new file mode 100644 index 000000000..085d001da Binary files /dev/null and b/apps/tauri/Spacedrive.icon/Assets/spacedrive.png differ diff --git a/apps/tauri/Spacedrive.icon/icon.json b/apps/tauri/Spacedrive.icon/icon.json index 86ee04bd4..cb4d1e6b3 100644 --- a/apps/tauri/Spacedrive.icon/icon.json +++ b/apps/tauri/Spacedrive.icon/icon.json @@ -1,80 +1,16 @@ { - "fill-specializations" : [ - { - "value" : "automatic" - }, - { - "appearance" : "dark", - "value" : "system-dark" - } - ], + "fill" : "automatic", "groups" : [ { "layers" : [ { - "blend-mode-specializations" : [ - { - "appearance" : "tinted", - "value" : "screen" - } - ], - "fill-specializations" : [ - { - "appearance" : "tinted", - "value" : { - "solid" : "display-p3:1.00000,0.72781,0.41766,1.00000" - } - } - ], - "glass" : true, - "hidden" : false, - "image-name" : "Ball.png", - "name" : "Ball", - "opacity-specializations" : [ - { - "value" : 0.4 - }, - { - "appearance" : "dark", - "value" : 0 - }, - { - "appearance" : "tinted", - "value" : 0.53 - } - ], + "image-name" : "spacedrive.png", + "name" : "spacedrive", "position" : { - "scale" : 2, + "scale" : 2.86, "translation-in-points" : [ - 1.7218333746113785, - 2.7640092574830533 - ] - } - }, - { - "blend-mode-specializations" : [ - { - "appearance" : "tinted", - "value" : "normal" - } - ], - "fill-specializations" : [ - { - "appearance" : "tinted", - "value" : { - "solid" : "display-p3:1.00000,0.72781,0.41766,1.00000" - } - } - ], - "glass" : true, - "hidden" : false, - "image-name" : "Ball.png", - "name" : "Ball", - "position" : { - "scale" : 2, - "translation-in-points" : [ - 1.7218333746113785, - 2.7640092574830533 + 1.4300000000000637, + 1.4300000000000637 ] } } diff --git a/apps/tauri/assets/exports/Icon-iOS-ClearDark-1024x1024@1x.png b/apps/tauri/assets/exports/Icon-iOS-ClearDark-1024x1024@1x.png new file mode 100644 index 000000000..31d354ab9 Binary files /dev/null and b/apps/tauri/assets/exports/Icon-iOS-ClearDark-1024x1024@1x.png differ diff --git a/apps/tauri/assets/exports/Icon-iOS-ClearLight-1024x1024@1x.png b/apps/tauri/assets/exports/Icon-iOS-ClearLight-1024x1024@1x.png new file mode 100644 index 000000000..863143c1b Binary files /dev/null and b/apps/tauri/assets/exports/Icon-iOS-ClearLight-1024x1024@1x.png differ diff --git a/apps/tauri/assets/exports/Icon-iOS-Dark-1024x1024@1x.png b/apps/tauri/assets/exports/Icon-iOS-Dark-1024x1024@1x.png new file mode 100644 index 000000000..0d4d37d7a Binary files /dev/null and b/apps/tauri/assets/exports/Icon-iOS-Dark-1024x1024@1x.png differ diff --git a/apps/tauri/assets/exports/Icon-iOS-Default-1024x1024@1x.png b/apps/tauri/assets/exports/Icon-iOS-Default-1024x1024@1x.png new file mode 100644 index 000000000..c08919a7a Binary files /dev/null and b/apps/tauri/assets/exports/Icon-iOS-Default-1024x1024@1x.png differ diff --git a/apps/tauri/assets/exports/Icon-iOS-TintedDark-1024x1024@1x.png b/apps/tauri/assets/exports/Icon-iOS-TintedDark-1024x1024@1x.png new file mode 100644 index 000000000..46620855d Binary files /dev/null and b/apps/tauri/assets/exports/Icon-iOS-TintedDark-1024x1024@1x.png differ diff --git a/apps/tauri/assets/exports/Icon-iOS-TintedLight-1024x1024@1x.png b/apps/tauri/assets/exports/Icon-iOS-TintedLight-1024x1024@1x.png new file mode 100644 index 000000000..92c22c57d Binary files /dev/null and b/apps/tauri/assets/exports/Icon-iOS-TintedLight-1024x1024@1x.png differ diff --git a/apps/tauri/scripts/dev-with-daemon.ts b/apps/tauri/scripts/dev-with-daemon.ts index 5b6910637..f9ed7569d 100755 --- a/apps/tauri/scripts/dev-with-daemon.ts +++ b/apps/tauri/scripts/dev-with-daemon.ts @@ -8,50 +8,46 @@ * 4. Starts Vite dev server * 5. Cleans up daemon on exit */ - -import { spawn, execSync } from "child_process"; -import { existsSync, unlinkSync } from "fs"; -import { join, resolve, dirname } from "path"; -import { homedir, platform } from "os"; -import { fileURLToPath } from "url"; +import {execSync, spawn} from 'child_process'; +import {existsSync, unlinkSync} from 'fs'; +import {homedir, platform} from 'os'; +import {dirname, join, resolve} from 'path'; +import {fileURLToPath} from 'url'; // Get script directory const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename); // Detect Platform -const IS_WIN = platform() === "win32"; +const IS_WIN = platform() === 'win32'; // Paths relative to this script (apps/tauri/scripts/) // Script is at: PROJECT_ROOT/apps/tauri/scripts/ // So PROJECT_ROOT is: ../../../ -const PROJECT_ROOT = resolve(__dirname, "../../../"); +const PROJECT_ROOT = resolve(__dirname, '../../../'); // Resolve target directory from Cargo config (supports custom target-dir) function getCargoTargetDir(): string { - try { - const output = execSync("cargo metadata --format-version 1 --no-deps", { - cwd: PROJECT_ROOT, - encoding: "utf8", - stdio: ["pipe", "pipe", "pipe"], - }); - const metadata = JSON.parse(output); - return metadata.target_directory; - } catch { - return join(PROJECT_ROOT, "target"); - } + try { + const output = execSync('cargo metadata --format-version 1 --no-deps', { + cwd: PROJECT_ROOT, + encoding: 'utf8', + stdio: ['pipe', 'pipe', 'pipe'] + }); + const metadata = JSON.parse(output); + return metadata.target_directory; + } catch { + return join(PROJECT_ROOT, 'target'); + } } -const BIN_NAME = IS_WIN ? "sd-daemon.exe" : "sd-daemon"; -const DAEMON_BIN = join(getCargoTargetDir(), "debug", BIN_NAME); +const BIN_NAME = IS_WIN ? 'sd-daemon.exe' : 'sd-daemon'; +const DAEMON_BIN = join(getCargoTargetDir(), 'debug', BIN_NAME); const DAEMON_PORT = 6969; const DAEMON_ADDR = `127.0.0.1:${DAEMON_PORT}`; -// Fix Data Directory for Windows (Optional but recommended) -const DATA_DIR = IS_WIN - ? join(homedir(), "AppData/Roaming/spacedrive") - : join(homedir(), "Library/Application Support/spacedrive"); +const DATA_DIR = join(homedir(), '.spacedrive'); let daemonProcess: any = null; let viteProcess: any = null; @@ -59,158 +55,162 @@ let startedDaemon = false; // Cleanup function function cleanup() { - console.log("\nCleaning up..."); + console.log('\nCleaning up...'); - if (viteProcess) { - console.log("Stopping Vite..."); - viteProcess.kill(); - } + if (viteProcess) { + console.log('Stopping Vite...'); + viteProcess.kill(); + } - if (daemonProcess && startedDaemon) { - console.log("Stopping daemon (started by us)..."); - daemonProcess.kill(); - } else if (!startedDaemon) { - console.log("Leaving existing daemon running..."); - } + if (daemonProcess && startedDaemon) { + console.log('Stopping daemon (started by us)...'); + daemonProcess.kill(); + } else if (!startedDaemon) { + console.log('Leaving existing daemon running...'); + } - process.exit(0); + process.exit(0); } // Handle signals -process.on("SIGINT", cleanup); -process.on("SIGTERM", cleanup); +process.on('SIGINT', cleanup); +process.on('SIGTERM', cleanup); async function main() { - // Check if daemon is already running by trying to connect to TCP port - let daemonAlreadyRunning = false; - console.log(`Checking if daemon is running on ${DAEMON_ADDR}...`); - try { - const { connect } = await import("net"); - await new Promise((resolve, reject) => { - const client = connect(DAEMON_PORT, "127.0.0.1"); - client.on("connect", () => { - daemonAlreadyRunning = true; - client.end(); - resolve(); - }); - client.on("error", () => { - reject(); - }); - setTimeout(() => reject(), 1000); - }); - } catch (e) { - // Connection failed, daemon not running - daemonAlreadyRunning = false; - } + // Check if daemon is already running by trying to connect to TCP port + let daemonAlreadyRunning = false; + console.log(`Checking if daemon is running on ${DAEMON_ADDR}...`); + try { + const {connect} = await import('net'); + await new Promise((resolve, reject) => { + const client = connect(DAEMON_PORT, '127.0.0.1'); + client.on('connect', () => { + daemonAlreadyRunning = true; + client.end(); + resolve(); + }); + client.on('error', () => { + reject(); + }); + setTimeout(() => reject(), 1000); + }); + } catch (e) { + // Connection failed, daemon not running + daemonAlreadyRunning = false; + } - if (daemonAlreadyRunning) { - console.log("Daemon already running, skipping build and using existing instance"); - startedDaemon = false; - } else { - console.log("Building daemon (dev profile)..."); - console.log("Project root:", PROJECT_ROOT); - console.log("Daemon binary:", DAEMON_BIN); + if (daemonAlreadyRunning) { + console.log( + 'Daemon already running, skipping build and using existing instance' + ); + startedDaemon = false; + } else { + console.log('Building daemon (dev profile)...'); + console.log('Project root:', PROJECT_ROOT); + console.log('Daemon binary:', DAEMON_BIN); - // Build daemon - // On Windows, the binary target name is still just "sd-daemon" (Cargo handles the .exe) - const build = spawn("cargo", ["build", "--bin", "sd-daemon"], { - cwd: PROJECT_ROOT, - stdio: "inherit", - shell: IS_WIN, // shell: true is often needed on Windows for spawn to work correctly - }); + // Build daemon + // On Windows, the binary target name is still just "sd-daemon" (Cargo handles the .exe) + const build = spawn('cargo', ['build', '--bin', 'sd-daemon'], { + cwd: PROJECT_ROOT, + stdio: 'inherit', + shell: IS_WIN // shell: true is often needed on Windows for spawn to work correctly + }); - await new Promise((resolve, reject) => { - build.on("exit", (code) => { - if (code === 0) { - resolve(); - } else { - reject(new Error(`Daemon build failed with code ${code}`)); - } - }); - }); + await new Promise((resolve, reject) => { + build.on('exit', (code) => { + if (code === 0) { + resolve(); + } else { + reject(new Error(`Daemon build failed with code ${code}`)); + } + }); + }); - console.log("Daemon built successfully"); - // Start daemon - console.log("Starting daemon..."); - startedDaemon = true; + console.log('Daemon built successfully'); + // Start daemon + console.log('Starting daemon...'); + startedDaemon = true; - // Verify binary exists - if (!existsSync(DAEMON_BIN)) { - throw new Error(`Daemon binary not found at: ${DAEMON_BIN}`); - } + // Verify binary exists + if (!existsSync(DAEMON_BIN)) { + throw new Error(`Daemon binary not found at: ${DAEMON_BIN}`); + } - const depsLibPath = join(PROJECT_ROOT, "apps/.deps/lib"); - const depsBinPath = join(PROJECT_ROOT, "apps/.deps/bin"); + const depsLibPath = join(PROJECT_ROOT, 'apps/.deps/lib'); + const depsBinPath = join(PROJECT_ROOT, 'apps/.deps/bin'); - daemonProcess = spawn(DAEMON_BIN, ["--data-dir", DATA_DIR], { - cwd: PROJECT_ROOT, - stdio: ["ignore", "pipe", "pipe"], - env: { - ...process.env, - // macOS library path - DYLD_LIBRARY_PATH: depsLibPath, - // Windows: Add DLLs directory to PATH - PATH: IS_WIN - ? `${depsBinPath};${process.env.PATH || ""}` - : process.env.PATH, - }, - }); + daemonProcess = spawn(DAEMON_BIN, ['--data-dir', DATA_DIR], { + cwd: PROJECT_ROOT, + stdio: ['ignore', 'pipe', 'pipe'], + env: { + ...process.env, + // macOS library path + DYLD_LIBRARY_PATH: depsLibPath, + // Windows: Add DLLs directory to PATH + PATH: IS_WIN + ? `${depsBinPath};${process.env.PATH || ''}` + : process.env.PATH + } + }); - // Log daemon output - daemonProcess.stdout.on("data", (data: Buffer) => { - const lines = data.toString().trim().split("\n"); - for (const line of lines) { - console.log(`[daemon] ${line}`); - } - }); + // Log daemon output + daemonProcess.stdout.on('data', (data: Buffer) => { + const lines = data.toString().trim().split('\n'); + for (const line of lines) { + console.log(`[daemon] ${line}`); + } + }); - daemonProcess.stderr.on("data", (data: Buffer) => { - const lines = data.toString().trim().split("\n"); - for (const line of lines) { - console.log(`[daemon] ${line}`); - } - }); + daemonProcess.stderr.on('data', (data: Buffer) => { + const lines = data.toString().trim().split('\n'); + for (const line of lines) { + console.log(`[daemon] ${line}`); + } + }); - // Wait for daemon to be ready - console.log("Waiting for daemon to be ready..."); - for (let i = 0; i < 30; i++) { - try { - const { connect } = await import("net"); - await new Promise((resolve, reject) => { - const client = connect(DAEMON_PORT, "127.0.0.1"); - client.on("connect", () => { - client.end(); - resolve(); - }); - client.on("error", reject); - setTimeout(() => reject(), 500); - }); - console.log(`Daemon ready at ${DAEMON_ADDR}`); - break; - } catch (e) { - if (i === 29) { - throw new Error("Daemon failed to start (connection not available)"); - } - await new Promise((resolve) => setTimeout(resolve, 1000)); - } - } - } + // Wait for daemon to be ready + console.log('Waiting for daemon to be ready...'); + for (let i = 0; i < 30; i++) { + try { + const {connect} = await import('net'); + await new Promise((resolve, reject) => { + const client = connect(DAEMON_PORT, '127.0.0.1'); + client.on('connect', () => { + client.end(); + resolve(); + }); + client.on('error', reject); + setTimeout(() => reject(), 500); + }); + console.log(`Daemon ready at ${DAEMON_ADDR}`); + break; + } catch (e) { + if (i === 29) { + throw new Error( + 'Daemon failed to start (connection not available)' + ); + } + await new Promise((resolve) => setTimeout(resolve, 1000)); + } + } + } - // Start Vite - console.log("Starting Vite dev server..."); - - // Use 'bun' explicitly, with shell true for Windows compatibility - viteProcess = spawn("bun", ["run", "dev"], { - stdio: "inherit", - shell: IS_WIN, - }); + // Start Vite + console.log('Starting Vite dev server...'); - // Keep running - await new Promise(() => {}); + // Use 'bun' explicitly, with shell true for Windows compatibility + viteProcess = spawn('bun', ['run', 'dev'], { + stdio: 'inherit', + shell: IS_WIN + }); + + // Keep running + await new Promise(() => {}); } main().catch((error) => { - console.error("Error:", error); - cleanup(); - process.exit(1); -}); \ No newline at end of file + console.error('Error:', error); + cleanup(); + process.exit(1); +}); diff --git a/apps/tauri/sd-tauri-core/src/lib.rs b/apps/tauri/sd-tauri-core/src/lib.rs index b56428d7c..294b45d72 100644 --- a/apps/tauri/sd-tauri-core/src/lib.rs +++ b/apps/tauri/sd-tauri-core/src/lib.rs @@ -41,22 +41,11 @@ pub mod commands { // Following the pattern from sd-ios-core but for Tauri's IPC } -/// Platform-specific data directory resolution +/// Default data directory: `~/.spacedrive` pub fn default_data_dir() -> anyhow::Result { - #[cfg(target_os = "macos")] - let dir = dirs::data_dir() - .ok_or_else(|| anyhow::anyhow!("Could not determine data directory"))? - .join("spacedrive"); - - #[cfg(target_os = "windows")] - let dir = dirs::data_dir() - .ok_or_else(|| anyhow::anyhow!("Could not determine data directory"))? - .join("Spacedrive"); - - #[cfg(target_os = "linux")] - let dir = dirs::data_local_dir() - .ok_or_else(|| anyhow::anyhow!("Could not determine data directory"))? - .join("spacedrive"); + let dir = dirs::home_dir() + .ok_or_else(|| anyhow::anyhow!("Could not determine home directory"))? + .join(".spacedrive"); // Create directory if it doesn't exist std::fs::create_dir_all(&dir)?; diff --git a/apps/tauri/src-tauri/build.rs b/apps/tauri/src-tauri/build.rs index ffc4afae9..888ec7e00 100644 --- a/apps/tauri/src-tauri/build.rs +++ b/apps/tauri/src-tauri/build.rs @@ -74,19 +74,22 @@ fn main() { "" }; - let daemon_source = format!("{}/target/{}/sd-daemon{}", workspace_dir, profile, exe_ext); - let daemon_target = format!( - "{}/target/{}/sd-daemon-{}{}", - workspace_dir, profile, target_triple, exe_ext - ); + for source_profile in [profile.as_str(), "release"] { + let daemon_source = format!( + "{}/target/{}/sd-daemon{}", + workspace_dir, source_profile, exe_ext + ); + let daemon_target = format!( + "{}/target/{}/sd-daemon-{}{}", + workspace_dir, source_profile, target_triple, exe_ext + ); - if std::path::Path::new(&daemon_source).exists() { - // Remove existing file if it exists - let _ = std::fs::remove_file(&daemon_target); + if std::path::Path::new(&daemon_source).exists() { + let _ = std::fs::remove_file(&daemon_target); - // Copy the daemon binary with target architecture suffix - if let Err(e) = std::fs::copy(&daemon_source, &daemon_target) { - eprintln!("Warning: Failed to copy daemon: {}", e); + if let Err(e) = std::fs::copy(&daemon_source, &daemon_target) { + eprintln!("Warning: Failed to copy daemon: {}", e); + } } } diff --git a/apps/tauri/src-tauri/icons/128x128.png b/apps/tauri/src-tauri/icons/128x128.png index c7266c728..1dda499fc 100644 Binary files a/apps/tauri/src-tauri/icons/128x128.png and b/apps/tauri/src-tauri/icons/128x128.png differ diff --git a/apps/tauri/src-tauri/icons/128x128@2x.png b/apps/tauri/src-tauri/icons/128x128@2x.png index 3822a5c61..f753c7bf5 100644 Binary files a/apps/tauri/src-tauri/icons/128x128@2x.png and b/apps/tauri/src-tauri/icons/128x128@2x.png differ diff --git a/apps/tauri/src-tauri/icons/32x32.png b/apps/tauri/src-tauri/icons/32x32.png index 7a31a5c45..97da93ce6 100644 Binary files a/apps/tauri/src-tauri/icons/32x32.png and b/apps/tauri/src-tauri/icons/32x32.png differ diff --git a/apps/tauri/src-tauri/icons/icon.icns b/apps/tauri/src-tauri/icons/icon.icns index c0ac005ce..e066ccc55 100644 Binary files a/apps/tauri/src-tauri/icons/icon.icns and b/apps/tauri/src-tauri/icons/icon.icns differ diff --git a/apps/tauri/src-tauri/icons/icon.ico b/apps/tauri/src-tauri/icons/icon.ico index b7878b6d3..841913a53 100644 Binary files a/apps/tauri/src-tauri/icons/icon.ico and b/apps/tauri/src-tauri/icons/icon.ico differ diff --git a/core/src/config/mod.rs b/core/src/config/mod.rs index 1d9224af0..202ba7aa2 100644 --- a/core/src/config/mod.rs +++ b/core/src/config/mod.rs @@ -12,22 +12,12 @@ pub mod migration; pub use app_config::{AppConfig, JobLoggingConfig, LogStreamConfig, LoggingConfig, ServiceConfig}; pub use migration::Migrate; -/// Platform-specific data directory resolution +/// Default data directory: `~/.spacedrive` on desktop, platform data dir on mobile. pub fn default_data_dir() -> Result { - #[cfg(target_os = "macos")] - let dir = dirs::data_dir() - .ok_or_else(|| anyhow!("Could not determine data directory"))? - .join("spacedrive"); - - #[cfg(target_os = "windows")] - let dir = dirs::data_dir() - .ok_or_else(|| anyhow!("Could not determine data directory"))? - .join("Spacedrive"); - - #[cfg(target_os = "linux")] - let dir = dirs::data_local_dir() - .ok_or_else(|| anyhow!("Could not determine data directory"))? - .join("spacedrive"); + #[cfg(not(any(target_os = "ios", target_os = "android")))] + let dir = dirs::home_dir() + .ok_or_else(|| anyhow!("Could not determine home directory"))? + .join(".spacedrive"); #[cfg(target_os = "ios")] let dir = dirs::data_dir() diff --git a/core/src/volume/platform/macos.rs b/core/src/volume/platform/macos.rs index 65ddfb573..70256e1c5 100644 --- a/core/src/volume/platform/macos.rs +++ b/core/src/volume/platform/macos.rs @@ -163,6 +163,7 @@ pub async fn detect_non_apfs_volumes( color: None, icon: None, error_message: None, + supports_block_cloning: false, }; volumes.push(volume); } diff --git a/docs/core/design/data-repositories.md b/docs/core/design/data-repositories.md new file mode 100644 index 000000000..bcb6dc332 --- /dev/null +++ b/docs/core/design/data-repositories.md @@ -0,0 +1,477 @@ +# Data Repositories Integration Design + +## Purpose + +Fold the `spacedrive-data` prototype into the official `spacedrive` codebase as a new library-scoped feature without forcing convergence between the VDFS index and the archival repository engine. + +This keeps Spacedrive's file-native architecture intact while adding a second data plane for extracted, adapter-driven repositories such as Gmail, Obsidian, Chrome History, Slack, and GitHub. + +## Decision + +Spacedrive will support two storage systems inside a library: + +1. The VDFS library database, which remains the source of truth for files, entries, content identities, tags, spaces, sidecars, sync metadata, and file operations. +2. A repository engine, which manages archived external data sources as isolated repositories with their own SQLite database, vector index, schema, cursor state, and processing pipeline. + +These systems are linked at the library boundary, not merged into one database. + +## Why This Shape + +- The VDFS already solves file indexing, sync, sidecars, jobs, and device-aware lifecycle well. +- The prototype already solves schema-driven repositories, adapter ingestion, hybrid search, and isolated archival storage well. +- Forcing repository records into the main library database would create a large migration with little product value. +- Keeping repositories isolated preserves portability, adapter flexibility, schema evolution, and per-source lifecycle control. +- Registering repositories at the library level gives us one user-facing primitive for ownership, sync, permissions, and UI. + +## Goals + +- Add archival repositories to official Spacedrive without rewriting the VDFS. +- Reuse the existing daemon, RPC, type generation, ops, job system, and UI infrastructure. +- Keep each repository self-contained on disk. +- Make repositories library-scoped so they can participate in library sync and lifecycle. +- Translate the prototype's pipeline into Spacedrive's ops and jobs model. + +## Non-Goals + +- Do not merge repository records into the VDFS entry index. +- Do not split a library into multiple primary databases yet. +- Do not ship a separate OpenAPI server for this feature. +- Do not force the repository engine to use SeaORM for dynamic schema tables. +- Do not redesign the whole search stack to unify files and repositories in the first slice. + +## User Model + +A library owns: + +- its existing VDFS database and sidecars +- zero or more archival repositories + +A repository is a managed library resource. It is visible in the library UI, syncable across devices, and controlled by library operations. Its payload stays in a separate repository folder. + +Examples: + +- Library: `Personal` + - VDFS index for files and locations + - Repository: `Work Gmail` + - Repository: `Obsidian Vault` + - Repository: `Chrome History` + +## Storage Layout + +Each library gets a repository root inside `.sdlibrary`. + +```text +.sdlibrary/ + library.db + sidecars/ + repositories/ + registry.db + / + data.db + embeddings.lance/ + schema.toml + state/ + cache/ +``` + +### Library Database Responsibilities + +The library database stores repository metadata only: + +- repository id +- library id +- display name +- adapter id +- repository path +- trust tier +- visibility +- status +- last sync timestamps +- sync policy +- pipeline policy +- device sync metadata + +### Repository Folder Responsibilities + +Each repository folder stores source-specific payload: + +- generated SQLite tables from schema +- FTS tables and triggers +- vector index +- adapter cursor state +- adapter-specific caches +- local processing artifacts that are not sidecars + +## Integration Boundary + +The integration point is `Library`, not `CoreContext` globally and not the VDFS entry graph. + +The library becomes the owner of a `RepositoryManager` and a `RepositoryRegistry` alongside its existing services. + +```text +Core + -> LibraryManager + -> Library + -> VDFS database and services + -> Repository subsystem + -> repository registry + -> repository manager + -> adapter runtime + -> repository search + -> repository pipeline jobs +``` + +This means repository operations should mostly be library actions and library queries. + +## Proposed Code Shape + +Add a new subsystem under `core/src/repository/` or `core/src/data/`. + +Recommended shape: + +```text +core/src/data/ + mod.rs + manager.rs # library-scoped repository manager + registry.rs # metadata persisted in library db or library-owned registry db + repository.rs # open/create/delete repository folders + engine.rs # orchestration facade used by ops/jobs + adapter/ + mod.rs + script.rs + schema/ + mod.rs + parser.rs + codegen.rs + migration.rs + db/ + mod.rs + repository_db.rs + search/ + mod.rs + router.rs + fts.rs + vector.rs + secrets/ + mod.rs + safety/ + mod.rs + classify/ + mod.rs +``` + +This subsystem can keep `sqlx` and raw SQL internally where that makes dynamic schemas practical. The outer integration surface should still follow Spacedrive conventions: ops, jobs, events, and typed outputs. + +## Why Keep `sqlx` Internals + +The repository engine generates tables dynamically from TOML schemas. That fits raw SQL much better than SeaORM entities. + +The integration rule should be: + +- use existing Spacedrive infrastructure for lifecycle, dispatch, jobs, events, sync, and UI +- keep raw SQL and schema codegen inside the repository engine where it reduces friction + +This avoids rewriting working prototype internals just to satisfy the ORM used by the VDFS. + +## Library Registration Model + +Repositories should be first-class library resources. + +We add a new library-scoped domain concept, likely `library_repository`. + +Suggested fields: + +```text +id +library_id +name +adapter_id +repository_root +trust_tier +visibility +status +last_synced_at +last_screened_at +last_embedded_at +sync_cursor +search_enabled +agent_enabled +created_at +updated_at +``` + +The exact cursor should live inside repository storage if it is adapter-owned. The library record only needs summary metadata for UI and sync orchestration. + +## Operations Mapping + +The prototype's API should be translated into V2 ops. + +### Library Actions + +- `repositories.create` +- `repositories.update` +- `repositories.delete` +- `repositories.sync` +- `repositories.sync_all` +- `repositories.set_visibility` +- `repositories.set_policy` +- `repositories.release_quarantined` +- `repositories.delete_record` +- `repositories.adapters.install` if adapter installation remains user-facing + +### Library Queries + +- `repositories.list` +- `repositories.get` +- `repositories.search` +- `repositories.records.list` +- `repositories.records.get` +- `repositories.adapters.list` +- `repositories.schemas.list` +- `repositories.quarantine.list` +- `repositories.status` + +These should register through the existing macros in `core/src/ops/registry.rs` and flow through the current daemon transport. + +## Job System Mapping + +The prototype's pipeline should become library jobs. + +### Core Jobs + +- `RepositorySyncJob` +- `RepositoryScreeningJob` +- `RepositoryEmbeddingJob` +- `RepositoryClassificationJob` +- `RepositoryReindexJob` +- `RepositoryDeleteJob` for heavy cleanup + +### Sync Flow + +First slice: + +```text +repositories.sync action + -> enqueue RepositorySyncJob + -> adapter subprocess emits JSONL + -> repository DB upsert/delete/link + -> enqueue or run screening stage + -> enqueue or run embedding stage + -> update library repository status + -> emit events +``` + +The job system gives us resumability, progress reporting, cancellation, and a natural home for future classification work. + +## Search Model + +Initial scope: + +- repository search is separate from file search +- the UI can offer a dedicated repository search surface inside a library +- unified cross-surface search is deferred + +This avoids destabilizing `core/src/ops/search/` in the first integration slice. + +### Future + +Later we can add a federated search query that fans out to: + +- VDFS file search +- repository search + +and merges results at the query layer without forcing a shared storage model. + +## Sync Between Devices + +Repositories are library-scoped resources, so library sync should distribute repository metadata and availability. + +Recommended phases: + +### Phase 1 + +- sync repository metadata through the library +- do not sync repository payload automatically +- a second device sees that the repository exists and can pull or restore it later + +### Phase 2 + +- sync repository payloads as managed library resources +- transport repository bundles or deltas through the existing file sync machinery where practical +- preserve repository isolation on disk + +This lets us ship the feature before solving full multi-device replication. + +## Adapters + +Keep the script adapter model from the prototype. + +Adapter shape: + +- `adapter.toml` +- schema block or schema reference +- sync command +- config fields +- trust tier +- optional OAuth definition + +Why keep it: + +- it is simple +- it broadens the adapter ecosystem +- it does not fight the main Rust architecture +- it keeps external-source support decoupled from the VDFS + +Adapters should be installed and discovered through the library feature, not as a separate server product. + +## Secrets and OAuth + +Do not keep the prototype's app-level silo if the library already has stronger primitives available. + +Recommended direction: + +- secrets remain managed by Spacedrive's existing secure storage model where possible +- repository config stores secret references, not raw values +- OAuth tokens are attached to the library-owned repository resource + +If the prototype secrets store is easier to land initially, it can be embedded behind the repository engine and migrated later. The UI and ops surface should not expose that implementation detail. + +## Safety, Classification, and Trust + +These concepts map well to Spacedrive. + +- trust tier belongs to the repository metadata +- safety verdict and quality metadata belong to repository records +- quarantine is a repository view, not a VDFS concept +- classification and embedding stages should use library jobs + +We should keep the prototype's pipeline ordering: + +```text +adapter ingest + -> screening + -> classification + -> embedding + -> searchable +``` + +The first slice can ship with: + +- screening +- embedding +- trust tiers +- quarantine visibility + +Classification can follow as a later job-backed phase. + +## UI Integration + +Do not mirror the prototype desktop app structure directly. + +Instead, add repository features into the existing interface: + +- library section for repositories +- add-source flow +- repository detail page +- repository search page +- quarantine queue +- adapter settings and installed adapters + +This keeps one product and one navigation model. + +## Migration Strategy + +### Phase 0: Design and Carve-Out + +- add this design doc +- choose final module path under `core/src/` +- decide which prototype modules copy over largely intact +- define library metadata schema for repository registration + +### Phase 1: Engine Import + +- port schema parser, codegen, repository DB, search router, adapter runtime +- remove OpenAPI and standalone server assumptions +- wrap the imported subsystem in a library-scoped manager + +### Phase 2: Library Registration + +- add library models and migrations for repository metadata +- create repository folders under `.sdlibrary/repositories/` +- expose create/list/get/delete ops + +### Phase 3: Sync and Pipeline Jobs + +- add sync action and `RepositorySyncJob` +- port screening and embedding stages +- wire progress and status events + +### Phase 4: UI Slice + +- add repository list, create flow, detail page, and search page +- expose quarantine state + +### Phase 5: Device Sync + +- propagate repository metadata through library sync +- later add payload transfer strategy + +### Phase 6: Unified Search + +- optional federated search query across VDFS and repositories + +## Expected Reuse from the Prototype + +Likely to port mostly intact: + +- schema parser/codegen/migration +- repository manager and repository DB +- script adapter runtime +- vector store integration +- search router and result model +- safety model and trust policy model + +Needs translation into V2 concepts: + +- top-level `Engine` +- standalone CLI surface +- standalone Tauri/OpenAPI server surface +- app-owned secrets and settings flows +- desktop route structure + +## Risks + +### Dynamic Schema vs Existing ORM + +Dynamic repository tables are a poor fit for SeaORM. Forcing convergence here would slow the project down. + +### Search Creep + +Trying to unify file search and repository search in the first pass will expand scope fast. + +### Sync Scope + +Full repository payload sync is valuable, but not required to land the first product slice. + +### Over-Refactoring + +The goal is not to perfect the architecture first. The goal is to land the repository engine in the official product with clean boundaries. + +## Open Questions + +1. Should repository metadata live in `library.db` directly or in a small library-owned `repositories/registry.db`? +2. Should repository secrets attach to the global key manager immediately or stay engine-local for the first slice? +3. Should repository search results reuse the existing search result envelope or define a repository-specific output first? +4. How do we want repository payload sync to package data, file sync of repository folders, bundle export/import, or delta protocol? + +## Recommendation + +Start with the smallest honest integration: + +- keep repositories separate from the VDFS index +- make them library-scoped resources +- port the prototype internals with minimal rewrites +- expose everything through ops and jobs +- ship repository-local search before unified search + +This gets the archival product into official Spacedrive quickly without abandoning the VDFS or reopening the whole architecture. diff --git a/justfile b/justfile new file mode 100644 index 000000000..f81418544 --- /dev/null +++ b/justfile @@ -0,0 +1,43 @@ +# Spacedrive development commands + +# Install JS dependencies and set up native deps + cargo config +setup: + bun install + cargo xtask setup + +# Run the daemon (default dev workflow: just dev-daemon + just dev-desktop) +dev-daemon *ARGS: + cargo run --bin sd-daemon {{ARGS}} + +# Run the desktop app in dev mode +dev-desktop: + cd apps/tauri && bun run tauri:dev + +# Run the headless server (web UI, no desktop app) +dev-server *ARGS: + cargo run --bin sd-server {{ARGS}} + +# Run all workspace tests +test: + cargo test --workspace + +# Build everything (default members) +build: + cargo build + +# Build in release mode +build-release: + cargo build --release + +# Format and lint +check: + cargo fmt --check + cargo clippy --workspace + +# Format code +fmt: + cargo fmt + +# Run the CLI +cli *ARGS: + cargo run --bin sd-cli -- {{ARGS}}