mirror of
https://github.com/spacedriveapp/spacedrive.git
synced 2026-11-03 05:45:04 -05:00
Jamie Pineandcapy-ai-staging[bot]
312dbf39b7
* Extension tasks run through the host: spacedrive_op, task.begin/end with deadline, manifest grants parsed strictly
JobContext::run executes a #[task] with the retries and timeout the attribute
declares. The host records each attempt in the job log and enforces the
deadline: once it passes, every host operation the task makes fails with
timeout, which is the only way a single-threaded guest can be stopped.
spacedrive_op is the one import every data-returning operation goes through;
the answer is written into memory from the guest's own wasm_alloc. The old
spacedrive_call, its fixed-offset writer and the Wire-method permission model
go; ManifestPermissions is the grant shape photos declares (read_records,
read/write_sidecars, write_tags, write_custom_fields, dispatch_jobs,
use_models) and refuses unknown fields.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Entropy and clock host functions
wasm32-unknown-unknown has neither. spacedrive_random fills guest memory
from the host's RNG and the SDK installs it as getrandom's custom backend,
so Uuid::new_v4 and rand work in a guest built with
--cfg getrandom_backend="custom". spacedrive_now_ms is the host's clock
behind spacedrive_sdk::clock::now(), which replaces Utc::now() in guests.
Neither needs a manifest grant: random ids and the time are not library
data.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Records replace entries in the SDK, read from the library's source stores
The entry world is gone, so the SDK's query and read surface is defined
over source-store records: Record (uuid, source, name, kind, extension,
source-relative path, size, mtime, content uuid), get_record, read_record
and a RecordQuery narrowed by source, directory and extension. The host
answers from the stores of the job's library and reads bytes through the
source's resolved root; the manifest's read_records grant (its glob names
extensions) is checked on every call. A tag filter is refused rather than
ignored until tags have a host side.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sidecars for extensions, keyed by content uuid and an extension-declared kind
SidecarKind opens: Extension { extension_id, kind } beside the core kinds,
living under extensions/<id>/<kind> in the library's content-addressed
sidecar tree (sd-sidecar-path owns the directory), so an extension's kinds
never collide with the core's or another's. The SDK's sidecar_exists,
save_sidecar and read_sidecar write and read one JSON document per content
and kind; write_sidecars in the manifest grants writing, read_sidecars or
the extension's own write kinds grant reading. save_sidecar drops its
extension id argument, since the host supplies the namespace.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Extension data models: #[model] facets in an extension-owned store
#[model] derives a facet definition from the struct (strings and uuids,
integers, floats, booleans, datetimes; anything else as json) and
#[extension(models = [...])] declares each one to the host in plugin_init.
The host collects them into a DataTypeSchema and opens an sd-store database
under <library>/extensions/<extension id>/ on the first model operation:
the same record and facet shape as a source store, owned by the extension,
which is the store the extension-kinds plan registers as an app-owned
source later. create_model, get_model and update_model key a model by its
own uuid; the _for_content and _by_content forms key it by the content it
describes; query_models lists (filters are refused until a host side
exists). sd-store gains facet_rows, which reads a model's declared fields
back with their types.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* AI inference answers not_available; config.get; photos handles a missing detector
ctx.ai() calls go to the host as ai.infer with the model id, task and raw
input. The host checks the manifest's use_models grant and then, since the
core has no provider for face detection, scene classification, embeddings
or language models, answers not_available naming the category; nothing
builds inference here. analyze_photos handles it: one warning, the photo
keeps no faces sidecar so a later run with a provider picks it up, and the
job completes. config.get serves config.json beside the manifest (or {})
behind ctx.config::<C>(), which the clustering task reads. The photos
extension's DBSCAN and person matching are implemented over the model
store so the job runs to its end instead of a todo!().
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* extension: fixture library and end-to-end acceptance for host functions
The test extension gains a catalog job that reads every record of the
library's sources, digests each file through a task, writes a digest
sidecar and a CatalogEntry model, and skips what a previous run already
digested. The acceptance suite builds a library with twelve JPEG-magic
files and two text files, runs catalog twice, then runs the photos
extension's analyze_photos over the same library and asserts the
not_available path: one warning, no faces sidecar, every task attempt
logged.
Checkpointed guest state now tolerates fields added since the
checkpoint was written (serde(default) on the state structs), and
SourceDb::facet_rows numbers every bind parameter: sqlx binds a bare ?
after a ?N by position, so the external id landed in LIMIT.
Regenerates the TypeScript types for the SidecarKind::Extension variant.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* docs: extension host functions phase 1
The extension README describes spacedrive_op, the operations table and
the model registry in place of permissions.rs and spacedrive_call. The
SDK reference records the Entry to Record rename, the save_sidecar
signature, what has a host side and what does not. The extension READMEs
show the manifest permissions shape and the record, sidecar and model
calls that run today. PROJECT_STATUS gains the phase 1 register row.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* extension: grant checks fail closed; guest identifiers validated
A read_records glob the parser cannot read used to mean no filter for
records.query and no access for records.get, so the manifest is refused
at load unless the glob ends in *.ext or *.{a,b}, and granted_extensions
names nothing for anything else. The loader also refuses an extension id
that is not one path-safe segment, since it becomes a directory under
the library for sidecars and the model store. register_model checks
every field name as an identifier (and refuses record_uuid, the facet
key) because the guest can hand it any JSON. spacedrive_random resolves
the guest slice before allocating the entropy buffer.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
Spacedrive Core
Rust library implementing the Virtual Distributed File System (VDFS) architecture for local-first, AI-native file management.
Architecture
Structure
src/
├── domain/ # Core data models (Entry, Library, Device)
├── ops/ # Operations (actions and queries, CQRS pattern)
├── infra/ # Infrastructure (database, events, wire protocol)
├── service/ # High-level services (network, jobs, sessions)
├── location/ # Location management and indexing
├── library/ # Library lifecycle and operations
├── device/ # Device identity and management
├── volume/ # Volume detection and fingerprinting
├── config/ # Application configuration
├── crypto/ # Cryptographic primitives
└── bin/ # Binaries (cli, daemon)
Core Components
Core Manager (Core struct in lib.rs)
- Coordinates all subsystems
- Manages application lifecycle
- Provides unified access to capabilities
Library (library/)
- File-based storage (
.sdlibrarydirectories) - SQLite database with SeaORM
- Job management and thumbnail generation
- Device registration and sync coordination
Entry-Centric Model (domain/entry.rs)
- Unified representation for files and directories
- Conditional UUIDs (directories immediate, files after content ID)
- On-demand UserMetadata creation
- Relative paths from location roots
Device Management (device/)
- Single device identity per installation
- Persistent across restarts (
device.json) - Sync leadership model per library
- Network address tracking for P2P
Operations Layer (ops/)
- CQRS pattern: Actions (mutations) and Queries (reads)
- Wire protocol with automatic type generation
- Registry system using
inventorycrate - Shared across daemon RPC, iOS FFI, and WASM extensions
Infrastructure (infra/)
api/: Wire protocol dispatcher and RPC serverevent/: Event bus for state changesaction/: Transactional action system with preview-commit-verifyquery/: Read-optimized query handlers
Networking (service/network/)
- Iroh-based P2P networking
- Device pairing protocol
- File transfer (Spacedrop)
- mDNS local discovery
Jobs (service/jobs/)
- Durable, resumable operations
- MessagePack serialization for state
- Progress reporting and cancellation
- Per-library job managers
Indexing (location/indexer/)
- Five-phase pipeline (discover, classify, extract, thumbnail, cleanup)
- File system watcher integration
- Rules engine with glob patterns
- Resumable with checkpoints
Volume Management (volume/)
- Cross-platform volume detection
- Fingerprinting for identity
- Mount point tracking
Communication Patterns
Daemon-Client (Desktop/CLI)
- Unix socket JSON-RPC 2.0
- Wire method strings (e.g.,
query:vdfs.list_entries) - Operations auto-register at compile time
Embedded FFI (iOS/Mobile)
- Direct Rust library integration
- Same JSON-RPC protocol over FFI
- Swift client with Specta-generated types
Extensions (WASM)
- Sandboxed WASM modules
- Minimal host functions (log, register_job)
- SDK with procedural macros (
crates/sdk/) - Models, jobs, actions, agents, UI manifests
Key Technologies
- Async runtime: tokio
- Database: SQLite via SeaORM
- Serialization: serde, Specta for type generation
- Networking: Iroh (P2P), mdns-sd (local discovery)
- WASM: wasmer runtime, spacedrive-sdk
- Jobs: inventory for registration, rmp-serde for state
- Crypto: blake3 (content addressing), ed25519 (signing)
- Indexing: notify (fs watcher), globset (rules)
Building
# Full build
cargo build --release
# With optional features
cargo build --features ffmpeg,ai,heif
# Specific binary
cargo build --bin spacedrive
cargo build --bin daemon
# Run CLI
cargo run --bin spacedrive -- --help
Binaries
spacedrive: CLI interfacedaemon: Background daemon process
Development
- Uses CQRS and DDD patterns
- Operations register via
inventorycrate macros - Resumable jobs with MessagePack state serialization
- Type-safe Wire protocol with Specta generation
- Event-driven architecture with EventBus
- No layered architecture (direct Rust patterns)
Testing
# All tests
cargo test
# Specific module
cargo test --lib location::indexer
# Integration tests
cargo test --test indexer_test
See /docs/core/ for detailed architecture documentation.