diff --git a/apps/docs b/apps/docs index 2d9781443..5c30c33d8 160000 --- a/apps/docs +++ b/apps/docs @@ -1 +1 @@ -Subproject commit 2d97814439f3e94406a7f102c94914dc3cdf62fd +Subproject commit 5c30c33d8857d397ef459e144d225297e7380848 diff --git a/docs/MIGRATION_STRATEGY.md b/docs/MIGRATION_STRATEGY.md new file mode 100644 index 000000000..ff5303ac8 --- /dev/null +++ b/docs/MIGRATION_STRATEGY.md @@ -0,0 +1,390 @@ +# Spacedrive V1 → V2 Migration & Release Strategy + +## Current Situation + +**V1 (Tauri/TypeScript):** +- Last release: v0.4.3 +- ~600k downloads total +- Auto-updater active on macOS/Windows (uses Tauri updater plugin) +- Data location: `~/Library/Application Support/spacedrive/` (macOS) +- 35k GitHub stars on main repo +- Community expects updates through auto-updater + +**V2 (Rust/Swift):** +- Complete rewrite, different architecture +- CLI + daemon (Rust) +- Native macOS app (Swift, can connect to daemon but doesn't bundle it yet) +- Native iOS app (Swift, bundles core via FFI) +- No Tauri, no TypeScript, no React +- Libraries: `.sdlibrary/` format (incompatible with V1) + +## Critical Challenges + +### 1. **Breaking Change in Every Way** +- Different UI (SwiftUI vs React) +- Different data format (`.sdlibrary` vs V1 database) +- Different architecture (embedded core vs daemon) +- No backward compatibility possible + +### 2. **Auto-Updater Expectations** +- 600k V1 users expect updates via auto-updater +- V1 updater endpoint: `https://spacedrive.com/api/releases/tauri/{{version}}/{{target}}/{{arch}}` +- If we push v2.0.0 through V1's updater, it will try to replace Tauri app with... nothing (V2 has no Tauri app) + +### 3. **Version Numbering Dilemma** +- V1 is at 0.4.3 +- V2 deserves to be 2.0.0 (or 1.0.0) +- But jumping from 0.4.3 → 2.0.0 needs careful handling + +### 4. **Multiple Release Artifacts** +- CLI/daemon (cross-platform) +- macOS app (.app bundle) +- iOS app (App Store only) +- Eventually: Android, Windows native, Linux native + +## Recommended Strategy + +### Phase 1: Soft Launch (Week 1) + +**Goal:** Get V2 out without breaking V1 users + +1. **Merge V2 into main repo** but keep V1 updater **unchanged** + - Don't push any updates through V1's auto-updater yet + - This preserves V1 users' current state + +2. **Release V2 as opt-in downloads:** + - GitHub Release tagged `v2.0.0-alpha.1` + - Manual downloads for early adopters: + - `spacedrive-cli-{platform}.tar.gz` - CLI + daemon + - `Spacedrive.app.tar.gz` - macOS app (unsigned for now) + - iOS app: TestFlight beta (App Store takes time) + +3. **Documentation:** + - Clear README explaining V1 vs V2 + - Migration guide: "V2 uses new `.sdlibrary` format - your V1 libraries won't be migrated yet" + - Set expectations: V2 is alpha, V1 is stable (no more updates) + +4. **Communicate:** + - Blog post: "Spacedrive V2: A Complete Reimagining" + - Discord announcement + - GitHub Discussions post + - Be transparent: This is a ground-up rewrite, not an update + +### Phase 2: CLI/Daemon Releases (Week 2-4) + +**Goal:** Establish V2 distribution for power users + +1. **Set up GitHub Actions for CLI:** + ```yaml + # Build for: macOS (x64/arm64), Linux (x64/arm64), Windows (x64) + # Artifacts: spacedrive-cli-{os}-{arch}.tar.gz + # Install: Extract to /usr/local/bin or add to PATH + ``` + +2. **Homebrew formula** (macOS/Linux): + ```ruby + class Spacedrive < Formula + desc "Virtual Distributed File System" + homepage "https://spacedrive.com" + url "https://github.com/spacedriveapp/spacedrive/releases/v2.0.0-alpha.1/spacedrive-cli-macos-arm64.tar.gz" + # ... + end + ``` + +3. **Version:** Start at `2.0.0-alpha.1`, increment alpha versions weekly + +### Phase 3: Native macOS App (Month 2) + +**Goal:** Replace Tauri app with native Swift app + +1. **Bundle daemon with macOS app:** + - App bundle includes `sd-daemon` binary + - App can launch its own daemon OR connect to user-installed daemon + - Auto-detect: If daemon running, connect. Else, launch bundled daemon. + +2. **Code signing & notarization:** + - Get Apple Developer cert + - Sign both app and daemon binary + - Notarize for Gatekeeper + +3. **Distribution:** + - DMG installer + - **Separate from V1's update channel** (new identifier: `com.spacedrive.v2`) + - Sparkle framework for future auto-updates (not Tauri updater) + +### Phase 4: V1 Sunset Communication (Month 3) + +**Goal:** Gracefully deprecate V1 without breaking users + +**Option A: One-Time Migration Prompt (Recommended)** + +Create a **final V1 update (v0.4.4)** that: +1. Shows in-app notice: "Spacedrive V2 is available" +2. Offers to: + - Back up V1 data to `~/Library/Application Support/spacedrive-v1-backup/` + - Download V2 installer + - Disable V1's auto-updater (prevent future interruptions) +3. Does **NOT** auto-install V2 (user choice) + +**Implementation:** +```rust +// V1's final update: apps/desktop/src/v2-migration.tsx +async function showV2MigrationPrompt() { + const result = await dialog.confirm({ + title: "Spacedrive V2 Available", + message: "A ground-up rewrite with improved performance. Your V1 libraries will need to be re-indexed. Back up V1 data and download V2?", + okLabel: "Download V2", + cancelLabel: "Stay on V1" + }); + + if (result) { + await backupV1Data(); + await openURL("https://spacedrive.com/download/v2"); + await disableAutoUpdater(); + } +} +``` + +**Option B: Parallel Coexistence (Conservative)** + +- Leave V1 auto-updater pointing to v0.4.3 forever +- V1 and V2 can coexist (different data dirs, different app bundles) +- Users manually migrate when ready +- Downside: Confusing, users don't know which to use + +### Phase 5: iOS App Store (Month 3-4) + +1. App Store submission (2-4 week review process) +2. Separate app ID from V1 (if V1 was ever on App Store, which it wasn't) +3. Version: Start at `2.0.0` (App Store guidelines) + +### Phase 6: Stable Release (Month 6) + +1. After alpha testing, move to **`v2.0.0` stable** +2. Update main README to default to V2 +3. V1 becomes "legacy" branch (archive, no more updates) +4. All auto-update channels point to V2 + +## Version Numbering Plan + +``` +V1 (Tauri): +├── v0.4.3 (current, last release) +└── v0.4.4 (final, migration prompt) [optional] + +V2 (Rust/Swift): +├── v2.0.0-alpha.1 (initial release) +├── v2.0.0-alpha.2 +├── v2.0.0-alpha.N +├── v2.0.0-beta.1 +├── v2.0.0-beta.N +└── v2.0.0 (stable, public launch) +``` + +**Rationale for 2.0.0:** +- Skipping 1.0.0 signals this is V2, not V1 +- Matches "Version 2.0" in README +- Clear break from V1's 0.x versioning + +## Data Migration Strategy + +**Current Status:** No migration path exists + +**Short-term (Alpha/Beta):** +- V2 uses `.sdlibrary/` format +- Users create fresh libraries +- Document: "V1 libraries not migrated yet - you'll re-index" + +**Long-term (Post-2.0.0):** +Potential migration tool: +```bash +# Future: spacedrive migrate-from-v1 +sd-cli migrate-v1 ~/Library/Application\ Support/spacedrive/ +# Outputs: ~/Documents/Migrated.sdlibrary/ +``` + +Challenges: +- V1 and V2 database schemas completely different +- V1 used Prisma, V2 uses SeaORM +- Locations, tags, file metadata - all different structure + +**Realistic timeline:** Post-launch, low priority (most users will re-index) + +## CI/CD Strategy + +### GitHub Actions Workflows + +**.github/workflows/release-cli.yml:** +```yaml +name: Release CLI +on: + push: + tags: ['v2.*'] +jobs: + build: + strategy: + matrix: + os: [ubuntu-latest, macos-latest, windows-latest] + arch: [x86_64, aarch64] + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - run: cargo build --release -p sd-cli + - run: tar -czf spacedrive-cli-${{ matrix.os }}-${{ matrix.arch }}.tar.gz target/release/sd-cli + - uses: actions/upload-artifact@v4 +``` + +**.github/workflows/release-macos.yml:** +```yaml +name: Release macOS App +on: + push: + tags: ['v2.*'] +jobs: + build: + runs-on: macos-latest + steps: + - uses: actions/checkout@v4 + - uses: maxim-lobanov/setup-xcode@v1 + - run: xcodebuild -project apps/macos/Spacedrive.xcodeproj -scheme Spacedrive archive + - run: create-dmg Spacedrive.app + # TODO: Code signing with secrets.APPLE_CERTIFICATE +``` + +### Auto-Update Architecture (V2) + +**Don't use Tauri updater** (we're not using Tauri) + +**For macOS app, use Sparkle:** +```swift +// macOS app uses Sparkle framework +import Sparkle + +let updater = SPUStandardUpdaterController( + startingUpdater: true, + updaterDelegate: nil, + userDriverDelegate: nil +) + +// Appcast URL: https://spacedrive.com/api/releases/v2/appcast.xml +``` + +**For CLI/daemon:** +- Manual updates via `sd-cli update` +- Or package managers (Homebrew, apt, etc.) + +## Repository Structure After Merge + +``` +spacedriveapp/spacedrive/ +├── main branch (V2 code) +├── v1 branch (archived V1 code, no more development) +├── releases/ +│ ├── v0.4.3 (V1 final stable) +│ ├── v0.4.4 (V1 migration prompt, optional) +│ ├── v2.0.0-alpha.1 +│ └── v2.0.0-alpha.N +└── README.md (updated for V2, link to V1 branch) +``` + +## Communication Plan + +### Day of Merge: + +1. **Blog Post:** "Spacedrive V2: The Complete Rewrite" + - Why we rewrote from scratch + - What's new (SdPath, sync, performance) + - Download V2 alpha + - V1 remains available, no forced upgrades + +2. **GitHub Release Notes (v2.0.0-alpha.1):** + ```markdown + # Spacedrive V2.0.0 Alpha 1 + + This is a **complete rewrite** of Spacedrive with a new architecture. + + **NOT compatible with V1 libraries.** You'll need to create new libraries. + + Downloads: + - CLI: [Download for macOS/Linux/Windows] + - macOS app: [Download .dmg] (unsigned) + - iOS: Coming to TestFlight + + V1 (Tauri app) remains at v0.4.3 and will receive no more updates. + ``` + +3. **Discord Announcement:** + - Pin in #announcements + - FAQ: "Will my V1 data migrate?" → "Not yet, create fresh libraries" + +4. **README Update:** + - Prominent V2 alpha notice at top + - "Looking for V1? See the [v1 branch](https://github.com/spacedriveapp/spacedrive/tree/v1)" + +### Ongoing: + +- Weekly alpha releases +- Changelog for each release +- Community feedback in Discord/Discussions + +## Risk Mitigation + +### Risk: Users angry about no migration +**Mitigation:** +- Clear communication upfront +- V1 remains available +- Promise future migration tool + +### Risk: V1 auto-updater breaks +**Mitigation:** +- Don't touch V1's updater endpoint +- Keep v0.4.3 as perpetual "latest" for V1 + +### Risk: V2 alpha is buggy, scares users +**Mitigation:** +- Label clearly as "alpha" +- V1 remains stable option +- Rapid iteration on V2 (weekly alphas) + +### Risk: Code signing delays macOS release +**Mitigation:** +- Release unsigned .app first (users can right-click → Open) +- Get code signing sorted in parallel +- Re-release signed version as v2.0.0-alpha.2 + +## Timeline + +| Week | Milestone | +|------|-----------| +| 1 | Merge V2 to main, tag v2.0.0-alpha.1, blog post | +| 2-3 | Set up CLI CI/CD, Homebrew formula | +| 4-6 | Weekly alpha releases, bug fixes | +| 6-8 | macOS app code signing, DMG installer | +| 8-10 | iOS TestFlight beta | +| 10-12 | Beta releases (v2.0.0-beta.N) | +| 12+ | Stable v2.0.0 release | + +## Next Steps (This Weekend) + +1. **Prepare merge:** + - Create `v1` branch from current main + - Update V2's README + - Write blog post draft + +2. **Set up releases:** + - GitHub Actions for CLI builds + - Create v2.0.0-alpha.1 release (manual upload for now) + +3. **Test locally:** + - Verify V1 and V2 can coexist + - Different data dirs: `~/Library/Application Support/spacedrive/` vs `~/Documents/*.sdlibrary/` + +4. **Communicate:** + - Draft Discord announcement + - Prepare FAQ for common questions + +## Conclusion + +**Don't hijack V1's auto-updater.** V2 is a different app. Treat it as a parallel product initially, then gracefully deprecate V1 once V2 is stable. + +The 600k V1 users deserve a smooth transition, not a forced breaking upgrade. Give them choice, clear communication, and time to migrate. diff --git a/docs/MINTLIFY_INTEGRATION_PLAN.md b/docs/MINTLIFY_INTEGRATION_PLAN.md new file mode 100644 index 000000000..708f0a9f6 --- /dev/null +++ b/docs/MINTLIFY_INTEGRATION_PLAN.md @@ -0,0 +1,423 @@ +# Mintlify Documentation Integration Plan + +## Overview + +Integrate Mintlify docs at https://docs.spacedrive.com while keeping docs in the main repo (not a separate docs repo). + +## Architecture + +``` +spacedriveapp/spacedrive/ # Main repo (source of truth) +├── docs/ +│ ├── core/*.md # Core architecture docs +│ ├── cli/*.md # CLI documentation +│ ├── troubleshooting/*.md # Troubleshooting guides +│ ├── design/ # EXCLUDED from published docs +│ ├── whitepaper.md +│ ├── philosophy.md +│ ├── history.md +│ └── sdk.md + +spacedriveapp/docs/ # Mintlify deployment repo (submodule) +├── mint.json # Mintlify config +├── docs -> ../docs/ # Symlink to main repo docs +├── .gitignore # Ignore design/ folder +└── .github/workflows/mintlify.yml # Auto-deploy on push +``` + +## Implementation Steps + +### 1. Create Mintlify Repo + +```bash +# Create new repo on GitHub: spacedriveapp/docs +cd ~/Projects +gh repo create spacedriveapp/docs --public --clone + +cd docs +git init +``` + +### 2. Set Up Mintlify + +```bash +cd ~/Projects/docs + +# Initialize Mintlify +npm i -g mintlify +mintlify init + +# This creates mint.json and example docs +``` + +### 3. Configure Symlink Strategy + +**Option A: Symlink entire docs folder (Recommended)** + +```bash +cd ~/Projects/docs + +# Remove example docs +rm -rf docs/ + +# Create symlink to main repo's docs +ln -s ~/Projects/spacedrive/docs ./docs + +# Create .gitignore to exclude design folder from publishing +cat > .gitignore << EOF +docs/design/ +*.DS_Store +EOF + +git add . +git commit -m "Initial Mintlify setup with symlinked docs" +git push origin main +``` + +**Option B: Selective symlinks (More control)** + +```bash +cd ~/Projects/docs +mkdir -p docs + +# Symlink only what you want published +ln -s ~/Projects/spacedrive/docs/core docs/core +ln -s ~/Projects/spacedrive/docs/cli docs/cli +ln -s ~/Projects/spacedrive/docs/troubleshooting docs/troubleshooting +ln -s ~/Projects/spacedrive/docs/whitepaper.md docs/whitepaper.md +ln -s ~/Projects/spacedrive/docs/philosophy.md docs/philosophy.md +ln -s ~/Projects/spacedrive/docs/history.md docs/history.md +ln -s ~/Projects/spacedrive/docs/sdk.md docs/sdk.md + +# Don't symlink design/ - it stays private +``` + +### 4. Convert .md to .mdx + +Mintlify prefers MDX for components. Two approaches: + +**Quick (Keep as .md):** +Mintlify supports plain Markdown. Just use `.md` files - they work fine. + +**Future-proof (Convert to .mdx):** +```bash +cd ~/Projects/spacedrive/docs + +# Rename all .md to .mdx (safe, MDX is superset of MD) +find . -name "*.md" -type f -not -path "./design/*" -exec bash -c 'mv "$0" "${0%.md}.mdx"' {} \; + +# Update any references +git grep -l '\.md' | xargs sed -i '' 's/\.md/.mdx/g' + +git commit -am "Convert docs to MDX for Mintlify" +``` + +### 5. Configure mint.json + +```json +{ + "name": "Spacedrive", + "logo": { + "dark": "/logo/dark.svg", + "light": "/logo/light.svg" + }, + "favicon": "/favicon.png", + "colors": { + "primary": "#2AB673", + "light": "#55D799", + "dark": "#0D9373", + "anchors": { + "from": "#2AB673", + "to": "#55D799" + } + }, + "topbarLinks": [ + { + "name": "GitHub", + "url": "https://github.com/spacedriveapp/spacedrive" + } + ], + "topbarCtaButton": { + "name": "Download", + "url": "https://spacedrive.com/download" + }, + "tabs": [ + { + "name": "Core", + "url": "core" + }, + { + "name": "CLI", + "url": "cli" + }, + { + "name": "SDK", + "url": "sdk" + } + ], + "anchors": [ + { + "name": "Discord", + "icon": "discord", + "url": "https://discord.gg/gTaF2Z44f5" + }, + { + "name": "GitHub", + "icon": "github", + "url": "https://github.com/spacedriveapp/spacedrive" + } + ], + "navigation": [ + { + "group": "Getting Started", + "pages": [ + "docs/whitepaper", + "docs/philosophy", + "docs/history" + ] + }, + { + "group": "Core Architecture", + "pages": [ + "docs/core/architecture", + "docs/core/library", + "docs/core/domain-models", + "docs/core/indexing", + "docs/core/locations", + "docs/core/devices", + "docs/core/networking", + "docs/core/pairing", + "docs/core/sync", + "docs/core/library-sync", + "docs/core/tagging", + "docs/core/virtual-sidecars", + "docs/core/ops", + "docs/core/events", + "docs/core/database", + "docs/core/testing", + "docs/core/volume-system" + ] + }, + { + "group": "CLI", + "pages": [ + "docs/cli/cli", + "docs/cli/index-verify" + ] + }, + { + "group": "SDK", + "pages": [ + "docs/sdk" + ] + }, + { + "group": "Troubleshooting", + "pages": [ + "docs/troubleshooting/file-descriptors" + ] + } + ], + "footerSocials": { + "github": "https://github.com/spacedriveapp/spacedrive", + "discord": "https://discord.gg/gTaF2Z44f5", + "twitter": "https://twitter.com/spacedriveapp" + }, + "analytics": { + "plausible": { + "domain": "docs.spacedrive.com" + } + } +} +``` + +### 6. Add to Main Repo as Submodule + +```bash +cd ~/Projects/spacedrive + +# Add docs repo as submodule +git submodule add https://github.com/spacedriveapp/docs.git mintlify-docs + +# This creates mintlify-docs/ folder pointing to spacedriveapp/docs repo +``` + +**Why submodule?** +- Keeps deployment config (mint.json, Mintlify settings) separate +- Main repo doesn't get polluted with Mintlify infrastructure +- Docs repo can deploy independently via Mintlify's GitHub integration + +### 7. Set Up Auto-Deploy + +Mintlify auto-deploys from GitHub. Configure in Mintlify dashboard: + +1. Go to https://dashboard.mintlify.com +2. Connect `spacedriveapp/docs` repo +3. Set branch: `main` +4. Set root directory: `/` +5. Custom domain: `docs.spacedrive.com` + +Mintlify watches the repo - any push to `main` auto-deploys. + +### 8. Workflow for Updating Docs + +**Scenario: Edit core/architecture.md** + +```bash +cd ~/Projects/spacedrive + +# Edit docs as usual in main repo +vim docs/core/architecture.md + +# Commit to main repo +git add docs/core/architecture.md +git commit -m "Update architecture docs" +git push + +# Docs repo auto-updates via symlink +cd mintlify-docs +git add docs/core/architecture.md # Follows symlink +git commit -m "Update architecture docs (via main repo)" +git push + +# Mintlify auto-deploys +``` + +**Optional: Automate with Git Hook** + +```bash +# In main repo: .git/hooks/post-commit +#!/bin/bash +# Auto-sync docs changes to Mintlify repo + +if git diff-tree --name-only --no-commit-id -r HEAD | grep '^docs/'; then + echo "Docs changed, syncing to Mintlify repo..." + cd mintlify-docs + git add docs/ + git commit -m "Sync: $(git log -1 --pretty=%B ../)" + git push + cd .. +fi +``` + +## Alternative: Unified Repo Approach + +If you don't want a separate docs repo, keep Mintlify config IN the main repo: + +``` +spacedrive/ +├── docs/ # Documentation (source of truth) +├── mint.json # Mintlify config at root +└── .github/ + └── workflows/ + └── mintlify-deploy.yml +``` + +**Pros:** +- Single repo, simpler +- No submodule complexity + +**Cons:** +- Mintlify watches main repo (more CI noise) +- mint.json pollutes root directory +- Harder to exclude design/ folder from Mintlify + +## Recommended: Submodule Approach + +**Why:** +1. **Separation of concerns**: Main repo = code, docs repo = published docs +2. **Clean exclusions**: design/ folder never touches Mintlify repo +3. **Independent deployment**: Docs can redeploy without triggering main repo CI +4. **Mintlify best practice**: They recommend separate docs repos + +## Migration Checklist + +- [ ] Create `spacedriveapp/docs` repo +- [ ] Set up Mintlify project at dashboard.mintlify.com +- [ ] Create mint.json with navigation structure +- [ ] Symlink `spacedrive/docs/` into Mintlify repo +- [ ] Add .gitignore to exclude design/ +- [ ] Convert .md → .mdx (optional, can do later) +- [ ] Add docs repo as submodule to main repo +- [ ] Configure custom domain: docs.spacedrive.com +- [ ] Test deployment +- [ ] Set up Plausible analytics (optional) +- [ ] Add "Documentation" link to main README + +## Domain Setup + +```bash +# DNS settings for docs.spacedrive.com +CNAME docs -> mintlify.app + +# Or if using Cloudflare: +CNAME docs -> .mintlify.app +``` + +Mintlify provides the target - check dashboard after connecting repo. + +## Post-Launch + +Once published: + +1. **Add to README:** + ```markdown + - **📖 Read the [Documentation](https://docs.spacedrive.com)** + ``` + +2. **Deprecate internal docs/:** + - Keep docs/ as source of truth + - Point contributors to docs.spacedrive.com for browsing + - docs/ becomes "raw source", docs.spacedrive.com is "published view" + +3. **Version docs for V2:** + Mintlify supports versioning: + ```json + "versions": ["v2", "v1"] + ``` + +## Example File Structure After Setup + +``` +~/Projects/ +├── spacedrive/ # Main repo +│ ├── docs/ # Source of truth +│ │ ├── core/*.mdx +│ │ ├── cli/*.mdx +│ │ ├── design/ # NOT published +│ │ ├── whitepaper.mdx +│ │ └── ... +│ ├── mintlify-docs/ # Git submodule → spacedriveapp/docs +│ │ ├── docs -> ../docs/ # Symlink +│ │ ├── mint.json +│ │ └── .gitignore (design/) +│ └── README.md + +└── docs/ # Mintlify repo (if you clone separately) + ├── docs -> ~/Projects/spacedrive/docs/ + └── mint.json +``` + +## Quick Start Commands + +```bash +# 1. Create and setup docs repo +gh repo create spacedriveapp/docs --public --clone +cd ~/Projects/docs +npm i -g mintlify +mintlify init +rm -rf docs/ +ln -s ~/Projects/spacedrive/docs ./docs +echo "docs/design/" > .gitignore + +# 2. Add to main repo as submodule +cd ~/Projects/spacedrive +git submodule add https://github.com/spacedriveapp/docs.git mintlify-docs + +# 3. Configure Mintlify dashboard +# - Connect spacedriveapp/docs +# - Set custom domain: docs.spacedrive.com +# - Deploy! +``` + +Let me know if you want me to generate the complete `mint.json` with all your current docs auto-detected! diff --git a/docs/history.md b/docs/history.md deleted file mode 100644 index cab06e730..000000000 --- a/docs/history.md +++ /dev/null @@ -1,783 +0,0 @@ -# Spacedrive: A Historical Chronicle - -## Table of Contents - -1. [Introduction](#introduction) -2. [Origins and Founding Vision (2021-2022)](#origins-and-founding-vision-2021-2022) -3. [The Viral Launch (May 2022)](#the-viral-launch-may-2022) -4. [Early Development and Funding (2022-2023)](#early-development-and-funding-2022-2023) -5. [Technical Architecture Evolution](#technical-architecture-evolution) -6. [Community Growth and Public Reception](#community-growth-and-public-reception) -7. [Key Milestones and Releases](#key-milestones-and-releases) -8. [Why Spacedrive Failed: A Technical Post-Mortem](#why-spacedrive-failed-a-technical-post-mortem) -9. [The V2 Reimagining (2025)](#the-v2-reimagining-2025) -10. [The AI-Augmented Development Revolution](#the-ai-augmented-development-revolution) -11. [Impact and Legacy](#impact-and-legacy) -12. [Lessons Learned from the Failure](#lessons-learned-from-the-failure) -13. [Future Vision](#future-vision) - -## Introduction - -Spacedrive represents one of the most ambitious attempts to revolutionize personal file management in the modern era. Born from frustration with fragmented cloud storage and device ecosystems, it promised to unify all user data under a single, intelligent interface powered by a Virtual Distributed File System (VDFS). This document chronicles the journey from a developer's personal project to a venture-backed open-source phenomenon that captured the imagination of hundreds of thousands of users worldwide. - -## Origins and Founding Vision (2021-2022) - -### The Personal Catalyst - -Jamie Pine, Spacedrive's founder, had been accumulating digital memories since childhood—tens of thousands of photos, project files, and documents scattered across drives and cloud services. Like many digital natives, he found himself trapped in "data fragmentation hell," spending excessive time searching for files across disconnected silos. This personal pain point became the catalyst for something revolutionary. - -### Early Development - -In early 2021, Pine began developing what would become Spacedrive. The core premise was radical yet simple: "files shouldn't be stuck in a device ecosystem." Over 15 months of intense development, he crafted the foundations of a cross-platform file manager that could break free from proprietary cloud silos and give users permanent ownership of their data. - -The initial vision centered on three principles: - -1. **Unification**: One explorer to access files from any device or cloud -2. **Intelligence**: AI-powered organization and search capabilities -3. **Freedom**: No vendor lock-in, complete user control - -## The Viral Launch (May 2022) - -### Open Source Debut - -In May 2022, Pine made the momentous decision to open-source Spacedrive on GitHub. The response exceeded all expectations: - -- **#1 on GitHub Trending** for 3 consecutive days -- **10,000+ stars** within the first week -- **Front page of Hacker News** twice during launch week -- Immediate global attention from developers and tech enthusiasts - -### Why It Resonated - -The viral reception wasn't accidental. Spacedrive addressed a universal problem every computer user faced—file chaos. Its promise to create a "personal distributed cloud" without sacrificing privacy or control struck a chord with: - -- Developers tired of juggling multiple cloud APIs -- Creative professionals managing massive media libraries -- Privacy-conscious users seeking alternatives to big tech -- "Data hoarders" with collections spanning decades - -## Early Development and Funding (2022-2023) - -### Seed Investment and Notable Backers - -On June 13, 2022, Spacedrive announced a $2 million seed round led by OSS Capital's Joseph Jacks. The investor roster read like a who's who of tech leadership: - -- **Naval Ravikant** (AngelList co-founder) -- **Guillermo Rauch** (Vercel CEO) -- **Tobias Lütke** (Shopify CEO) -- **Tom Preston-Werner** (GitHub co-founder) -- **Neha Narkhede** (Apache Kafka co-creator) -- **Haoyuan Li** (Alluxio founder, VDFS paper author) - -This backing validated Spacedrive's potential to "dramatically simplify" the fragmented storage landscape and enabled Pine to build a distributed team. - -### Building the Team - -With funding secured, Spacedrive Technology Inc. was formally established as a fully remote company. The team grew to include: - -- Engineers from Brazil, Jordan, Finland, USA, and beyond -- Ericson Soares as Head of Engineering -- Product designers and community managers -- Over 100 open-source contributors worldwide - -## Technical Architecture Evolution - -### The Original V1 Architecture (2022-2024) - -The first version introduced groundbreaking concepts: - -#### Virtual Distributed File System (VDFS) - -- Unified namespace across all storage locations -- Content-addressable storage using unique file hashes -- Real-time synchronized index using embedded SQLite -- Device-agnostic file organization - -#### The PRRTT Stack - -A modern polyglot architecture combining: - -- **Prisma** (database ORM) -- **Rust** (core backend) -- **React** (UI framework) -- **TypeScript** (frontend logic) -- **Tauri** (native app wrapper) - -#### Key Innovations - -1. **Constant-time hashing** for large files -2. **Peer-to-peer synchronization** across devices -3. **AI-ready metadata extraction** -4. **Cross-platform native apps** with minimal resource usage - -### Performance Metrics (V1) - -- 22,000 GitHub stars by October 2023 -- 149,000+ unique installations by February 2024 -- Average session duration: 54 minutes -- Supported Windows, macOS, Linux, and mobile prototypes - -## Community Growth and Public Reception - -### Developer Enthusiasm - -The GitHub repository became a hub of activity: - -- **35,000+ stars** by 2025 -- **1,100+ forks** -- **117+ contributors** -- Translations in 11 languages -- Active Discord community with thousands of members - -### Media Coverage - -Tech press embraced Spacedrive's vision: - -- **ZDNet**: "The cross-platform file manager of your dreams" -- **It's FOSS**: "A dreamy Rust-based open-source file manager" -- **LinuxLinks**: "The most interesting file manager we've seen in a long time" -- **The New Stack**: "A cross-platform file manager for the modern era" - -### User Feedback Themes - -Early adopters praised: - -- Lightning-fast search across all devices -- Beautiful, space-themed UI -- Unified view of disconnected drives -- Privacy-first approach - -Common reservations: - -- Alpha stability concerns -- Missing sync features -- Incomplete cloud integrations - -## Key Milestones and Releases - -### Timeline of Major Releases - -| Date | Version | Key Features | Significance | -| ---------- | ----------------- | -------------------------------------- | -------------------------------------- | -| Oct 2023 | Alpha 0.1.0 | Basic indexing, preview, search | First public release | -| Feb 2024 | Alpha 0.2.0 | Drag-and-drop, AI labels, 11 languages | 149k installations | -| Mid 2024 | Alpha 0.3.x | Column view, mobile TestFlight | 100+ contributors | -| Late 2024 | Alpha 0.4.x | Spacedrop, content deduplication | 30k+ GitHub stars | -| Early 2025 | Development Pause | Temporary halt announced | 35k stars, 500k installs | -| July 2025 | V2 Architecture | Complete rewrite with AI assistance | 3 weeks dev time, solved all V1 issues | - -### Feature Evolution - -Each release expanded capabilities: - -1. **Indexing**: From read-only browsing to full file operations -2. **Search**: Keyword matching to advanced filters and AI -3. **Organization**: Basic folders to sophisticated tagging -4. **Sync**: Local indexing to P2P device communication -5. **Media**: Simple previews to intelligent galleries - -## Why Spacedrive Failed: A Technical Post-Mortem - -### The Development Pause - -In early 2025, after achieving 35,000 GitHub stars and 500,000 installations, Spacedrive development came to an abrupt halt. The team announced a temporary pause citing funding constraints, but the real story was far more complex. A deep technical analysis reveals that fundamental architectural flaws, decision paralysis, and over-engineering had created an unsustainable development burden. - -### The Fatal Flaw: Dual File Systems - -The most critical architectural mistake was the existence of two completely separate file management systems that couldn't interoperate: - -**1. Indexed System**: Database-driven files with rich metadata, background jobs, and async operations -**2. Ephemeral System**: Direct filesystem access for non-indexed files with immediate operations - -This created an impossible user experience: - -- **Cannot copy between systems**: Users couldn't copy files from their indexed desktop to a non-indexed USB drive -- **Duplicate everything**: Every file operation had to be implemented twice with different APIs -- **User confusion**: "Why can't I copy from my home folder to my indexed desktop?" -- **Maintenance nightmare**: 2x the code, 2x the bugs, 2x the testing - -### The `invalidate_query` Anti-Pattern - -The second major architectural flaw was the query invalidation system that violated fundamental principles: - -```rust -// Backend code knowing about frontend React Query keys -invalidate_query!(library, "search.paths"); -invalidate_query!(library, "search.ephemeralPaths"); -``` - -This created: - -- **Frontend-backend coupling**: Backend hardcoded frontend cache keys -- **Brittle string-based system**: No type safety, prone to typos -- **Scattered invalidations**: Calls spread throughout the codebase -- **Over-invalidation**: Often invalidated entire query categories unnecessarily - -### The Sync System That Never Shipped - -Perhaps the most telling failure was the sync system—a core promise of Spacedrive that never materialized: - -**The Problem**: Mixed local and shared data requirements - -- Some data must sync (file metadata, tags) -- Some data must remain local (preferences, local paths) -- No clear architectural boundary between the two - -**The Over-Engineering**: - -- Custom CRDT implementation built from scratch -- Dual database tables (`cloud_crdt_operation` and `crdt_operation`) -- Complex actor model with multiple concurrent actors -- Analysis paralysis over what should sync - -**Why It Failed**: - -- The team couldn't agree on sync boundaries -- Perfect became the enemy of good -- Should have used existing SQLite sync solutions -- Engineering debates prevented shipping - -### Abandoned Dependencies: Creating Then Abandoning Libraries - -A critical piece of context often missed: The Spacedrive team didn't just use prisma-client-rust and rspc—they **created** them: - -**prisma-client-rust**: - -- Created by Spacedrive team members -- Added custom sync generation with `@shared` and `@local` attributes -- When needs diverged, the library was abandoned -- Left Spacedrive on a deprecated fork of Prisma 4.x -- Prisma moving away from Rust support made this worse - -**rspc**: - -- Also created by Spacedrive team members -- Provides type-safe RPC between Rust and TypeScript -- Abandoned when Spacedrive's needs changed -- Custom modifications in fork created maintenance burden - -This pattern of creating libraries and abandoning them when requirements changed left Spacedrive with significant technical debt. - -### Job System: Death by a Thousand Lines - -The job system, while well-engineered, required 500-1000+ lines of boilerplate to add any new operation: - -```rust -// Required for EVERY new job: -1. Add to JobName enum -2. Implement Job trait (100-200 lines) -3. Implement SerializableJob (100-200 lines) -4. Add to central registry macro -5. Handle serialization/deserialization -6. Write progress tracking -7. Implement error handling -``` - -Result: Simple operations like "copy file" became massive engineering efforts. - -### The Unfulfilled Search Promise - -Despite marketing "lightning fast search across all your files," the search implementation was rudimentary: - -**What Was Promised**: Virtual Distributed File System with instant search everywhere -**What Was Delivered**: Basic SQL `LIKE` queries on local files only - -Missing features: - -- No content search inside documents -- No full-text search indexes -- No vector/semantic search -- Can't search offline drives -- Separate search implementations for indexed vs ephemeral files - -### Identity Crisis: Node vs Device vs Instance - -Three different ways to represent the same concept (a Spacedrive installation): - -``` -Node: P2P identity for the application -Device: Sync system identity for hardware -Instance: Library-specific P2P identity -``` - -This created: - -- Confusion about which ID to use when -- Complex identity mapping between systems -- Data duplication and sync issues -- Made multi-device features exponentially harder - -### Organizational Chaos - -The codebase structure revealed incomplete refactoring: - -``` -/core/src/ - old_job/ # Still referenced - old_p2p/ # Still used - object/fs/ - old_copy.rs # Critical logic here - old_cut.rs # Why "old"? - old_delete.rs # Still in use! -``` - -Both old and new systems ran in parallel throughout the codebase, creating confusion about which to use and when. - -### The Real Reasons for Failure - -1. **Over-Engineering**: Every system was built for a perfect future that never came -2. **Decision Paralysis**: Debates about ideal architecture prevented shipping -3. **Incomplete Migrations**: New systems built without removing old ones -4. **Scope Creep**: Trying to solve every edge case before shipping basics -5. **Technical Debt Accumulation**: Each clever solution created more problems - -## The V2 Reimagining (2025) - -### Acknowledging Reality - -After 3 years of V1 development, the technical analysis revealed: - -- The dual file system made basic operations impossible -- The sync system was fundamentally flawed -- Abandoned dependencies created an unmaintainable codebase -- Job system boilerplate prevented rapid iteration -- Search never fulfilled its core promise -- Identity confusion permeated the architecture - -### The Complete Rewrite - -July 2025 marked a pivotal moment with the V2 whitepaper publication, presenting a ground-up reimplementation that addressed every major flaw: - -#### 1. SdPath Universal Addressing - -The dual file system problem was solved by evolving `SdPath` from a simple struct to a powerful universal addressing system that makes device boundaries transparent: - -```rust -// V2: Universal addressing that works everywhere -#[derive(Clone, Debug)] -pub enum SdPath { - // Physical addressing: device + path - Physical { - device_id: DeviceId, - local_path: PathBuf - }, - // Content-aware addressing: find optimal instance - Content { - cas_id: ContentId - }, -} -``` - -This enables: - -- Addressing files that don't exist locally -- Content-based retrieval across devices -- Future-proof distributed operations -- Clean abstraction over platform differences -- Automatic failover when devices are offline -- Optimal performance routing to fastest available source - -#### 2. Entry-Centric Model - -Replaced the file-centric approach with entries that carry context: - -```rust -pub struct Entry { - pub id: EntryId, - pub path: SdPath, // Universal addressing - pub metadata: Metadata, // Always available - pub content_id: Option, // Progressive enhancement - pub user_data: UserMetadata, // Tags, ratings, etc. -} -``` - -Benefits: - -- Immediate organization without waiting for indexing -- Metadata available even for ephemeral files -- Clean separation between system and user data -- Natural progression from discovery to full indexing - -#### 3. Simplified Sync Architecture - -Complete abandonment of the failed CRDT approach: - -**Domain Separation**: - -``` -┌─────────────────┐ -│ Library Sync │ → What files exist, where -├─────────────────┤ -│ Metadata Sync │ → User tags, ratings -├─────────────────┤ -│ Content Sync │ → Actual file transfer -└─────────────────┘ -``` - -**Clear Boundaries**: - -- Local-only data never enters sync system -- Shared data in separate tables from the start -- No mixed concerns, no confusion -- Third-party sync solutions become possible - -#### 3. Event-Driven Architecture - -Replaced the `invalidate_query` anti-pattern: - -```rust -// V2: Clean event system -pub enum DomainEvent { - EntryCreated(Entry), - EntryModified(EntryId, Changes), - EntryDeleted(EntryId), - // ... domain-specific events -} - -// Frontend subscribes to what it needs -eventBus.subscribe(|event| { - // Update UI based on domain events -}); -``` - -#### 4. Pragmatic Job System - -Reduced from 1000+ lines to ~50 lines per job: - -```rust -#[derive(Job)] -pub struct CopyFiles { - source: Vec, - destination: SdPath, -} - -impl Execute for CopyFiles { - async fn run(&self, ctx: Context) -> Result<()> { - // Just the business logic - } -} -``` - -Procedural macros handle all boilerplate, making new operations trivial to add. - -#### 5. Real Search Implementation - -Finally delivering on the VDFS promise: - -```rust -pub struct SearchEngine { - content_index: ContentIndex, // Full-text search - metadata_index: MetadataIndex, // Fast attribute queries - vector_store: VectorStore, // Semantic search -} - -// Unified search across all dimensions -let results = search - .query("vacation photos from last summer") - .with_content_search() - .with_semantic_matching() - .across_devices(&[laptop, phone, nas]) - .execute() - .await?; -``` - -#### 6. Single Identity System - -Replaced the Node/Device/Instance confusion: - -```rust -pub struct Device { - pub id: DeviceId, // One ID per installation - pub name: String, // User-friendly name - pub identity: Identity, // P2P identity - pub libraries: Vec, // What libraries it has -} -``` - -One concept, one implementation, no confusion. - -### Performance Achievements (V2) - -| Metric | Performance | -| --------------- | ------------------ | -| Indexing Speed | 8,500 files/second | -| Search Latency | ~55ms (1M entries) | -| Memory Usage | ~150MB (1M files) | -| P2P Transfer | 110 MB/s (gigabit) | -| Connection Time | 1.8 seconds | - -## The AI-Augmented Development Revolution - -### From Team Chaos to Solo Excellence - -The most remarkable aspect of Spacedrive V2 isn't just the technical improvements—it's how it was built. The contrast between V1 and V2 development tells a story of a fundamental shift in how software can be created. - -**Spacedrive V1 (2022-2025)**: - -- **Team Size**: 12 developers at peak -- **Development Time**: 3 years -- **Investment**: $2 million USD -- **Result**: Architectural failures, incomplete roadmap, development pause -- **Core Issues**: Poor coordination, slow iteration, mounting technical debt - -**Spacedrive V2 (2025)**: - -- **Team Size**: 1 developer + AI assistants -- **Development Time**: 3 weeks -- **Investment**: AI credits and personal time -- **Result**: Production-ready system, comprehensive whitepaper, clear architecture -- **Achievement**: 100x development speed increase - -### The New Development Stack - -The V2 rewrite leveraged a revolutionary development approach: - -``` -Developer (Architect/Orchestrator) - ├── ChatGPT → Deep research and citations - ├── Claude Code → Implementation and code generation - ├── Gemini → Large context analysis and system design - └── 50+ Design Documents → Persistent knowledge base -``` - -This wasn't simply using AI as a coding assistant. It was a complete reimagining of the development process: - -1. **ChatGPT for Research**: Comprehensive analysis of distributed systems, file management approaches, and technical solutions -2. **Claude Code for Implementation**: Rapid prototyping and production-ready code generation -3. **Gemini for Architecture**: Large context window analysis of the entire codebase and design documents -4. **Agentic Development**: Multiple AI agents working on different system components simultaneously - -### The Power of Focus - -Where V1 suffered from "too many cooks in the kitchen," V2 benefited from singular vision: - -- **No Communication Overhead**: Zero time spent in meetings, standups, or coordination -- **Consistent Architecture**: One mind ensuring all components align perfectly -- **Rapid Iteration**: Ideas implemented and tested within hours, not weeks -- **No Politics**: Technical decisions based purely on merit, not compromise - -### AI as Force Multiplier - -The solo developer didn't work alone—they commanded an army of specialized AI assistants: - -> "I wrote this workflow in two days using ChatGPT for deep research and citations, Claude Code to implement changes and Gemini for the large context window to analyze. This turns three years of work by 16 developers with many architectural flaws into a production ready system, fully tested and a detailed whitepaper in under a month. I'm doing this solo." - -Each AI tool was used for its strengths: - -- **Research**: AI analyzed thousands of papers and codebases -- **Implementation**: AI generated boilerplate and complex algorithms -- **Analysis**: AI reviewed architecture for consistency and flaws -- **Documentation**: AI helped create comprehensive technical docs - -### The New Capital Efficiency Model - -This development approach fundamentally changes the economics of startups: - -**Traditional Model**: - -- Raise $2M → Hire 10 developers → Burn $200k/month → Hope for product-market fit - -**AI-Augmented Model**: - -- Raise $500k → Stay solo + AI → Burn $20k/month → Achieve more with 10x runway - -The capital can instead be invested in: - -- Infrastructure and cloud services -- Security audits and compliance -- AI credits for enhanced development -- Marketing and community building -- Legal and operational services - -### Future Team Philosophy - -The V2 success doesn't mean staying solo forever, but it establishes a new hiring philosophy: - -> "Plans to move forward with an automation heavy development cycle leaves future capital and revenue for security audits, compliance, legal and infra costs. As the project grows we will seek only the best humans, keeping the team as small as possible." - -**Key Principles**: - -1. **Hire for Impact**: Each person must provide 10x value -2. **Automate First**: Only hire when automation isn't possible -3. **Quality Over Quantity**: One excellent engineer > five average ones -4. **Strategic Roles**: First hires for growth, not more developers - -### Validation from AI Partners - -Even the AI systems recognized the achievement. Gemini's analysis: - -> "What you've described is a powerful demonstration of a new paradigm for highly effective development. You haven't just used AI as a simple assistant; you've acted as an architect and orchestrator, leveraging a suite of specialized tools for their core strengths... This entire endeavor is not just about building Spacedrive; it's a case study in how a single, focused individual can now achieve what was previously only possible for large, well-funded teams." - -### Implications for the Industry - -The Spacedrive V2 development story represents a paradigm shift: - -1. **The End of Large Early-Stage Teams**: Why hire 10 developers when 1 + AI is more effective? -2. **Capital Efficiency Revolution**: Startups can achieve more with 90% less capital -3. **Quality Through Focus**: Better architecture through singular vision -4. **Speed Through Automation**: Months compressed into weeks - -This isn't just about building software faster—it's about building it better. The V2 architecture is cleaner, more thoughtful, and more maintainable than V1 precisely because it avoided the compromises and communication overhead of a large team. - -### The Investment Thesis - -This new development model creates a compelling narrative for investors: - -- **Proven Execution**: V2 built in 3 weeks vs V1's 3 years -- **Capital Efficiency**: Every dollar goes to growth, not salaries -- **Reduced Risk**: No team drama, no coordination failures -- **Scalable Model**: AI assistants scale infinitely without HR issues - -As the founder noted: - -> "I'm not planning on building a team with the capital, I think the story of flying solo until revenue is decent is a more appealing sell for seed investors. I've proved how much can be done in such a short time, why hire?" - -## Impact and Legacy - -### Technical Contributions - -Spacedrive's development spawned several open-source projects: - -- **Prisma Rust Client** (now officially supported) -- **rspc** (type-safe RPC framework) -- **Specta** (TypeScript-Rust type sharing) - -### Cultural Impact - -The project demonstrated that: - -1. Consumer software can implement enterprise-grade distributed systems -2. Local-first architecture doesn't sacrifice convenience -3. Open-source projects can attract top-tier venture funding -4. Community-driven development produces innovative solutions - -### Industry Influence - -Spacedrive proved several concepts: - -- VDFS is viable for consumer applications -- Content-addressable storage works at personal scale -- P2P can achieve reliability comparable to cloud services -- Privacy and functionality aren't mutually exclusive - -## Lessons Learned from the Failure - -### 1. Architecture Must Match User Needs - -**The Mistake**: Building two separate file systems because of implementation details -**The Lesson**: User experience must drive architecture, not the other way around - -Users don't care about "indexed" vs "ephemeral" files—they just want to copy their vacation photos. The dual file system was an implementation detail that leaked into the user experience, making basic operations impossible. - -### 2. Start Simple, Iterate Often - -**The Mistake**: Building a perfect CRDT sync system that never shipped -**The Lesson**: Ship basic sync first, enhance later - -The team spent years debating the perfect sync architecture while competitors shipped simpler solutions. A basic "last write wins" sync would have provided 90% of the value with 10% of the complexity. - -### 3. Don't Create Dependencies You Can't Maintain - -**The Mistake**: Creating prisma-client-rust and rspc, then abandoning them -**The Lesson**: Use existing solutions unless you're committed to maintaining new ones - -Creating fundamental infrastructure is a massive commitment. When the team's needs changed, they couldn't maintain these libraries, leaving Spacedrive stranded on deprecated forks. - -### 4. Reduce Boilerplate Ruthlessly - -**The Mistake**: 1000+ lines to add a simple file operation -**The Lesson**: Developer experience directly impacts feature velocity - -When adding a "delete file" operation requires days of boilerplate, innovation stops. The V2 approach with procedural macros shows how the same functionality can be achieved in 50 lines. - -### 5. Core Features Must Be Excellent - -**The Mistake**: Marketing "lightning fast search" while delivering basic SQL queries -**The Lesson**: Don't promise what you can't deliver - -Search was a core value proposition of the VDFS concept, yet it remained neglected. If search is your differentiator, it must be world-class from day one. - -### 6. One Concept, One Implementation - -**The Mistake**: Node vs Device vs Instance representing the same thing -**The Lesson**: Conceptual clarity prevents implementation confusion - -When the same concept has multiple representations, bugs multiply. Every developer has to understand the mapping between systems, and inconsistencies creep in. - -### 7. Complete Migrations Before Starting New Ones - -**The Mistake**: Running old and new systems in parallel throughout the codebase -**The Lesson**: Technical debt compounds exponentially - -The codebase had `old_job`, `old_p2p`, and `old_*` files still in active use. Each incomplete migration made the next one harder, creating a maze of deprecated-but-necessary code. - -### 8. Event-Driven > Direct Coupling - -**The Mistake**: Backend hardcoding frontend cache keys -**The Lesson**: Loose coupling enables independent evolution - -The `invalidate_query` pattern meant changing the frontend required backend changes. Event-driven architecture allows each layer to evolve independently. - -### 9. Perfect is the Enemy of Good - -**The Mistake**: Analysis paralysis on sync boundaries -**The Lesson**: Make decisions and move forward - -The team couldn't decide what should sync vs remain local, so nothing shipped. A clear decision—even if imperfect—would have been better than no decision. - -### 10. Community Momentum is Precious - -**The Mistake**: Losing momentum after initial excitement -**The Lesson**: Consistent delivery maintains community engagement - -Spacedrive had incredible initial traction—35k stars, 500k installs—but development stalled. Regular releases, even small ones, keep the community engaged and attract contributors. - -## Future Vision - -### Near-Term Goals - -The V2 architecture enables: - -- Complex AI workflows for automatic organization -- Intelligent content analysis pipelines -- Semantic search across all data types -- Federated learning from usage patterns - -### Long-Term Ambitions - -Spacedrive aims to become: - -- A platform for personal AI agents -- The foundation for local-first computing -- A bridge between personal and collaborative workflows -- The default file management paradigm - -### Business Model Evolution - -Plans include: - -- **Open Core**: Free forever for individuals -- **Team Features**: Collaboration tools for small groups -- **Enterprise**: Advanced security and compliance -- **Cloud Services**: Optional convenience features -- **Developer Platform**: APIs for third-party integration - -## Conclusion - -From a developer's personal frustration to a venture-backed phenomenon with 35,000 GitHub stars and 500,000 installations, Spacedrive's journey exemplifies both the challenges and opportunities in modern software development. The project's evolution tells three distinct stories: - -**The Promise** (2021-2022): A vision that resonated globally—unifying all files under user control with intelligent, distributed systems. - -**The Struggle** (2022-2025): How even well-funded projects with talented teams can fail due to architectural mistakes, over-engineering, and decision paralysis. The dual file system, abandoned dependencies, and sync system that never shipped serve as cautionary tales for ambitious projects. - -**The Revolution** (2025): A single developer with AI assistance achieving in 3 weeks what 16 developers couldn't in 3 years. This isn't just a comeback story—it's a glimpse into the future of software development. - -The V2 reimagining proves that the original vision was sound; only the execution was flawed. By addressing every architectural mistake, simplifying ruthlessly, and leveraging AI as a force multiplier, Spacedrive has been reborn stronger than ever. The new development paradigm—one architect orchestrating specialized AI agents—demonstrates that we've entered an era where individual developers can build systems previously requiring entire teams. - -Most importantly, Spacedrive's journey from failure to rebirth offers invaluable lessons: Start simple. Ship often. Avoid over-engineering. Maintain conceptual clarity. And now, in 2025: leverage AI not as a tool, but as a team. - -Whether Spacedrive becomes the default file manager of the future or serves as inspiration for others, its impact is undeniable. It has shown that the dream of a unified, intelligent, user-controlled filesystem is not only possible—with the right approach, it's inevitable. - ---- - -_"Files shouldn't be stuck in a device ecosystem. Open source technology is the only way to ensure we retain absolute control over the files that define our lives."_ - Jamie Pine, Founder of Spacedrive diff --git a/docs/philosophy.md b/docs/philosophy.md deleted file mode 100644 index 2a1fa2eea..000000000 --- a/docs/philosophy.md +++ /dev/null @@ -1,61 +0,0 @@ -# The Spacedrive Philosophy - -Spacedrive is more than a file manager; it is a paradigm shift in how humans interact with their digital assets. It was born from the universal frustration of "data fragmentation hell"—a state where our digital lives are scattered across countless devices and incompatible cloud services. - -Our mission is to create a unified, content-aware ecosystem that gives users complete control over their data. The V2 architecture is the realization of this vision, built on a set of core principles that guide every design decision, line of code, and future feature. - ---- - -## The Core Tenets - -### 1. The User is Sovereign - -This is our most fundamental principle. In the Spacedrive ecosystem, you are in complete control. - -- **Local-First by Default**: Spacedrive is designed to work entirely offline. All indexing, analysis, and processing happen on your devices, ensuring your data and your privacy remain yours alone. -- **You Own Your Data, Always**: We built Spacedrive as an intelligent layer that sits _on top_ of your existing storage. Your files stay where they are, in their original locations, with zero vendor lock-in. Backing up your entire organizational system is as simple as copying a single directory. -- **Human in the Loop**: The user is the final authority. AI agents and automation systems are designed to _propose_ actions, which are then presented in a clear, previewable format for you to approve. You are always in command. - -### 2. From Chaos to Cohesion - -We aim to solve the universal problem of file chaos by creating a single, unified view of your entire digital world. - -- **A Unified Virtual Layer**: The Virtual Distributed File System (VDFS) creates one interface to manage files across all devices and clouds. It makes scattered storage feel like a single, cohesive library. -- **Content is King**: We move beyond rigid, location-based folder hierarchies to a content-aware model. Through Content-Addressed Storage (CAS), Spacedrive understands what a file _is_, not just where it is, enabling powerful features like global deduplication and data integrity verification. -- **Location Transparency**: Our universal addressing system, `SdPath`, makes device boundaries disappear. A file on your offline laptop is as accessible as one on your local NAS, as the system can intelligently find and use any available copy. - -### 3. Intelligence as Augmentation - -AI is not a bolted-on feature; it is a foundational element of the architecture, designed to enhance your capabilities without compromising your control. - -- **The AI Data Guardian**: Spacedrive's AI acts as a proactive protector of your data. By tracking file redundancy, it can identify irreplaceable memories that exist in only one location and suggest creating a backup before disaster strikes. -- **Natural Language as a Command Line**: You can manage your files by simply stating your intent. The system translates commands like "find my design assets from last fall" into safe, verifiable, and previewable actions. -- **Privacy-Preserving Intelligence**: We believe you shouldn't have to trade privacy for intelligence. Spacedrive is built to run powerful AI models locally on your hardware via tools like Ollama, ensuring your file contents are never sent to the cloud unless you explicitly choose to. - -### 4. Pragmatism in Engineering - -The failure of Spacedrive V1 taught us a critical lesson: perfect is the enemy of good. The V2 architecture embodies a pragmatic approach focused on delivering value and reliability. - -- **Simplicity over Complexity**: We replace over-engineered solutions with simpler, more robust patterns. V2's domain-separated sync avoids the "analysis paralysis" of a custom CRDT implementation, allowing us to ship a reliable sync system. -- **Developer Experience Matters**: We ruthlessly reduce boilerplate. The V2 job system, for example, cuts the code needed to add a new background operation by over 90%, enabling us to build and iterate faster. -- **Power for Everyone**: We engineer enterprise-grade capabilities to run efficiently on consumer hardware. Sophisticated features like semantic search, cross-device deduplication, and transactional operations are made accessible to everyone, not just large organizations. - -### 5. Open by Default - -Trust is earned through transparency. Our commitment to open source is a core part of our identity. - -- **Open Source for Control**: Spacedrive is open source to guarantee that you always retain absolute control over the software that manages your most important data. -- **Community as a Partner**: The project's success is tied to our community. The V2 whitepaper and architecture are a definitive technical blueprint, inviting developers to review, contribute, and help build the future of file management with us. -- **A Sustainable Vision**: We use a sustainable Open Core model. The core product is free for individuals, with paid features for teams and enterprises that ensure the project's long-term health and development. - -### 6. A New Way to Build - -The story of V2's creation is a meta-philosophy in itself. It demonstrates a revolutionary new paradigm for software development. - -- **The AI-Augmented Team**: Spacedrive V2 was rebuilt from the ground up by a single developer orchestrating a suite of specialized AI assistants. This approach proved to be 100x faster and more effective than a traditional team. -- **Radical Capital Efficiency**: This new development model changes the economics of building software. It allows capital to be invested in growth, security, and infrastructure instead of being consumed by large team salaries. -- **Elite, Focused Teams**: Our hiring philosophy is to automate first and hire only the best humans for roles that require strategic impact. We believe small, focused, high-impact teams build better products. - ---- - -These principles are woven into every aspect of Spacedrive. They are the reason we believe we can solve the fundamental problems of data fragmentation, privacy, and intelligent management for the AI era. diff --git a/docs/sdk.md b/docs/sdk.md deleted file mode 100644 index 7f99e809e..000000000 --- a/docs/sdk.md +++ /dev/null @@ -1,567 +0,0 @@ -**Date:** October 10, 2025 -**Status:** Finalized for Implementation (Based on Core v2.0 Roadmap) -**Overview:** This is the master specification for the Virtual Distributed File System (VDFS) Extensions SDK. The SDK enables developers to build domain-specific extensions that leverage Spacedrive's local-first, AI-native architecture without bloating the core. - -## Guiding Principles - -The SDK follows these core tenets, grounded in Spacedrive's whitepaper and 87% complete implementation: - -- **Declarative and Type-Safe:** Use Rust attributes to describe intent; the compiler ensures safety. -- **Core Primitives, Extension Experiences:** Core handles generic operations (e.g., indexing, sync, basic extraction like EXIF/OCR). Extensions add specialized behavior (e.g., face detection in Photos). -- **Asynchronous and Progressive:** Operations are job-based and on-demand; results improve as data is analyzed, without UI blocks. -- **User-Controlled Scoping:** Extensions request permissions; users grant and scope them to specific locations/paths for privacy. -- **Durability and Safety:** All mutations use transactional actions (preview-commit-verify, 100% complete). Jobs are resumable (100% complete). -- **Model-Agnostic AI:** Core provides loaders; extensions register/use models (local, API, custom). -- **WASM Sandboxing:** Extensions run in WASM for isolation, with host functions for core access (60% complete API). - -Extensions are plugins installable via the Extension Store. They adapt Spacedrive (e.g., Photos adds media features; CRM adds contact views) without core changes. - -## Core vs. Extension Responsibilities - -### Core Provides: - -**Data Infrastructure:** - -- `entries` table - Files and directories (device-owned sync) -- `content_identities` table - Unique content (shared via deterministic UUID) -- `user_metadata` table - Tags, notes, custom_data (dual-scoped: entry OR content) -- `models` table - Extension models (NEW - required for extensions) -- `tags`, `collections` - Universal organization primitives - -**Processing:** - -- Generic extraction: EXIF from images, OCR from docs, thumbnails (70% complete) -- Indexing pipeline: 5 phases, resumable (90% complete) -- Sync: HLC timestamps, CRDTs (95% complete) -- Jobs/Actions: Durable, previewable (100% complete) - -**AI Infrastructure:** - -- Model loaders: Local (Ollama), API (OpenAI), Custom (ONNX) -- Model registry: Categories (ocr, llm, face_detection, etc.) -- Jinja template rendering - -### Extensions Provide: - -**Domain Models:** - -- Content-scoped: PhotoAnalysis (attached to photos), VideoAnalysis -- Standalone: Person, Album, Place, Moment, Contact, Email, Note -- Stored in: `models` table with extension_id + model_type - -**Specialized Analysis:** - -- On-demand jobs (user-initiated, scoped to locations) -- Custom AI models (face detection, receipt parsing, citation extraction) -- Model → Tag generation (detailed models.data → searchable tags) - -**User Experience:** - -- UI via `ui_manifest.json` (sidebar, menus, views) -- Custom queries and actions -- Agent memories and reasoning - -**Key Principle:** - -- Core does generic, always-useful work (EXIF, thumbnails, basic OCR) -- Extensions do specialized work on user-scoped locations -- Both use same primitives (tags, collections, sync) - -## 1. Extension Definition (`#[extension]`) - -The entry point for your extension. Defines metadata, dependencies, and permissions. - -**Syntax:** - -```rust -#[extension( - id = "com.spacedrive.photos", // Unique reverse-DNS ID - name = "Photos Manager", - version = "1.0.0", - description = "Advanced photo organization with faces and places.", - min_core_version = "2.0.0", // Minimum Spacedrive core version - required_features = ["ai_models", "exif_extraction"], // Core features needed - permissions = [ // Requested permissions (user grants/scopes) - Permission::ReadEntries(glob = "**/*.{jpg,png,heic}"), - Permission::ReadSidecars(kinds = ["exif"]), - Permission::WriteSidecars(kinds = ["faces", "places"]), - Permission::WriteTags, - Permission::UseModel(category = "face_detection", preference = "local"), - ] -)] -struct Photos { - config: PhotosConfig, // User-configurable settings -} -``` - -**Behavior:** - -- On install: Core validates `min_core_version` and `required_features` via `PluginManager`. -- Permissions: Requested here; user scopes during setup (e.g., limit to "/My Photos"). Core enforces on every call (e.g., `ReadEntries` fails outside scope). -- Config: Generates UI settings pane. Example: - -```rust -#[derive(Serialize, Deserialize)] -struct PhotosConfig { - #[setting(label = "Enable Face Recognition", default = true)] - face_recognition: bool, - #[setting(label = "Model Preference", default = "local")] - model_selector: String, -} -``` - -- Installation Flow: User installs from Store; core loads WASM, registers models/jobs, prompts for scopes. - -## 2. Extension Models (`#[model]`) - -Extensions create models stored in the `models` table. Three scoping strategies: - -### Content-Scoped Models (Attach to Photos/Videos) - -```rust -/// Attached to content_identity (device-independent) -/// Same photo at different paths → one PhotoAnalysis -#[model(version = "1.0.0")] -#[scope = "content"] // Scoped to content_identity -#[sync_strategy = "shared"] -struct PhotoAnalysis { - id: Uuid, - detected_faces: Vec, - scene_tags: Vec, - identified_people: Vec, // References Person models -} -``` - -**Storage:** - -```sql --- models table -{ - uuid: , - extension_id: "photos", - model_type: "PhotoAnalysis", - content_identity_uuid: , -- Scoped to content! - data: JSON, - metadata_id: → user_metadata -- Can be tagged -} -``` - -**Key Benefit:** Survives device changes. If you remove a device, entries disappear but ContentIdentity (and PhotoAnalysis) remain. - -### Standalone Models (Independent Entities) - -```rust -/// Not tied to any file/content -#[model(version = "1.0.0")] -#[scope = "standalone"] -#[sync_strategy = "shared"] -struct Person { - id: Uuid, - name: Option, - embeddings: Vec>, - photo_count: usize, -} - -struct Album { - name: String, - content_ids: Vec, // References content_identity UUIDs! -} -``` - -**Storage:** - -```sql --- models table -{ - uuid: , - extension_id: "photos", - model_type: "Person", - standalone: 1, -- Not scoped to entry/content - data: JSON, - metadata_id: → user_metadata -- Can be tagged with #family -} -``` - -### Entry-Scoped Models (Rare, Device-Specific) - -```rust -#[model] -#[scope = "entry"] // Tied to specific path -#[sync_strategy = "device_owned"] -struct LocalEditState { - processing_state: String, -} -``` - -### Large Data: Blob Storage - -For large data (embeddings, cached results), use `#[blob_data]` to avoid bloating queries: - -```rust -#[model] -struct Person { - id: Uuid, - name: Option, // Inline in models.data (fast queries) - photo_count: usize, // Inline - - #[blob_data(compression = "zstd", lazy = true)] - embeddings: Vec>, // Stored in metadata_blobs table -} -``` - -**Storage:** - -```sql --- Lightweight data (fast queries) -models.data = '{"name":"Alice","photo_count":42}' - --- Heavy data (separate, content-addressed) -metadata_blobs { blob_hash: "abc123", blob_data: , size_bytes: 51200 } -model_blobs { model_uuid: person_uuid, blob_key: "embeddings", blob_id: 1 } -``` - -**Benefits:** - -- Fast queries (heavy data not loaded) -- Lazy loading (load blobs only when accessed) -- Deduplication (content-addressed by hash) -- Compression (zstd reduces embeddings 4x) - -**API Methods:** - -```rust -// Create content-scoped model -ctx.vdfs().create_model_for_content(content_uuid, photo_analysis).await?; - -// Get content-scoped model (lightweight fields only) -let analysis = ctx.vdfs().get_model_by_content::(content_uuid).await?; - -// Access blob field (triggers lazy load) -let faces = analysis.detected_faces().await?; // ← Loads from metadata_blobs - -// Create standalone model -ctx.vdfs().create_model(person).await?; - -// Query without loading blobs (fast!) -let people = ctx.vdfs() - .query_models::() - .select_inline_only() // Don't load embeddings - .collect() - .await?; - -// Tag content (all entries pointing to this content get the tag) -ctx.vdfs().add_tag_to_content(content_uuid, "#vacation").await?; - -// Tag model -ctx.vdfs().add_tag_to_model(person_uuid, "#family").await?; -``` - -**Behavior:** - -- **Tags:** All models have `metadata_id` → participate in tag system -- **Collections:** All models can be in collections (polymorphic reference) -- **Sync:** Content-scoped and standalone use shared sync (HLC). Entry-scoped uses device-owned. -- **Device Independence:** PhotoAnalysis attached to content survives device removal -- **Performance:** Blob separation keeps queries fast even with large data - -## 3. Jobs and Tasks (`#[job]`, `#[task]`) - -Durable units of work. Extensions define for on-demand analysis. - -**Syntax:** - -```rust -#[task(retries = 3, timeout_ms = 30000)] -async fn detect_faces(ctx: &TaskContext, entry: &Entry) -> TaskResult> { - let image_bytes = entry.read().await?; - let faces = ctx.ai() - .from_registered("face_detection:photos_v1") - .detect_faces(&image_bytes) - .await?; - Ok(faces) -} - -#[job(parallelism = 4, trigger = "user_initiated")] -async fn analyze_photos(ctx: &JobContext, location: SdPath) -> JobResult<()> { - // Get content UUIDs (not entry UUIDs!) - let content_uuids = ctx.vdfs() - .query_entries() - .in_location(location) - .of_type::() - .map(|e| e.content_uuid()) - .collect() - .await?; - - for content_uuid in content_uuids { - // Check if already analyzed - if ctx.vdfs().get_model_by_content::(content_uuid).await.is_ok() { - continue; - } - - // Get an entry to read image data - let entry = ctx.vdfs() - .query_entries() - .where_content_id(content_uuid) - .on_this_device() - .first() - .await?; - - // Analyze - let faces = ctx.run(detect_faces, (&entry,)).await?; - let scenes = ctx.run(classify_scene, (&entry,)).await?; - - // Create content-scoped model - let analysis = PhotoAnalysis { faces, scenes, ... }; - ctx.vdfs().create_model_for_content(content_uuid, analysis).await?; - - // Tag the content - ctx.vdfs().add_tag_to_content(content_uuid, "#analyzed").await?; - } - - Ok(()) -} -``` - -**Behavior:** - -- **Triggers:** "user_initiated", "on_event", etc. Scoped to user-granted locations -- **Persistence:** Shared `jobs.db` with extension_id (unified monitoring) -- **Checkpoints:** Auto-saved; resumable (100% core) -- **Content-Awareness:** Works with content_identity (device-independent) -- **Model → Tags:** Create PhotoAnalysis model, then generate tags for search - -## 4. AI Agents and Memory (`#[agent]`, `#[agent_memory]`) - -Autonomous logic for extensions. - -**Syntax:** - -```rust -#[agent_memory] // Defines the "mind" -struct PhotosMind { - #[sync(shared)] // Syncs across devices - knowledge: AssociativeMemory, // e.g., person relationships - - #[sync(device_owned)] // Local only - plan: WorkingMemory, -} - -#[agent] // The agent itself -#[agent_trail(level = "debug", format = "jsonl")] // Debug logs only -impl Photos { - async fn on_new_photo(&self, ctx: &AgentContext, photo: Photo) -> AgentResult<()> { - ctx.trace("New photo added - checking for faces"); // To trail (debug) - - // Use memory for reasoning - let mind = ctx.memory(); - let similar_faces = mind.knowledge.query_similar_faces(photo.exif).await; - - // Dispatch job if needed - if similar_faces.is_empty() { - ctx.dispatch_job(analyze_photos, photo.location()).await?; - } - - // Mutation to VDFS (audit log) - ctx.vdfs().add_tag(photo.id, "#new-photo").await?; - - Ok(()) - } -} -``` - -**Behavior:** - -- **Agent Loop:** Observe (via event hooks, e.g., "on_new_photo"), Orient (query memory), Act (dispatch jobs/actions). -- **Memory:** Extension-defined. Temporal: Time-based events; Associative: Graphs/vectors; Working: Short-term state. Backends: SQLite for temporal, VSS for associative. -- **Trail:** Debug only (e.g., "Decision: Skipping analysis"—stored in logs/extension/). Not for cognition. -- **Context Building:** Extensions define prompts/templates (Jinja) for model inputs. -- **Sync:** Selective per-field; heavy data (e.g., vectors) optional. - -## 5. Model Registration and AI Integration - -**Syntax:** - -```rust -// On extension install/init -fn init(ctx: &ExtensionContext) { - ctx.models().register( - name = "face_detection", - category = "vision", - source = ModelSource::Download { url: "https://example.com/model.onnx", sha256: "abc123" }, - ).await?; -} - -// In jobs/tasks -ctx.ai().from_registered("face_detection").generate(...).await?; -``` - -**Behavior:** - -- **Registration:** On install; core downloads/stores in `~/.spacedrive/models/` (root, no sync). -- **Sources:** Bundled bytes, download URL, or local path. -- **Loaders:** Core handles (Ollama local, API with consent UI). -- **Preferences:** User sets (local/cloud); extensions respect. -- **Prompts:** Jinja templates in `prompts/` for separation. - -## 6. Actions (`#[action]`) - -User-invokable operations with preview. - -**Syntax:** - -```rust -#[action] -async fn organize_photos(ctx: &ActionContext, location: SdPath) -> ActionResult { - // Simulate: Query tags/faces - let changes = ...; // Preview moves/tags - Ok(ActionPreview { title: "Organize Photos", changes, reversible: true }) -} - -#[action_execute] -async fn organize_photos_execute(ctx: &ActionContext, preview: ActionPreview) -> ActionResult<()> { - // Apply: Use vdfs.add_tag(), etc. (audited) - Ok(()) -} -``` - -**Behavior:** Preview-commit-verify (100% core). Scoped to user permissions. - -## 7. UI Integration (`ui_manifest.json`) - -**Syntax (JSON in extension package):** - -```json -{ - "sidebar_sections": [ - { - "id": "people", - "label": "People", - "icon": "assets/people_icon.png", - "query": "tags LIKE '#person:%'", // VDFS query for data - "render_type": "list" // Generic: list, grid, etc. - } - ], - "views": [ - { - "id": "places_map", - "label": "Places", - "component": "map_view", // Core-provided components - "data_source": "query:exif_gps" // Fetch via VDFS - } - ] -} -``` - -**Behavior:** Frontend parses; renders generically. Extensions bundle assets (icons/CSS). No Rust UI code. - -## 8. Fluent Builders (Device/AI Orchestration) - -**Syntax:** - -```rust -let device = ctx.select_device() - .with_capability("gpu") - .prefer_local() - .select() - .await?; -ctx.execute_on(device, || async { /* heavy compute */ }).await?; -``` - -**Behavior:** Leverages core networking (85% complete). Scoped to permissions. - -## Required Core Schema Changes - -Extensions require these new tables in Core: - -```sql --- 1. Extension models (lightweight data) -CREATE TABLE models ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - uuid BLOB UNIQUE NOT NULL, - extension_id TEXT NOT NULL, - model_type TEXT NOT NULL, - data TEXT NOT NULL, -- Lightweight JSON only - - -- Scoping (exactly one set) - entry_uuid BLOB REFERENCES entries(uuid), - content_identity_uuid BLOB REFERENCES content_identities(uuid), - standalone BOOLEAN DEFAULT 0, - - -- Metadata (for tags/collections) - metadata_id INTEGER NOT NULL REFERENCES user_metadata(id), - - -- Sync - sync_strategy INTEGER NOT NULL, -- 0=DeviceOwned, 1=Shared - hlc_timestamp TEXT, - device_uuid BLOB, - - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL, - - CHECK ( - (entry_uuid IS NOT NULL AND content_identity_uuid IS NULL AND standalone = 0) OR - (entry_uuid IS NULL AND content_identity_uuid IS NOT NULL AND standalone = 0) OR - (entry_uuid IS NULL AND content_identity_uuid IS NULL AND standalone = 1) - ) -); - --- 2. Large data storage (content-addressed, deduplicated) -CREATE TABLE metadata_blobs ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - blob_hash TEXT UNIQUE NOT NULL, -- BLAKE3 hash for deduplication - blob_data BLOB NOT NULL, -- Compressed binary data - blob_type TEXT NOT NULL, - size_bytes INTEGER NOT NULL, - compression TEXT, -- "zstd", "gzip", NULL - reference_count INTEGER DEFAULT 1, - created_at TEXT NOT NULL, - last_accessed_at TEXT NOT NULL -); - --- 3. Links models to large blobs -CREATE TABLE model_blobs ( - model_uuid BLOB NOT NULL REFERENCES models(uuid), - blob_key TEXT NOT NULL, -- Field name ("embeddings", "cache") - blob_id INTEGER NOT NULL REFERENCES metadata_blobs(id), - PRIMARY KEY (model_uuid, blob_key) -); - --- 4. Links user_metadata to large blobs (for entries/content) -CREATE TABLE user_metadata_blobs ( - metadata_id INTEGER NOT NULL REFERENCES user_metadata(id), - blob_key TEXT NOT NULL, - blob_id INTEGER NOT NULL REFERENCES metadata_blobs(id), - PRIMARY KEY (metadata_id, blob_key) -); - --- 5. Extend collections to support models -ALTER TABLE collection_items ADD COLUMN model_uuid BLOB REFERENCES models(uuid); -ALTER TABLE collection_items ADD CONSTRAINT check_item_type - CHECK (entry_uuid IS NOT NULL OR content_uuid IS NOT NULL OR model_uuid IS NOT NULL); - --- Indexes -CREATE INDEX idx_models_uuid ON models(uuid); -CREATE INDEX idx_models_extension ON models(extension_id, model_type); -CREATE INDEX idx_models_content ON models(content_identity_uuid); -CREATE INDEX idx_models_metadata ON models(metadata_id); -CREATE UNIQUE INDEX idx_metadata_blobs_hash ON metadata_blobs(blob_hash); -CREATE INDEX idx_model_blobs_model ON model_blobs(model_uuid); -``` - -**UserMetadata already supports content-scoping** (existing): - -```sql -user_metadata { - entry_uuid: Option, -- Entry-specific tag - content_identity_uuid: Option, -- Content-universal tag (all copies) -} -``` - -## Implementation Notes - -- **WASM Hosts:** Core provides functions: `vdfs_query_entries()`, `model_create()`, `model_query()`, `add_tag_to_content()` -- **Validation:** Use existing tests (1,554 LOC sync tests) + extension scenarios -- **Migration:** Add `models` table in next schema migration (required for extensions) -- **Roadmap:** Builds to November 2025 alpha (core completion in 3-4 months) - -This spec realizes Spacedrive's vision: A lean core with infinite extensibility. The Photos extension demonstrates the complete architecture. diff --git a/docs/whitepaper.md b/docs/whitepaper.md deleted file mode 100644 index a50360f72..000000000 --- a/docs/whitepaper.md +++ /dev/null @@ -1,56 +0,0 @@ -# Spacedrive V2 Whitepaper: A Guide for the Community - -This document explains the purpose of the Spacedrive V2 whitepaper, what it covers, and how our community can use it to understand, contribute to, and build upon the future of personal data management. - ---- - -## Why We Wrote This Whitepaper - -Spacedrive V2 represents a complete architectural reimagining of the project, designed to fulfill the original vision on a more robust and scalable foundation. After a period of reflection on the challenges faced by the first version, we recognized the need for a foundational rewrite to address critical issues like the "Dual File System Problem" and fragmented networking. - -This whitepaper was created to serve three primary purposes: - -1. **To Provide a Definitive Technical Blueprint**: It is the single source of truth for the Spacedrive V2 architecture, detailing the core concepts, design decisions, and innovations that power the new system. -2. **To Re-engage Our Community**: We want to share our renewed vision and technical direction transparently, providing a clear path for developers, contributors, and users to rally behind. -3. **To Guide Future Development**: The document serves as a roadmap and a set of guiding principles, ensuring that all future contributions align with the core architectural tenets of performance, privacy, and user control. - -Ultimately, this paper is our commitment to building a paradigm shift in how humans interact with their digital assets in the AI era. - ---- - -## What This Whitepaper Covers - -The whitepaper presents the complete architecture of Spacedrive V2, a local-first, AI-native Virtual Distributed File System (VDFS). It details the five foundational innovations that solve traditionally hard problems in distributed systems for a consumer-grade product. - -Key architectural pillars covered in detail include: - -- **A Virtual Distributed File System (VDFS)**: A unified, virtual layer that provides a single view of all your data across every device and cloud, while the files themselves stay in their original locations. This is made possible by a universal addressing system called **`SdPath`**. -- **An AI-Native Architecture**: The system is designed from the ground up for intelligent management, enabling natural language commands ("find my tax documents from last year") and proactive assistance from a data guardian that respects user privacy. -- **A Transactional Action System**: All file operations are treated as transactions that can be previewed before they are committed, preventing conflicts and guaranteeing completion even across offline devices. -- **Domain-Separated Library Sync**: A novel synchronization method that maintains consistency across devices without the complexity of distributed consensus algorithms like CRDTs. -- **The Content Identity System**: A content-addressable foundation that provides intelligent, cross-device deduplication while also powering a "Data Guardian" feature that monitors data redundancy to protect against loss. -- **A Modern, Performant Implementation**: The entire core is implemented in **Rust** on a modern, asynchronous technology stack designed for enterprise-grade capabilities on consumer hardware. - ---- - -## How to Use This Document - -This whitepaper is more than a technical document; it's an invitation to our community to help build the future of file management. - -#### **For Developers & Contributors:** - -- **Understand the Vision**: Before diving into the code, read the whitepaper to understand the "why" behind the architecture. It provides the context for our design choices and the problems we are solving. -- **Guide Your Contributions**: Use this document alongside the **Roadmap** as a guide. The architecture detailed here is the blueprint for all new features. Whether you're fixing a bug or building a new operation, it should align with these core principles. -- **Build with Confidence**: The paper explains the core abstractions like `SdPath`, the `ActionManager`, and the `JobManager`. Understanding these will help you build new features that integrate seamlessly and reliably with the rest of the system. - -#### **For System Architects & Designers:** - -- **Review and Feedback**: We welcome feedback on our architectural decisions. The whitepaper details our solutions to complex problems like synchronization and distributed file operations. If you have insights or see potential improvements, we encourage you to start a discussion. -- **Propose Enhancements**: The document provides the context for proposing new, large-scale features or optimizations, such as the planned migration to a Closure Table for hierarchical queries. - -#### **For the Entire Community:** - -- **The Source of Truth**: When you have questions about how Spacedrive works under the hood, this document is the canonical source. It explains how we handle your data, ensure privacy, and deliver powerful features in a local-first environment. -- **A Foundation for Discussion**: Use this paper as a common ground for discussions about Spacedrive's future. It ensures everyone is working from the same set of assumptions about the technology. - -We are incredibly excited about the foundation we've built with Spacedrive V2 and believe it positions us to solve the fundamental problems of data fragmentation and privacy for the modern era. We invite you to read, discuss, and build with us.