Files
spacedrive/xtask
Jamie Pineandcapy-ai-staging[bot] f69c1fee89 Add to Library core: defaults and overrides, store placement, volume registration, portable identity, removal keeps the catalog (#3121)
* Add to Library defaults: library settings with per-add overrides

Every entry point used to carry its own notion of what a source starts as:
volumes.track defaulted external drives to unfiltered, folder tracking to
filtered, and the Explorer sent unfiltered: false by hand. The library now
holds the defaults under settings.adding (store placement in library, an
offline copy kept, filtered capture, content identified), config.library.get
reports them and config.library.update changes them for later adds only.

sources.track and volumes.track take the same AddOverrides on top; an absent
field is the default and an override never writes back. The effective
settings are the source's SourceConfig, persisted with the registration and
returned in the output together with the saved name and the store path.
A name, settings or volume row that fails to persist now fails the add
instead of reporting a tracked source. A source added without content
identification gets its walk and nothing after. volumes.track keeps the
archival behavior by passing an explicit unfiltered override for an external
drive unless the caller set one.

The CLI gains --on-source, --in-library, --keep-offline-copy and
--no-identify on sd source track. Generated TypeScript types regenerated.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Resolve store placement per library in the source layer

Placement is intent recorded in each source's config column; the serving
device resolves it. VolumeIndex::store_dir is the one resolver: in library
under <data>/sources/<id>, on source under <root>/.spacedrive/sources/<id>,
following the source's current root so an on-source store tracks its volume
across remounts and resolves to nothing while the drive is away. Writer
opens, read-only opens, freezes, the status surface and sources.list all go
through it, and SourceStore::open takes the resolved stores directory rather
than the data-directory layout. On source is refused for network and cloud
volumes, whose WAL database cannot live on the share.

The volume index also gains register_source_with, which registers under an
identity a store on disk already carries, and forget_source, which drops a
registration and retires its writer and reader handles without touching the
directory.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Adding a path tracks its volume

ensure_volume_in_db wrote the row but skipped the sync publication and the
VolumeTracked event that track_volume emits, and the add logged its failure
and carried on. It now syncs and emits on insert, the add fails when the row
cannot be written, and the volume index maps the drive at the same moment so
the mount point check and the detached flag apply to the new source. A second
folder on the same drive finds the row already there.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Portable source identity and managed-directory exclusions

Every store now carries source.json beside data.db: a versioned descriptor
with the source id, the library that wrote it, its volume anchor and path
within the volume, and its settings. Adding a scope first looks for a store
whose descriptor binds to this library, this volume and this path, and
registers under that identity, so a removed folder re-added or a drive
plugged in with its catalog on it reopens the catalog instead of minting a
fresh source over an empty one. A descriptor from another library or another
drive is ignored and a new source starts beside it.

The own-data-directory check becomes crate::config::is_managed: the data
directory under every spelling, any .spacedrive directory and what is under
it, and the volume identity marker. It is a path-component check rather than
a registry lookup, so it holds for an unfiltered source whose rules are off,
for an enclosing source that was never told about the inner store, for a
store on a drive nobody here tracks, and across remounts and aliases. The
walk, the watcher and the writer all use it.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Removing a source keeps its catalog unless asked to delete it

sources.delete used to delete the store directory through the adapter engine
before unregistering, which would have turned removal of an on-source store
into destruction of the portable catalog, assertions included. It now
retires the source's handles, drops the registration and leaves the store
where it is; delete_catalog is an explicit second choice. The volume row is
never touched: removing the last folder on a drive does not forget the
hardware. sd source untrack wraps the action.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Add to Library acceptance suite and status

One Core over temporary directories tracks a folder with defaults and one
with overrides, proves one volume row for both, finds the on-source store
excluded from its own and an enclosing source, removes a source and sees
its catalog and record identities survive the re-add, and sees deletion
start over. The suite joins the acceptance group of xtask test-core;
docs/core/acceptance/add-to-library.md maps the plan's cases to tests and
names the rows the core half cannot prove. PROJECT_STATUS and the plan's
status line record what landed and what the modal thread builds next.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Review fixes: placement-aware reads and backup, re-track keeps settings, adoption across placements

The library's per-source read queries (sources.list_items, list_records,
media_listing) opened the store through the engine's in-library layout, so
an on-source catalog was unreachable from the explorer; they now resolve a
registered filesystem source through the volume index and fall through to
the engine only for adapter sources. libraries.backup reads each store from
where placement put it. A re-track carries the previous value of every
setting the add did not name instead of the library's current defaults,
and never changes placement. portable_identity searches both placements so
a drive arriving with its catalog is adopted whatever placement the add
asked for, and the adopted catalog keeps the placement it was found under.
SourceConfig::default agrees with its serde defaults.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Restore names an on-source catalog it cannot put back

The archive now holds a store backed up from its drive, but a restore lays
every store in the in-library layout, where an on-source registration does
not read. The manifest records each source's placement; a restore leaves an
on-source catalog where it landed, keeps it out of the restored count, and
names it in on_source_catalogs with its path, so the caller knows what was
not put back. Moving it onto the drive is relocation.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* catalog_reused means an identity was kept, not that a file exists

The acceptance run on CI saw a fresh add report its catalog reused: the
content identity job of an enclosing source opened the new source's store
between registration and the file probe. The flag now answers from the
registration (it existed, or a descriptor-bound catalog was adopted), which
is what it claims.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Regenerate TypeScript types for the catalog_reused doc

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Volume tracking test follows the one-library-per-path rule

volumes.track now surfaces a source setup failure instead of logging it,
so tracking a volume another open library holds is refused. The
multi-library test asserts the refusal, then the handover once the first
library untracks the drive and removes its source.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
2026-10-07 20:25:10 +00:00
..

xtask

Build automation tasks for Spacedrive using the xtask pattern.

What is xtask?

The xtask pattern is the idiomatic Rust way to handle project-specific build automation. Instead of using shell scripts, Makefiles, JavaScript, or external build tools, you write your build tasks as Rust code in a workspace member called xtask.

This approach is used by major Rust projects including:

  • rust-analyzer - IDE support
  • tokio - async runtime
  • cargo itself - Rust's package manager

Benefits

  • Pure Rust - No shell scripts or JavaScript tooling
  • Type-safe - Catch errors at compile time
  • Debuggable - Use standard Rust debugging tools
  • Cross-platform - Works on Windows, macOS, Linux
  • No external tools - Just cargo and rustup
  • Self-contained - No need for pnpm, node, or JavaScript

Usage

Run tasks using cargo xtask:

# Setup development environment (replaces pnpm prep)
cargo xtask setup

# Build iOS framework (device + simulator)
cargo xtask build-ios

# Or use the convenient alias:
cargo ios

Available Commands

test-core

Single source of truth for core integration tests!

Runs all sd-core integration tests with progress tracking and result summary. This command is used by both CI and local development, ensuring consistency.

Usage:

cargo xtask test-core           # Run with minimal output
cargo xtask test-core --verbose # Show full test output

Features:

  • Progress tracking (shows which test is running)
  • Timing for each test suite
  • Summary report showing passed/failed tests
  • Same test definitions used in CI workflows
  • Continues running even if some tests fail

All tests are defined in xtask/src/test_core.rs as the single source of truth. Add or remove tests there and they automatically apply to both CI and local runs.

setup

Replaces pnpm prep with a pure Rust implementation!

Sets up your development environment:

  1. Downloads native dependencies (FFmpeg, protoc, OpenSSL, libheif, etc.)
  2. Extracts them to apps/.deps/
  3. Downloads iOS-specific deps if iOS targets are installed
  4. Creates symlinks for shared libraries (macOS/Linux)
  5. Generates .cargo/config.toml from the template

Usage:

cargo xtask setup

First time setup:

# Install Rust if you haven't already
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Add iOS targets (macOS only, optional)
rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios

# Run setup
cargo xtask setup

# Build the CLI
cargo build

build-ios

Builds the sd-ios-core library for iOS (macOS only):

  1. Compiles for aarch64-apple-ios (physical devices)
  2. Compiles for aarch64-apple-ios-sim (M1/M2 simulator)
  3. Compiles for x86_64-apple-ios (Intel simulator)
  4. Creates universal simulator library (ARM64 + x86_64) using lipo
  5. Updates the existing XCFramework that Xcode is already using

Output: apps/ios/sd-ios-core/sd_ios_core.xcframework/

The XCFramework structure looks like:

sd_ios_core.xcframework/
├── Info.plist                           # Top-level XCFramework metadata
├── ios-arm64/
│   ├── libsd_ios_core.a                 # Device library
│   └── Info.plist                       # Device metadata
└── ios-arm64-simulator/
    ├── libsd_ios_core.a                 # Universal simulator library (ARM64 + x86_64)
    └── Info.plist                       # Simulator metadata

How It Works

The xtask pattern works through Cargo's workspace and alias features. When you run:

cargo ios

Cargo:

  1. Looks for an alias in .cargo/config.toml
  2. Expands it to cargo run --package xtask -- build-ios
  3. Builds and runs the xtask binary
  4. Passes build-ios as an argument

The xtask binary is just a regular Rust program that uses std::process::Command to invoke cargo builds and file operations.

Migration from Old Setup

Replaced: pnpm prep (JavaScript)

Old way:

pnpm i              # Install JS dependencies
pnpm prep           # Run JavaScript setup script

New way:

cargo xtask setup   # Pure Rust, no JS needed!

Replaced: scripts/build_ios_xcframework.sh (Bash)

Old way:

./scripts/build_ios_xcframework.sh

New way:

cargo ios           # Convenient alias
# or
cargo xtask build-ios

Why the Change?

  • Old: Required pnpm, node, JavaScript dependencies, bash scripts
  • New: Only requires Rust toolchain (cargo/rustup)
  • Result: Faster setup, fewer dependencies, more maintainable

Requirements

  • Rust toolchain with iOS targets installed:
    rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios
    
  • Xcode with iOS SDK (for lipo command)

Adding New Tasks

To add a new task:

  1. Add a new function in src/main.rs
  2. Add a match arm in main() to call your function
  3. Document it in the help message
  4. Optionally add a cargo alias in .cargo/config.toml

Example:

fn clean_ios() -> Result<()> {
    // Clean iOS builds
    todo!()
}

// In main():
match args[1].as_str() {
    "build-ios" => build_ios()?,
    "clean-ios" => clean_ios()?, // Add this
    _ => { /* ... */ }
}

Further Reading