A store has two halves. The generation is rebuildable for as long as its
origin still answers, the assertion layer never is, and the origin stops
answering without an event. So the invariant is a rule about what code may
assume rather than a claim about stores, and the registry row tracks what is
actually known about the origin.
Falls out of that: reindex and evict are transactions over one half, and sync
replicates the generation while merging assertions. Which means the assertion
tables need their key and their merge columns settled before P2 fills them,
so P1 gains the re-key, convergent content ids and the uuid narrowing.
New doc is docs/core/design/source-durability.md. Also records two
contradictions found on the way: whether sources may nest (contract 1 and the
previs disagree), and that nothing reconciles record uuids when two devices
index the same origin.
The README still advertised a screening pipeline — Prompt Guard 2, quarantine,
trust metadata on results — that was a stub and is gone. Cut the section
rather than describe an intention as a feature.
The two archive docs went stale from later commits in the same session: they
still put durable assertions in registry.db, still said deleting a source
leaves them behind, and did not mention record_overlay in the store layout.
Both now describe one file per source with no shared durable layer.
Also: the crate layering is stated in both docs, AGENTS.md points at
crates/store for the record table, and the crate docs stop promising
cross-source edges — edge relates records within one source, and anything
spanning sources arrives with the catalog.
docs/core/design/archive.md and docs/archive/README.md described archive as
a second data plane beside the VDFS. The README also documented an
adapter.toml format and a JSONL protocol that never existed in this tree,
and claimed a subprocess sandbox that doesn't. Both rewritten from source.
"spine" is now "the record table" across the remaining docs, and
2026-08-17-record-spine-transplant.md renamed to match.
Adds docs/plans/2026-08-22-source-convergence.md: the register for getting
to one store shape and one source registry before the entries teardown
continues. AGENTS.md and the teardown execution plan point at it.
Photos no longer runs its own demo bake; it follows a Spacedrive explorer
window via a new navigation-focus resource and renders that window's
folder as a media grid, sourced from thumbnails baked by a new daemon-owned
thumbs service. Supporting changes: pvcache moves to a v2 format that
stores each tile's true aspect (envelope + frame) instead of forcing a
square crop, peer mounts now refresh on a generation check instead of only
at reconnect, and the archive crate's `spine` module is renamed to `record`
throughout.
Absolute paths from one developer's machine sat in an example, a shell script,
recorded benchmark metadata, task docs and design docs. The example and the
script were broken for anyone else as a result: the example now resolves against
CARGO_MANIFEST_DIR and the script derives the repo root from git. The duplicate
copy of that script under core/scripts is gone.
Also drops a pointer to a repository that is not public.
The entry schema is the last thing standing from the old model and nothing
needs it. These cover the cutover, the destination written as though built, and
the hazards for work landing next to it.
Image thumbs no longer sit behind ffmpeg gates, sidecar versions bust the cache,
the sidecar layout moves to a shared crate, and metadata backfill is separated
from generation. Video dimensions are read for real; the cas_id tier and dead
settings are removed.
Read-only share on loopback, a byte-range peer protocol for remote sources, and
a block cache in front of both so a second read of the same file does not
re-fetch it.
Sources resolve at startup without onboarding. Adds the Apple Photos native
adapter, collections and the analyzer, the bundle lens, and assertions with
read-time binding.
Each source owns its partition with detached restore, volume fingerprints are
machine-independent so an external drive keeps its identity across hosts, and
directory-size rollups update incrementally. Snapshots gain a restore gate, a
save invariant, unique tmp writes and an fsync before rename.
crates/archive carries record, content and edge tables with a durable overlay
beside them, so the rebuildable generation can be dropped and recreated without
losing anything written through the assertion surface.
* feat: add redundancy awareness & cross-volume file comparison
Surface content redundancy data so users can answer "if this drive dies,
what do I lose?" — builds on existing content identity and volume systems.
Backend:
- New `redundancy.summary` library query with per-volume at-risk vs
redundant byte/file counts and a library-wide replication score
- Extend `SearchFilters` with `at_risk`, `on_volumes`, `not_on_volumes`,
`min_volume_count`, `max_volume_count` filters
- Add composite index migration on entries(content_id, volume_id)
Frontend:
- `/redundancy` dashboard with replication score, volume bars, at-risk callout
- `/redundancy/at-risk` paginated file list sorted by size
- `/redundancy/compare` two-volume comparison (unique/shared toggle)
- Sidebar ShieldCheck button linking to redundancy view
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* redundancy UI improvements + ZFS volume detection fix
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix ZFS pool capacity reporting and stats filtering
- ZFS: override total_capacity for pool-root volumes using zfs list
used+available. df under-reports pool-root Size because it only
counts the root dataset's own used bytes plus avail — on a 60 TB
raidz2 pool this shows as ~15 TB instead of ~62 TB. The pool root's
own used property includes descendants, so used+available is the
real usable capacity.
- Library stats: drop volumes where is_user_visible=false AND
re-apply should_hide_by_mount_path retroactively so stale DB rows
(detected before the Linux visibility filters existed) don't
inflate reported capacity.
- Extract should_hide_by_mount_path into volume/utils as a shared
helper used by both the list query and the stats calculation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* show capacity and visibility in sd volume list
Makes it possible to verify library-level capacity aggregation from
the CLI — previously the list only showed mount, fingerprint, and
tracked/mounted state, which meant debugging the ZFS pool capacity
issue required querying the library DB directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* document filesystem support matrix and detection
New docs/core/filesystems.mdx covering per-filesystem capabilities
(CoW, pool-awareness, visibility filtering, capacity correction),
platform detection strategies, the FilesystemHandler trait, Linux/
macOS/ZFS visibility rules, the ZFS pool-root capacity problem and
fix, copy strategy selection, and known limitations.
Registered under File Management in both mint.json and docs.json.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* WIP: redundancy filter wiring across search, CLI, and UI
- core/src/ops/search: wire redundancy filters (at_risk, on_volumes,
not_on_volumes, min/max volume_count) through the search query;
fix UUID-to-SQLite BLOB literal so volume UUID comparisons actually
match (volumes.uuid is stored as a 16-byte BLOB, quoted-string
comparison silently returned zero rows).
- apps/cli: new redundancy subcommand + populate the new
SearchFilters fields from search args.
- packages/interface: redundancy at-risk and compare pages reworked
to consume the new filter surface; explorer context/hook updates
to support redundancy-scoped views.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* add web context menu renderer and UI polish
- New WebContextMenuProvider + Radix DropdownMenu-based renderer anchored
at cursor via a 1x1 virtual trigger. Handles separators, submenus,
disabled, and the danger variant via text-status-error.
- useContextMenu now routes web clicks through the provider instead of
parking data in unused local state, and trims leading/trailing/adjacent
separators so condition-filtered menus don't render orphaned lines.
- Drop app-frame corner rounding on the web build.
- Add shrink-0 to the sidebar space switcher so the scrollable sibling
can't compress it vertically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* sd-server dev workflow: auto web build, shutdown watchdog, stable data dir
- build.rs runs `bun run build` in apps/web so `just dev-server` always
embeds the latest UI. rerun-if-changed covers apps/web/src,
packages/interface/src, and packages/ts-client/src so Rust-only edits
skip the rebuild. Skips gracefully when bun isn't on PATH or
SD_SKIP_WEB_BUILD is set; Dockerfile sets the latter since dist is
pre-built and bun isn't in the Rust stage.
- Graceful shutdown was hanging because the browser holds the /events
SSE stream open forever and axum waits for all connections to drain.
After the first signal, arm a background force-exit on second Ctrl+C
or 5s timeout so the process can't stick.
- Debug builds were starting from a fresh tempfile::tempdir() on every
run (the TempDir handle dropped at end of the closure, deleting the
dir we just took a path to). Default to ~/.spacedrive in debug so data
persists and `just dev-server` shares a data dir with the Tauri app.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add Sources space item alongside Redundancy in default library layout
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add TrueNAS native build script
Uses zig cc as C/C++ compiler on TrueNAS Scale where /usr is
read-only and no system gcc exists. Dev tools live at
/mnt/pool/dev-tools/ (zig, cmake, make, extracted deb headers).
Builds sd-server + sd-cli in ~4 min on a 12-core NAS. AI feature
disabled (whisper.cpp C11 atomics incompatible with zig clang-18).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* never block RPC on synchronous statistics calculation
On first load (fresh library, all stats zero), libraries.info used
to calculate statistics synchronously before responding. On large
libraries during active indexing this hangs indefinitely — the
closure-table walk in calculate_file_statistics loads every
descendant ID into a Vec then issues a WHERE IN(...) with millions
of entries, which SQLite can't finish while the indexer is writing.
Now always return cached (possibly zero) stats and let the
background recalculate_statistics task fill them in. The UI
refreshes via the ResourceChanged event when the calculation
completes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* self-heal protocol handler registration on re-init
Core::new() registers default protocol handlers after starting networking,
but swallows any failure (error is only logged). If the initial registration
fails — e.g. on a host where start_networking hasn't fully set up the event
loop command sender by the time register_default_protocol_handlers runs —
the registry is left empty. A subsequent call to Core::init_networking()
would see `services.networking().is_some()` and skip re-registration,
permanently leaving protocols unregistered for the life of the process.
sd-server calls init_networking() right after Core::new(), so it's the
client most exposed to this. Symptom: pairing over the web UI returns
"Pairing protocol not registered" while the same library works fine
from Tauri and mobile.
Fix: init_networking now queries the registry directly for the pairing
handler and re-registers the default set if it's missing, independent of
whether networking is already initialized.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fall back to pkarr+DNS discovery when mDNS port is unavailable
Iroh's endpoint.bind() fails wholesale if any configured discovery service
fails to initialize. MdnsDiscovery requires binding UDP :5353, which on
most Linux systems (including TrueNAS) is already owned by avahi-daemon.
Result: endpoint creation errors out with "Service 'mdns' error", the
event loop never starts, command_sender stays None, and protocol
registration fails — so sd-server has no working networking at all.
Make mDNS best-effort: on any error whose message mentions "mdns",
retry endpoint creation with only pkarr + DNS discovery. Local-network
auto-discovery is lost but remote pairing via node ID (which uses n0's
DNS infrastructure, not mDNS) continues to work normally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* succeed pairing if either mDNS or relay discovery wins
The dual-path discovery in start_pairing_as_joiner_with_code used
tokio::select! to race mDNS and relay. select! resolves on the first
branch to complete — including errors — so a host that can't bind
mDNS (e.g. a Linux box where avahi already owns UDP :5353) would fail
pairing wholesale: mDNS discovery errors out in <1ms with
"Failed to create mDNS discovery: Service 'mdns' error", that Err
wins the race, and relay discovery gets cancelled before it can even
begin.
Switch to futures::select_ok so we only return the error if EVERY
discovery path has failed. mDNS failing immediately now leaves relay
running to completion, which is the common case for remote pairing
into a NAS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added a new GitHub Actions workflow (`mobile.yml`) for manual dispatch to build and release the React Native app for iOS and Android.
- Updated the publish artifacts action to change Windows bundle type from MSI to NSIS and adjusted related paths.
- Modified the front-end bundle path to reflect the new structure in the mobile app.
- Enhanced documentation to include details about the new mobile release process and required secrets.
- Updated device versioning to use the `updated_at` timestamp for conflict resolution.
- Refactored sync methods to support shared changes with HLC-based conflict resolution.
- Implemented automatic device discovery via shared sync, allowing devices to propagate changes to all members in the library.
- Added migration to transition existing devices to shared sync.
- Enhanced documentation to reflect changes in device ownership and sync methods.
This commit refines the volume ownership model by ensuring that entries and locations inherit ownership from their respective volumes. It updates the documentation to clarify the sync ownership flow, emphasizing the seamless transfer of ownership when external drives are connected to different devices. Additionally, it improves the overall clarity of the sync state machine and related processes, ensuring that the documentation accurately reflects the system's behavior and enhances understanding for developers and users.
This commit updates the data model to replace the `device_id` field with `volume_id` in the `entries` and `locations` tables. The change allows entries to inherit ownership from their associated volumes, simplifying ownership management during volume transfers. Additionally, migrations have been added to facilitate this transition, ensuring that existing data is correctly updated and indexed. The codebase has been adjusted to reflect these changes across various modules, including sync and query functionalities.
- Added device_id parameter to DatabaseAdapter and DatabaseAdapterForJob for improved context handling.
- Updated ChangeDetector to utilize device_id when creating persistence instances, ensuring accurate indexing behavior.
- Refactored related tests to verify correct parent-child relationships and prevent duplicate entries during folder moves.
- Updated the fixture generation process to write to a temporary directory by default, improving test isolation and following best practices.
- Added instructions for regenerating source fixtures when needed, enhancing developer experience.
- Refactored the test suite structure to utilize a more flexible argument handling mechanism, allowing for easier addition of new tests.
- Removed deprecated test configurations and streamlined the test suite definitions for clarity and maintainability.
- Introduced TestDataDir for managing test data directories with automatic cleanup and optional snapshot support, ensuring all test data is created in the system temp directory.
- Added SnapshotManager to facilitate capturing snapshots of test state for post-mortem debugging, with platform-specific storage locations.
- Updated various integration tests to utilize the new TestDataDir structure, enhancing consistency and determinism in test execution.
- Revised documentation to reflect new conventions for test data management and snapshot usage.
- Modified test data paths in various integration tests to use the Spacedrive source code instead of user directories, ensuring consistent and deterministic test results across environments.
- Updated comments and documentation to reflect the new testing approach and clarify the purpose of using project source code for testing.
- Enhanced the GitHub Actions workflow to skip Rust toolchain setup on macOS self-hosted runners, assuming Rust is pre-installed.
- Introduced a new section on testing filesystem watcher functionality, detailing critical setup steps.
- Added instructions for enabling the watcher in test configurations and using home directory paths on macOS.
- Included best practices for ephemeral and persistent location watching, as well as event collection.
- Provided examples for expected event types and assertions to enhance clarity for developers testing filesystem events.
- Updated the TODO list to clarify the quick preview reporting issue.
- Enhanced the Tauri app's menu to include custom clipboard operations (cut, copy, paste) with appropriate state management.
- Introduced a new `statistics_listener` service to manage per-library statistics recalculation based on resource changes, improving performance and responsiveness.
- Added configuration options for enabling/disabling the statistics listener in the application settings.
- Refactored event handling in the job manager to improve logging and status monitoring.
- Updated tests to include the new statistics listener functionality and ensure proper event collection during file operations.
- Marked subproject commits as dirty for api, ios, macos, and workbench.
- Updated content identity UUID generation to be globally deterministic from content_hash only, enabling cross-device and cross-library deduplication.
- Refactored related documentation to reflect changes in UUID generation logic.
- Added new default group creation for Devices and Tags in LibraryManager.
- Improved keyboard navigation and selection handling in Explorer component.
- Introduced a new File Operation Modal for interactive file copy/move operations with conflict resolution options.
- Integrated sound effects for file operations, including copy and startup sounds.
- Updated the Explorer component to utilize the new modal for drag-and-drop file operations.
- Refactored job management hooks to unify job data handling and improve performance.
- Enhanced documentation for the File Operation Modal and sound effects integration.
- Rename indexing backend: DBWriter to DatabaseStorage
- Replace EphemeralWriter with MemoryAdapter across watcher and
ephemeral components
- Update module paths and imports in core indexing code, job, and
persistence layers to use DatabaseStorage and MemoryAdapter
- Update docs to reflect new names
- (DatabaseStorage, MemoryAdapter)
- Add SyncEventsArgs to export sync events from the CLI - Wire
SyncCmd::Events and implement export_events to fetch - format and write
results - Implement JSON, SQL, and Markdown exporters - with optional
device data in the output - Extend protocol with EventLogRequest and
EventLogResponse - Enable LogSyncHandler to handle event log requests
and return logs - Expose log_handler from BackfillManager for event
logging - Update docs with CLI examples and protocol overview
- Introduce deterministic UUID generation for library defaults (spaces,
groups, and items) - Add post-backfill rebuild support in the sync
registry (with_rebuild) - Export BatchFkMapResult type via the sync API
- Track per-peer RTT latency via SyncMetricsCollector integration - Use
deterministic IDs when creating default space and related items in
LibraryManager
- Revert to the November 16 baseline for realtime sync - Use a single
state watermark and drop per-resource watermarks, counts, and hashes
from watermark messages - Revert DataAvailableNotification handling and
the updated FK filtering approach in favor of NULL-based handling - Add
registry helper get_model_type_by_table to map tables to models for FK
resolution - Update mocks and tests to reflect the baseline (linkage
stats in summary) - Fix test infrastructure: remove block_in_place usage
and minor imports - Remove obsolete test fixture file
- Introduce an Axum-based HTTP server with an embedded daemon and a
JSON-RPC proxy to the daemon via a Unix socket - Bundle web UI assets
into the server with an assets feature and a build.rs that builds the
frontend using pnpm - Add multi-stage Dockerfile, docker-compose.yml,
and a Distroless runtime image - Provide TrueNAS deployment support with
a build script and setup guide - Add a new web UI (apps/web) with a
Vite-based dev/build flow and a web platform shim for the frontend -
Implement server logic (apps/server/src/main.rs): health, auth, /rpc
proxy and data-dir/socket-path wiring - Include server-specific
Cargo.toml and a comprehensive server README - Add architecture and
memory-focused docs to guide usage and design - Minor core tweak:
simplify location/resource event emission in
core/src/location/manager.rs to align with new flow - Tauri app: adjust
menus to add an Edit submenu and remove unused items
- Introduced a new `unique_bytes` field in the volume model to track deduplicated content size.
- Implemented a volume refresh action to recalculate unique bytes for all volumes owned by the device, emitting a cache refresh event to invalidate frontend caches.
- Updated the storage overview component to display unique bytes and other volume statistics.
- Enhanced the event system to handle cache invalidation events, ensuring real-time updates across the application.
- Added necessary migrations and database updates to support the new volume tracking features.
Jamie Pine