* feat: add redundancy awareness & cross-volume file comparison
Surface content redundancy data so users can answer "if this drive dies,
what do I lose?" — builds on existing content identity and volume systems.
Backend:
- New `redundancy.summary` library query with per-volume at-risk vs
redundant byte/file counts and a library-wide replication score
- Extend `SearchFilters` with `at_risk`, `on_volumes`, `not_on_volumes`,
`min_volume_count`, `max_volume_count` filters
- Add composite index migration on entries(content_id, volume_id)
Frontend:
- `/redundancy` dashboard with replication score, volume bars, at-risk callout
- `/redundancy/at-risk` paginated file list sorted by size
- `/redundancy/compare` two-volume comparison (unique/shared toggle)
- Sidebar ShieldCheck button linking to redundancy view
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* redundancy UI improvements + ZFS volume detection fix
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix ZFS pool capacity reporting and stats filtering
- ZFS: override total_capacity for pool-root volumes using zfs list
used+available. df under-reports pool-root Size because it only
counts the root dataset's own used bytes plus avail — on a 60 TB
raidz2 pool this shows as ~15 TB instead of ~62 TB. The pool root's
own used property includes descendants, so used+available is the
real usable capacity.
- Library stats: drop volumes where is_user_visible=false AND
re-apply should_hide_by_mount_path retroactively so stale DB rows
(detected before the Linux visibility filters existed) don't
inflate reported capacity.
- Extract should_hide_by_mount_path into volume/utils as a shared
helper used by both the list query and the stats calculation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* show capacity and visibility in sd volume list
Makes it possible to verify library-level capacity aggregation from
the CLI — previously the list only showed mount, fingerprint, and
tracked/mounted state, which meant debugging the ZFS pool capacity
issue required querying the library DB directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* document filesystem support matrix and detection
New docs/core/filesystems.mdx covering per-filesystem capabilities
(CoW, pool-awareness, visibility filtering, capacity correction),
platform detection strategies, the FilesystemHandler trait, Linux/
macOS/ZFS visibility rules, the ZFS pool-root capacity problem and
fix, copy strategy selection, and known limitations.
Registered under File Management in both mint.json and docs.json.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* WIP: redundancy filter wiring across search, CLI, and UI
- core/src/ops/search: wire redundancy filters (at_risk, on_volumes,
not_on_volumes, min/max volume_count) through the search query;
fix UUID-to-SQLite BLOB literal so volume UUID comparisons actually
match (volumes.uuid is stored as a 16-byte BLOB, quoted-string
comparison silently returned zero rows).
- apps/cli: new redundancy subcommand + populate the new
SearchFilters fields from search args.
- packages/interface: redundancy at-risk and compare pages reworked
to consume the new filter surface; explorer context/hook updates
to support redundancy-scoped views.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* add web context menu renderer and UI polish
- New WebContextMenuProvider + Radix DropdownMenu-based renderer anchored
at cursor via a 1x1 virtual trigger. Handles separators, submenus,
disabled, and the danger variant via text-status-error.
- useContextMenu now routes web clicks through the provider instead of
parking data in unused local state, and trims leading/trailing/adjacent
separators so condition-filtered menus don't render orphaned lines.
- Drop app-frame corner rounding on the web build.
- Add shrink-0 to the sidebar space switcher so the scrollable sibling
can't compress it vertically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* sd-server dev workflow: auto web build, shutdown watchdog, stable data dir
- build.rs runs `bun run build` in apps/web so `just dev-server` always
embeds the latest UI. rerun-if-changed covers apps/web/src,
packages/interface/src, and packages/ts-client/src so Rust-only edits
skip the rebuild. Skips gracefully when bun isn't on PATH or
SD_SKIP_WEB_BUILD is set; Dockerfile sets the latter since dist is
pre-built and bun isn't in the Rust stage.
- Graceful shutdown was hanging because the browser holds the /events
SSE stream open forever and axum waits for all connections to drain.
After the first signal, arm a background force-exit on second Ctrl+C
or 5s timeout so the process can't stick.
- Debug builds were starting from a fresh tempfile::tempdir() on every
run (the TempDir handle dropped at end of the closure, deleting the
dir we just took a path to). Default to ~/.spacedrive in debug so data
persists and `just dev-server` shares a data dir with the Tauri app.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add Sources space item alongside Redundancy in default library layout
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add TrueNAS native build script
Uses zig cc as C/C++ compiler on TrueNAS Scale where /usr is
read-only and no system gcc exists. Dev tools live at
/mnt/pool/dev-tools/ (zig, cmake, make, extracted deb headers).
Builds sd-server + sd-cli in ~4 min on a 12-core NAS. AI feature
disabled (whisper.cpp C11 atomics incompatible with zig clang-18).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* never block RPC on synchronous statistics calculation
On first load (fresh library, all stats zero), libraries.info used
to calculate statistics synchronously before responding. On large
libraries during active indexing this hangs indefinitely — the
closure-table walk in calculate_file_statistics loads every
descendant ID into a Vec then issues a WHERE IN(...) with millions
of entries, which SQLite can't finish while the indexer is writing.
Now always return cached (possibly zero) stats and let the
background recalculate_statistics task fill them in. The UI
refreshes via the ResourceChanged event when the calculation
completes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* self-heal protocol handler registration on re-init
Core::new() registers default protocol handlers after starting networking,
but swallows any failure (error is only logged). If the initial registration
fails — e.g. on a host where start_networking hasn't fully set up the event
loop command sender by the time register_default_protocol_handlers runs —
the registry is left empty. A subsequent call to Core::init_networking()
would see `services.networking().is_some()` and skip re-registration,
permanently leaving protocols unregistered for the life of the process.
sd-server calls init_networking() right after Core::new(), so it's the
client most exposed to this. Symptom: pairing over the web UI returns
"Pairing protocol not registered" while the same library works fine
from Tauri and mobile.
Fix: init_networking now queries the registry directly for the pairing
handler and re-registers the default set if it's missing, independent of
whether networking is already initialized.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fall back to pkarr+DNS discovery when mDNS port is unavailable
Iroh's endpoint.bind() fails wholesale if any configured discovery service
fails to initialize. MdnsDiscovery requires binding UDP :5353, which on
most Linux systems (including TrueNAS) is already owned by avahi-daemon.
Result: endpoint creation errors out with "Service 'mdns' error", the
event loop never starts, command_sender stays None, and protocol
registration fails — so sd-server has no working networking at all.
Make mDNS best-effort: on any error whose message mentions "mdns",
retry endpoint creation with only pkarr + DNS discovery. Local-network
auto-discovery is lost but remote pairing via node ID (which uses n0's
DNS infrastructure, not mDNS) continues to work normally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* succeed pairing if either mDNS or relay discovery wins
The dual-path discovery in start_pairing_as_joiner_with_code used
tokio::select! to race mDNS and relay. select! resolves on the first
branch to complete — including errors — so a host that can't bind
mDNS (e.g. a Linux box where avahi already owns UDP :5353) would fail
pairing wholesale: mDNS discovery errors out in <1ms with
"Failed to create mDNS discovery: Service 'mdns' error", that Err
wins the race, and relay discovery gets cancelled before it can even
begin.
Switch to futures::select_ok so we only return the error if EVERY
discovery path has failed. mDNS failing immediately now leaves relay
running to completion, which is the common case for remote pairing
into a NAS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Changed versioning from "2.0.0-pre.1" to "2.0.0-alpha.1" in various Cargo.toml files and package.json to reflect a new alpha release.
- Added support for ARM64 builds in the GitHub Actions workflow, including a new host configuration and installation of VA-API libraries for the ARM target.
- Introduced a new `bump` command in the xtask to facilitate version updates across all relevant files, improving version management.
Introduces a proper feature flag hierarchy for AI-related dependencies,
allowing lite builds without heavy speech-to-text dependencies.
Feature hierarchy:
- `ai` - umbrella feature for all AI capabilities
- `speech-to-text` - transcription capability (requires ffmpeg + whisper)
- `whisper` - Whisper speech recognition engine (internal)
- `ffmpeg` - audio/video processing (existing)
Changes:
- Make whisper-rs, hound, rubato optional deps behind `whisper` feature
- Add `speech-to-text` feature that requires both ffmpeg and whisper
- Add `ai` umbrella feature that enables all AI features
- Update all cfg guards to use semantic `#[cfg(feature = "speech-to-text")]`
- Expose new features in apps/server and apps/cli
- Update release workflow to include `ai` feature (maintains existing behavior)
This enables:
- Mobile builds without STT (smaller binary, already using default-features=false)
- Desktop "lite" builds for users who want minimal size
- Future AI features can be added under the `ai` umbrella
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Introduced new event types for proxy pairing confirmation and vouching readiness in the event summarization.
- Implemented a reload mechanism for protocol configurations to support dynamic updates during runtime.
- Enhanced the PairingProtocolHandler to manage joiner pairing handshakes and improve session handling.
- Added comprehensive unit tests for proxy pairing, covering session creation, state transitions, and vouching logic.
- Updated configuration structures to include default proxy pairing settings for better integration.
This commit introduces a new feature for safely ejecting removable volumes, enhancing the volume management capabilities. It adds the `VolumeEjectAction`, `VolumeEjectInput`, and `VolumeEjectOutput` structures to handle the eject operation. Additionally, the context menu in the UI is updated to include an option for ejecting volumes, improving user experience. The volume display logic is also refined to ensure accurate representation of volume properties. These changes collectively enhance the overall functionality and usability of the volume management system.
This commit refactors the volume management system by replacing the VolumeItem struct with the Volume struct in the VolumeListOutput, streamlining the data representation. It enhances the volume display logic to utilize display names and adds helper functions to skip serializing unknown disk and volume types. Additionally, the JobManager is updated to include a new field for event emission control, improving job tracking and management. These changes improve overall clarity and efficiency in volume handling and job management.
This commit introduces a detailed memory breakdown feature for the ephemeral indexing process, allowing users to view memory usage by component. It also enhances UUID management by implementing lazy generation and assignment of UUIDs during indexing and searching operations. These changes improve memory tracking and optimize UUID handling, contributing to better performance and user experience in the indexing system.
This commit introduces a new snapshot mechanism for ephemeral indexes, allowing for fast restoration between sessions. The snapshot functionality utilizes zero-copy binary serialization with zstd compression, significantly reducing load times from over 10 minutes to just 1-2 seconds. The implementation includes methods for saving and loading snapshots, as well as enhancements to the existing indexing structure to support this feature. Additionally, the codebase is updated to ensure compatibility with the new snapshot format, improving overall performance and user experience.
This commit improves the `CopyJobDetails` component by adding an auto-scroll feature that centers the currently copying file in the transfer queue. Additionally, it implements a mechanism to refetch copy metadata when the completed file count changes, ensuring that the displayed information is always up-to-date. The UI is further refined with updated icons and styles for better user experience during file operations.
- Fix 'access_token and refresh_token can not be set at the same time' error
- Update Dropbox to use only refresh_token (OpenDAL requirement)
- Add client_id/client_secret to OAuth credential storage
- Add validation for empty OAuth credentials
- Update CLI and volume manager for enhanced OAuth support
- Removed the static ASCII logo and implemented a new function to calculate brightness on a sphere, allowing for a dynamic rendering of the Spacedrive logo as a purple orb using ANSI colors and Unicode half-blocks.
- Enhanced the `print_logo_colored` function to utilize the new brightness calculation and color gradient, improving visual representation.
- Updated related documentation to reflect the changes in logo rendering.
- Added `remove_from_library` field to `DeviceRevokeInput` and `DeviceRevokeAction` to control whether a device should be removed from library databases during revocation.
- Updated the `from_input` method in `DeviceRevokeAction` to handle the new field.
- Modified the revocation logic to conditionally remove devices from libraries based on the `remove_from_library` flag, improving flexibility in device management.
- Adjusted the CLI argument parsing to default `remove_from_library` to false, ensuring devices remain in libraries unless explicitly specified for removal.
- Introduced a new command `ResetCache` in the `IndexCmd` enum to allow users to reset the ephemeral index cache.
- Implemented `EphemeralCacheResetAction` to handle the cache reset logic, including confirmation checks and logging.
- Added input and output structures for the cache reset action, providing clear feedback on the number of cleared paths.
- Enhanced the `EphemeralIndexCache` with a `clear_all` method to facilitate the clearing of all cached paths and entries.
- Added a new `LibraryLoadFailed` event to capture and report errors when loading libraries, including error type categorization for frontend notifications.
- Introduced a method to count `.sdlibrary` directories without loading them, improving the library initialization process.
- Updated the library manager to emit the new event upon load failures, enhancing error handling and user feedback.
- Enhanced logging to provide better insights during library loading and initialization.
- Updated event handling in job management to include device IDs for better tracking of job activities across devices.
- Introduced a new JobActivityClient for subscribing to job activity from remote devices.
- Added hardware specifications to device configuration and database schema, improving device information tracking.
- Refactored event structures to accommodate new fields related to job activities and device management.
- Updated the user interface to display job activity for devices, enhancing user experience in monitoring job statuses.
- Added a step to set up native dependencies using `cargo xtask setup` in both CI and release workflows.
- Updated the build commands for CLI binaries to include `heif` and `ffmpeg` features, ensuring the binaries are built with the necessary capabilities.
- Modified `Cargo.toml` files for `sd-cli`, `sd-server`, and `tauri` to define `heif` and `ffmpeg` features separately, improving modularity and clarity in dependency management.
- Remove TTL-based ephemeral cache and switch to a permanent in-memory
cache.
- Reuse ephemeral UUIDs when creating persistent entries to preserve
continuity of user data.
- Populate ephemeral UUIDs during the processing phase and expose
get_ephemeral_uuid in the indexer state.
- Remove the location invalidation hook and related UI usage.
- Add a complete ephemeral indexing subsystem
- core/src/ops/core/ephemeral_status with input/output and query types
- core/src/ops/indexing/ephemeral with arena, cache, registry,
index_cache, types
- expose EphemeralIndexCache and EphemeralIndex through core modules
- EphemeralIndexCache supports
get/insert/create_for_indexing/mark_indexing_complete eviction and
stats
- Implement EphemeralIndex data structures for memory-efficient storage
- NodeArena, NameCache, NameRegistry, and related types
- Add EphemeralIndex status API
- EphemeralCacheStatusInput and EphemeralCacheStatusQuery
- EphemeralCacheStatus with per-index details
- Wire ephemeral indexing into the indexing flow
- Change default Ephemeral Indexer behavior to shallow mode
- Align code to EphemeralIndex usage across the codebase
- Enhance content kind detection in UI
- Add getContentKind(file) helper (prefers content_identity.kind, then
content_kind)
- Use getContentKind in Explorer utilities and UI components
- Invalidate directory listings when location index_mode changes
- Add useLocationChangeInvalidation to trigger refetches for ephemeral
vs persistent indexing transitions
- Misc refactors and formatting to accommodate the new modules and APIs
- Add SyncEventsArgs to export sync events from the CLI - Wire
SyncCmd::Events and implement export_events to fetch - format and write
results - Implement JSON, SQL, and Markdown exporters - with optional
device data in the output - Extend protocol with EventLogRequest and
EventLogResponse - Enable LogSyncHandler to handle event log requests
and return logs - Expose log_handler from BackfillManager for event
logging - Update docs with CLI examples and protocol overview
- Remove relay-URL based pairing flow and rely on pkarr/DNS for remote
discovery - Extend pairing code with optional node_id and adopt version
2 QR format - Generate pairing codes using node_id and drop relay_url in
non-QR flows - Enable combined mDNS, pkarr, and DNS discovery in the
endpoint - Cache relay_url in persisted device info for reconnection
optimization - Update protocol handlers to propagate node_id and
pkarr-based flow - Adjust tests and logging to reflect the new
cross-network pairing
- Add show_paired flag to device listing (CLI arg and input) - Extend
LibraryDeviceInfo with is_paired and is_connected; include Paired
devices in library listing when requested - Add Devices group to UI
(DevicesGroup) and hook into SpaceGroup - Extend device queries/types to
support show_paired and paired devices - Refactor Dockerfile to
multi-stage Bun + Rust builds; reuse web assets - Remove obsolete
core/ops/entries/mod.rs
- Added `include_descendants` option to event filters, allowing recursive path matching for resource events.
- Updated `affects_path` method to support descendant matching, improving event handling accuracy.
- Refactored subscription logic to utilize the new filtering capabilities, ensuring only relevant events are processed.
- Introduced tests for event filtering to validate exact vs. recursive matching, enhancing reliability of event-driven updates.
- Updated related components to leverage the new filtering options, improving overall performance and user experience.
- Introduced a new `unique_bytes` field in the volume model to track deduplicated content size.
- Implemented a volume refresh action to recalculate unique bytes for all volumes owned by the device, emitting a cache refresh event to invalidate frontend caches.
- Updated the storage overview component to display unique bytes and other volume statistics.
- Enhanced the event system to handle cache invalidation events, ensuring real-time updates across the application.
- Added necessary migrations and database updates to support the new volume tracking features.
- Refactored the event monitoring logic in the CLI to improve readability and maintainability.
- Updated the event summarization function to enhance formatting and clarity of event messages.
- Introduced a dedicated log streaming mechanism in the core, separating log handling from event processing.
- Enhanced the RPC server to support real-time log subscriptions, allowing for more efficient log message delivery.
- Cleaned up legacy log event handling, ensuring a clearer distinction between events and logs in the system.
- Introduced a new Events domain for monitoring events in real-time, allowing users to filter events by type, library ID, job ID, and device ID.
- Updated the CLI to include an Events command, enabling users to access the new monitoring features.
- Enhanced the Cargo.lock and related modules to support the new functionality, ensuring seamless integration with existing systems.
- Added necessary arguments and command handling for the Events command in the CLI.
- Introduced a new field `unique_content_count` in `LibraryStatistics` to track the number of unique content identities.
- Updated various components to calculate and display unique content count, enhancing library statistics output.
- Modified the CLI and interface to reflect the new unique content metric, improving user insights into library data.
- Adjusted serialization and API responses to include the unique content count for better data consistency across the system.
