4883 Commits

Author SHA1 Message Date
Jamie Pine
60369e9f00 feat: redundancy awareness & cross-volume file comparison (#3053)
* feat: add redundancy awareness & cross-volume file comparison

Surface content redundancy data so users can answer "if this drive dies,
what do I lose?" — builds on existing content identity and volume systems.

Backend:
- New `redundancy.summary` library query with per-volume at-risk vs
  redundant byte/file counts and a library-wide replication score
- Extend `SearchFilters` with `at_risk`, `on_volumes`, `not_on_volumes`,
  `min_volume_count`, `max_volume_count` filters
- Add composite index migration on entries(content_id, volume_id)

Frontend:
- `/redundancy` dashboard with replication score, volume bars, at-risk callout
- `/redundancy/at-risk` paginated file list sorted by size
- `/redundancy/compare` two-volume comparison (unique/shared toggle)
- Sidebar ShieldCheck button linking to redundancy view

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* redundancy UI improvements + ZFS volume detection fix

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix ZFS pool capacity reporting and stats filtering

- ZFS: override total_capacity for pool-root volumes using zfs list
  used+available. df under-reports pool-root Size because it only
  counts the root dataset's own used bytes plus avail — on a 60 TB
  raidz2 pool this shows as ~15 TB instead of ~62 TB. The pool root's
  own used property includes descendants, so used+available is the
  real usable capacity.

- Library stats: drop volumes where is_user_visible=false AND
  re-apply should_hide_by_mount_path retroactively so stale DB rows
  (detected before the Linux visibility filters existed) don't
  inflate reported capacity.

- Extract should_hide_by_mount_path into volume/utils as a shared
  helper used by both the list query and the stats calculation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* show capacity and visibility in sd volume list

Makes it possible to verify library-level capacity aggregation from
the CLI — previously the list only showed mount, fingerprint, and
tracked/mounted state, which meant debugging the ZFS pool capacity
issue required querying the library DB directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* document filesystem support matrix and detection

New docs/core/filesystems.mdx covering per-filesystem capabilities
(CoW, pool-awareness, visibility filtering, capacity correction),
platform detection strategies, the FilesystemHandler trait, Linux/
macOS/ZFS visibility rules, the ZFS pool-root capacity problem and
fix, copy strategy selection, and known limitations.

Registered under File Management in both mint.json and docs.json.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* WIP: redundancy filter wiring across search, CLI, and UI

- core/src/ops/search: wire redundancy filters (at_risk, on_volumes,
  not_on_volumes, min/max volume_count) through the search query;
  fix UUID-to-SQLite BLOB literal so volume UUID comparisons actually
  match (volumes.uuid is stored as a 16-byte BLOB, quoted-string
  comparison silently returned zero rows).

- apps/cli: new redundancy subcommand + populate the new
  SearchFilters fields from search args.

- packages/interface: redundancy at-risk and compare pages reworked
  to consume the new filter surface; explorer context/hook updates
  to support redundancy-scoped views.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* add web context menu renderer and UI polish

- New WebContextMenuProvider + Radix DropdownMenu-based renderer anchored
  at cursor via a 1x1 virtual trigger. Handles separators, submenus,
  disabled, and the danger variant via text-status-error.
- useContextMenu now routes web clicks through the provider instead of
  parking data in unused local state, and trims leading/trailing/adjacent
  separators so condition-filtered menus don't render orphaned lines.
- Drop app-frame corner rounding on the web build.
- Add shrink-0 to the sidebar space switcher so the scrollable sibling
  can't compress it vertically.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* sd-server dev workflow: auto web build, shutdown watchdog, stable data dir

- build.rs runs `bun run build` in apps/web so `just dev-server` always
  embeds the latest UI. rerun-if-changed covers apps/web/src,
  packages/interface/src, and packages/ts-client/src so Rust-only edits
  skip the rebuild. Skips gracefully when bun isn't on PATH or
  SD_SKIP_WEB_BUILD is set; Dockerfile sets the latter since dist is
  pre-built and bun isn't in the Rust stage.
- Graceful shutdown was hanging because the browser holds the /events
  SSE stream open forever and axum waits for all connections to drain.
  After the first signal, arm a background force-exit on second Ctrl+C
  or 5s timeout so the process can't stick.
- Debug builds were starting from a fresh tempfile::tempdir() on every
  run (the TempDir handle dropped at end of the closure, deleting the
  dir we just took a path to). Default to ~/.spacedrive in debug so data
  persists and `just dev-server` shares a data dir with the Tauri app.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* add Sources space item alongside Redundancy in default library layout

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* add TrueNAS native build script

Uses zig cc as C/C++ compiler on TrueNAS Scale where /usr is
read-only and no system gcc exists. Dev tools live at
/mnt/pool/dev-tools/ (zig, cmake, make, extracted deb headers).

Builds sd-server + sd-cli in ~4 min on a 12-core NAS. AI feature
disabled (whisper.cpp C11 atomics incompatible with zig clang-18).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* never block RPC on synchronous statistics calculation

On first load (fresh library, all stats zero), libraries.info used
to calculate statistics synchronously before responding. On large
libraries during active indexing this hangs indefinitely — the
closure-table walk in calculate_file_statistics loads every
descendant ID into a Vec then issues a WHERE IN(...) with millions
of entries, which SQLite can't finish while the indexer is writing.

Now always return cached (possibly zero) stats and let the
background recalculate_statistics task fill them in. The UI
refreshes via the ResourceChanged event when the calculation
completes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* self-heal protocol handler registration on re-init

Core::new() registers default protocol handlers after starting networking,
but swallows any failure (error is only logged). If the initial registration
fails — e.g. on a host where start_networking hasn't fully set up the event
loop command sender by the time register_default_protocol_handlers runs —
the registry is left empty. A subsequent call to Core::init_networking()
would see `services.networking().is_some()` and skip re-registration,
permanently leaving protocols unregistered for the life of the process.

sd-server calls init_networking() right after Core::new(), so it's the
client most exposed to this. Symptom: pairing over the web UI returns
"Pairing protocol not registered" while the same library works fine
from Tauri and mobile.

Fix: init_networking now queries the registry directly for the pairing
handler and re-registers the default set if it's missing, independent of
whether networking is already initialized.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fall back to pkarr+DNS discovery when mDNS port is unavailable

Iroh's endpoint.bind() fails wholesale if any configured discovery service
fails to initialize. MdnsDiscovery requires binding UDP :5353, which on
most Linux systems (including TrueNAS) is already owned by avahi-daemon.
Result: endpoint creation errors out with "Service 'mdns' error", the
event loop never starts, command_sender stays None, and protocol
registration fails — so sd-server has no working networking at all.

Make mDNS best-effort: on any error whose message mentions "mdns",
retry endpoint creation with only pkarr + DNS discovery. Local-network
auto-discovery is lost but remote pairing via node ID (which uses n0's
DNS infrastructure, not mDNS) continues to work normally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* succeed pairing if either mDNS or relay discovery wins

The dual-path discovery in start_pairing_as_joiner_with_code used
tokio::select! to race mDNS and relay. select! resolves on the first
branch to complete — including errors — so a host that can't bind
mDNS (e.g. a Linux box where avahi already owns UDP :5353) would fail
pairing wholesale: mDNS discovery errors out in <1ms with
"Failed to create mDNS discovery: Service 'mdns' error", that Err
wins the race, and relay discovery gets cancelled before it can even
begin.

Switch to futures::select_ok so we only return the error if EVERY
discovery path has failed. mDNS failing immediately now leaves relay
running to completion, which is the common case for remote pairing
into a NAS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 18:56:51 -07:00
S.L
88e1412175 feat(tags): tag system with explorer integration, and media context menu fixes (#3054)
* fix(search): apply TagFilter in search.files query

Tags were silently ignored by FilterBuilder. Adds find_entry_ids_for_tag()
(batch lookup via user_metadata_tag → user_metadata → entry, handles both
entry_uuid and content_identity_uuid paths) and resolve_tag_filter() (AND
logic for include, OR for exclude). Applied in both execute_fast_search_no_fts()
and execute_fast_search() FTS path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tags): implement tags.by_id, tags.ancestors, tags.children, files.by_tag

Fixes "Tag not found" when clicking a tag in the sidebar, and shows the
actual tagged files in the tag view.

Backend (register_library_query via inventory):
- tags.by_id: find tag by UUID via TagManager::get_tags_by_ids
- tags.ancestors: get ancestor tags via TagManager::get_ancestors
- tags.children: get descendant tags via TagManager::get_descendants
- files.by_tag: find entries via user_metadata_tag → user_metadata → entry
  (handles both entry_uuid and content_identity_uuid paths)

Frontend:
- TagView: replace ExplorerView (used global Explorer context, ignored
  files.by_tag results) with a direct file list rendered from TaggedFileSummary

TODOs for tag feature follow-up:
- files.by_tag: return full File objects with sd_path so files are
  clickable/actionable (currently: id, name, extension, size only)
- tags.related handler (sidebar shows related tags)
- "Filters" button in TagView: secondary filters (type/date/size) within
  tagged files
- tags.children in TagView currently uses get_descendants (all levels);
  should use get_direct_children for the quick-filter chips
- DEL key binding for removing a tag from a file (#21 dependency resolved)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tags): prevent duplicate tag applications on the same file

The apply_tags_to_metadata() relied on catching a unique constraint error
to detect duplicates, but no such constraint existed — so every call to
tags.apply would silently create a new row.

- Migration m20260125: deduplicates existing rows (keeps MIN(id) per pair),
  then adds UNIQUE INDEX(user_metadata_id, tag_id)
- apply_tags_to_metadata(): explicit check-before-insert (upsert pattern),
  independent of DB constraint

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tags,ui): make tag view files navigable and wire Overview search button

- files.by_tag query now joins directory_paths/volumes/devices to build
  SdPath::Physical, enabling navigation from tag view to explorer
- Tag view: double-click navigates to parent folder (files) or into
  directory; use react-router navigate() instead of window.location.href
- Overview: search button now navigates to /explorer instead of no-op

* feat(tags): render tag view using standard explorer with full File objects

Backend:
- files.by_tag now returns Vec<File> (full domain objects with
  File::from_entity_model) instead of lightweight TaggedFileSummary,
  matching the same data format as directory_listing and search.files

Frontend:
- Add tag mode to explorer context (ENTER_TAG_MODE/EXIT_TAG_MODE)
- useExplorerFiles supports tag source via files.by_tag query
- Tag route activates tag mode and renders ExplorerView directly,
  giving tagged files the same UI as normal file browsing (list/grid
  views, thumbnails, selection, context menus, keyboard shortcuts)
- Fix ExplorerView empty state guard to allow tag/recents/search modes
  without requiring a currentPath

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(tags): add unapply/delete actions, fix tag sync and Inspector UX

Backend:
- Add tags.unapply action: remove tags from files by entry UUID,
  resolves via both entry_uuid and content_identity_uuid paths
- Add tags.delete action: delete a tag and all its relationships
  via TagManager::delete_tag()
- Add EntryUuid variant to TagTargets and ApplyToTargets to accept
  frontend UUIDs (fixes parseInt(uuid) bug that tagged wrong files)
- files.by_tag: batch load tags for returned files (same pattern as
  directory_listing) so Inspector shows tags in tag view
- navigateToPath exits tag mode to prevent empty directory on nav

Frontend:
- Tag primitive: add onRemove prop with X button for inline removal
- FileInspector: optimistic tag updates via updateSelectedFileTags,
  refetchQueries with correct query keys (query:files.by_tag prefix)
- TagsGroup: right-click delete with confirmation, active state
- useFileContextMenu: "Remove tag" option when in tag mode
- TagSelector: fix create+apply with EntryUuid fallback
- Generated types: add DeleteTagInput/Output, UnapplyTagsInput/Output,
  EntryUuid variant to TagTargets and ApplyToTargets

* refactor: extract shared useRefetchTagQueries hook

Extract duplicated refetchQueries calls from FileInspector,
useFileContextMenu, TagsGroup, and TagSelector into a single
useRefetchTagQueries hook. Removes direct useQueryClient usage
from those files.

* fix(core): use current device slug instead of \"unknown-device\" fallback

When the SQL join to devices table returns no result (volume_id or
device_id NULL), fall back to get_current_device_slug() instead of
the hardcoded \"unknown-device\" string. The previous fallback caused
SdPath::is_local() to return false, breaking ephemeral indexing when
navigating to directories from the tag view.

Fixed in both files.by_tag and search.files queries.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(media): replace broken useJobDispatch with direct mutations

Context menu used useJobDispatch/jobs.dispatch which has no backend
handler, causing all media processing (thumbnail, OCR, transcribe,
thumbstrip, proxy) to fail from the context menu.

- Replace all 15 runJob() calls with direct useLibraryMutation calls
  (media.thumbnail.regenerate, media.ocr.extract, etc.)
- Add forEachTarget helper for batch operations
- Add mime_from_extension() fallback in RegenerateThumbnailAction for
  indexed files where content_identity MIME lookup fails
- useJobDispatch.ts is now dead code (no remaining imports)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): address CodeRabbit review findings on tag system

- TOCTOU race: replace check-then-insert with atomic ON CONFLICT upsert
  in metadata manager (prevents duplicate tag applications under concurrency)
- children query: use get_direct_children (depth=1) instead of
  get_descendants (entire subtree) for tags.children endpoint
- delete atomicity: wrap tag cascade deletion in a transaction
  (relationships, closure, applications, usage patterns, tag)
- files_by_tag: implement include_children and min_confidence filters
  (were declared in input but ignored)
- files_by_tag: map content_id from SQL result instead of fabricating None
- files_by_tag: merge entry-scoped and content-scoped tags with dedup
  (previously content-scoped tags were silently dropped)
- unapply: emit resource events for all entries sharing content, not just
  the directly specified entries
- frontend: derive tagModeActive from mode.type instead of storing
  separately (prevents state desynchronization)
- Document sync deletion gaps with TODO(sync) comments

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(migration): keep newest row (MAX id) when deduplicating tag applications

The dedup query before creating the unique index on (user_metadata_id, tag_id)
was keeping MIN(id) — the oldest row. Since user_metadata_tag rows carry mutable
state (version, updated_at, device_uuid), keeping MAX(id) preserves the most
recent state.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* revert(tags): restore independent tagModeActive state

The previous commit incorrectly derived tagModeActive from mode.type,
conflating two separate concepts:
- mode: {type: "tag"} = viewing files by tag (sidebar navigation)
- tagModeActive = bulk tag assignment UI bar

These are independent: clicking a tag in the sidebar should show tagged
files without opening the assignment bar. Reverts the context.tsx portion
of 04a181535.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): address second round of CodeRabbit review

- Increment version on ON CONFLICT update path so sync detects changes
- Only report/notify entries that actually lost a tag (skip when 0 rows deleted)
- Exit tag mode on all navigation paths (navigateToView, goBack, goForward)
  to prevent tag mode leaking through non-path navigation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): skip rows with undecodable required fields instead of fabricating

entry_id=0 and modified_at=now() hide real decode failures. Required fields
(entry_id, entry_name, created_at, modified_at) now skip the row with a
warning log. Optional/numeric fields (size, child_count) keep sensible
defaults since 0 is a valid value.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): remove broken optimistic update and alert() dialog

- FileInspector: remove updateSelectedFileTags() which mutated
  selectedFiles while the pane renders from file.tags — the refetch
  on mutation success is what actually updates the UI
- TagsGroup: remove alert() on delete error (console.error suffices,
  alert() breaks platform-agnostic design)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): emit file events on tag delete, refetch files.by_id for inspector

- delete/action.rs: collect affected entry UUIDs before deleting the tag,
  then emit "file" resource events so the explorer grid updates (removes
  tag dots). Follows the same pattern as apply and unapply actions.
- useRefetchTagQueries: add files.by_id to the refetch list so the
  Inspector panel updates immediately after tag mutations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): add extension to root-level file paths, validate entry UUIDs

- files_by_tag.rs: root-level files (no parent_path) were missing their
  extension in the constructed path — now uses the same name+ext logic
  as the parent_path branch
- apply/action.rs: validate that entry UUIDs exist in the entry table
  before creating user_metadata rows, since there is no FK constraint
  at the SQLite level on user_metadata.entry_uuid — prevents orphaned
  metadata from invalid UUIDs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): pre-index content rows to avoid O(n²) tag merge, require entry_kind

- load_tags_for_entries: pre-build HashMap<content_uuid, Vec<entry_uuid>>
  from rows in a single pass, then lookup in the content-scoped branch
  instead of rescanning all rows per metadata record
- entry_kind treated as required field (skip row with warning instead of
  silently defaulting to 0, which would misclassify directories as files)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): secure FTS5 escaping, batch entry lookups for performance

- FTS5 search: wrap each query token in double quotes to prevent
  operator injection (AND, OR, NOT, -, *, etc.)
- apply/action.rs: replace per-entry UUID lookups with batch query
  (Entry branch: single WHERE IN instead of N round trips)
- apply/action.rs: replace per-entry existence validation with batch
  query (EntryUuid branch: single WHERE IN instead of N round trips)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): remove redundant inline sea_orm imports

ColumnTrait, EntityTrait, QueryFilter are already imported at top-level.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tags): validate entry UUIDs in create action before applying

* fix: address code review feedback from CodeRabbit

- Resolve merge conflict markers in TagSelector.tsx
- Add type: 'all' to refetchQueries for inactive cache refresh
- Replace browser confirm() with useContextMenu in TagsGroup
- Add onSuccess refetch to createTag mutation in TagsGroup
- Remove unnecessary 'as any' casts in OverviewTopBar
- Replace alert() with toast.error() in useFileContextMenu
- Remove 'any' casts in useExplorerFiles tag/directory queries
- Add nil UUID rejection in tag apply input validation
- Propagate SeaORM errors in thumbnail MIME lookup
- Wrap tag upsert loop in DB transaction for atomicity
- Fix tab indentation in migration mod.rs

* fix: validate create tag targets, confirm before delete, escape LIKE wildcards, remove dead code

* fix: prevent tagging ephemeral files and improve empty tag view UX

* fix: platform confirm, parameterized SQL, remove dead code, handle serialization errors

* fix: use native Tauri dialog for confirm on Windows (WebView2 broken)

* fix: prevent double callback in platform.confirm and distinguish tag error types

* fix: separate missing-target, null-UUID, and execution errors in tag apply

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: James Pine <ijamespine@me.com>
2026-04-18 18:53:12 -07:00
Jamie Pine
75479df3df fix library sync backfill O(N^2) hotspots on both sides of the protocol (#3061)
* fix library sync backfill O(N^2) hotspots on both sides of the protocol

Addresses #3058 (shared models, content_identity) and #3060 (device-owned
entries). The progressive slowdown on initial pair came from several
independent hotspots that compound:

Sender side
- PeerLog::get_since now pushes LIMIT into SQL; callers fetch limit + 1 to
  derive has_more. Previously every SharedChangeRequest reloaded and
  parsed the entire remaining shared_changes log in memory before
  truncating, making sender work O(N * batches).
- Entry and content_identity query_for_sync batch FK to UUID conversion
  across the whole batch via a new convert_fks_to_uuids_batch helper —
  one DB round trip per FK type instead of per record per FK.

Schema
- New index idx_entries_indexed_at_uuid backing the (indexed_at, uuid)
  cursor in Entry::query_for_sync. Without it every batch request fell
  back to a full-table scan.

Receiver side
- New task-local in_backfill scope wraps the backfill apply phases.
  Entry::apply_state_change uses it to skip per-entry entry_closure
  rebuild; the existing post_backfill_rebuild pass does a single bulk
  rebuild at the end. emit_batch_resource_events short-circuits during
  backfill and the coordinator emits one Event::Refresh after post-
  backfill rebuild so the UI invalidates cached views.
- Entry FK resolution in backfill splits mappings into self-referential
  (parent_id) and the rest. Non-self FKs resolve in one batch across the
  whole batch; only parent_id still runs per-entry so children can see
  just-inserted parents.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* address PR review feedback: drop unresolved FKs, guard limits

convert_fks_to_uuids_batch now returns the set of record indices whose FK
could not be resolved (missing target row or non-integer value). Callers
in entry::query_for_sync and content_identity::query_for_sync drop those
records from the outgoing sync batch instead of zipping the partially
converted payload back. Previously the sender would ship a record with
its local int field intact and no *_uuid field; the receiver's
map_sync_json_to_local interpreted that as already-resolved and wrote
the sender-local integer directly to the local DB, corrupting the FK.

Also: reject limit == 0 on SharedChangeRequest / get_shared_changes up
front (would have returned 0 rows with has_more = true and spun), and
clamp the SQL LIMIT bind with i64::try_from(..).unwrap_or(i64::MAX)
instead of a wrapping `as i64`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* further review fixes: treat absent FK fields as failed, drop NULL indexed_at

convert_fks_to_uuids_batch previously treated a missing local_field as
null. convert_fk_to_uuid errors in that case — only an explicit JSON
null is a legitimate null FK — so the batch helper now flags absent
fields as failed to match the per-record contract.

Entry::query_for_sync no longer falls back to modified_at when
indexed_at is NULL. The cursor filter/order uses indexed_at exclusively,
so a returned cursor derived from modified_at wouldn't match the next
query's predicate. The indexed_at backfill migration populated existing
rows; any NULL here is a data bug, logged and skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* cargo fmt

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 18:51:32 -07:00
S.L
c63f1b078b fix: update tests/examples for upstream API changes and fix sync loop busy-loop (#3057)
* fix: update tests and examples for upstream API changes

Align error types, struct fields, and constructor calls in 27 test/example
files to match current Core API:

- Box<dyn Error> -> Box<dyn Error + Send + Sync> to match Core::new()/shutdown()
- entry::ActiveModel device_id -> volume_id (field rename)
- location::ActiveModel add volume_id field (new nullable field)
- AppConfig add proxy_pairing and spacebot fields (config v5/v6)
- Re-export ProxyPairingConfig from config module
- VolumeFingerprint::new() -> from_primary/external/network_volume()
- DeviceInfo add device_type field, NetworkFingerprint public_key -> public_key_hash
- SecretKey::generate(thread_rng) -> from_bytes (rand_core version mismatch)
- SdPath::Physical.path String -> PathBuf, thumbnail_url &Uuid -> &str
- init_sync_service unwrap networking Option for Arc<dyn NetworkTransport>

* fix: prevent busy-loop in sync loop when real-time sync is active

The sync loop's Ready state used 'continue' when is_realtime_active()
returned true, which skipped the sleep at the bottom of the loop and
caused a CPU-saturating busy-loop logging 'Skipping catch-up' hundreds
of thousands of times per second. Replace with if/else so the
realtime-active branch falls through to the configurable sleep.
2026-04-18 18:51:13 -07:00
S.L
d14413a75c fix(ts-client): prevent Overview crash from single-resource cache seeding (#3055)
updateSingleResource seeded the TanStack cache with { files: [...] }
when oldData was null (buffer replay race). This list-shaped fallback
broke single-resource queries like libraries.info where the response
is a plain object, causing TypeError on statistics.total_capacity.

Return undefined instead so the queryFn delivers the correct shape.
Batch seeding in updateBatchResources is preserved for file listings.
2026-04-18 18:51:05 -07:00
James Pine
9945639853 retroactively hide stale tracked system/app volumes on Linux
Two follow-ups to the Linux volume filter:

- Expand is_nested_app_mount to match the root paths (/mnt/.ix-apps,
  /mnt/pool/ix-applications), not just their children.
- In volumes.list, when a tracked DB entry has no matching live volume
  (because the new filters skip it entirely — virtual fs, /sys/firmware,
  etc.), re-evaluate visibility by mount path and override the stale
  is_user_visible value that was persisted by earlier builds. This
  avoids needing a data migration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 14:37:17 -07:00
James Pine
fb5d4d7e7d filter system and app-managed volumes on Linux
Linux volume detection had almost no filtering beyond a basic virtual-fs
check, so on TrueNAS Scale we surfaced 60+ volumes (OS datasets, iX-apps
nested volumes, virtual filesystems) when the user only has ~6 real data
datasets. This mirrors the filtering macOS already does via
should_be_user_visible.

Adds three layers:

1. Expanded is_virtual_filesystem to cover cgroup, squashfs, efivarfs,
   overlay, and friends.

2. New is_system_mount_point and is_nested_app_mount helpers covering
   the standard Linux FHS (/usr, /var, /etc, etc.) plus nested
   app/container mount patterns (/.ix-apps/, /ix-applications/,
   docker overlay2, snap, ZFS snapshots). TrueNAS Scale's split-root
   datasets fall out of this naturally — they're still just /usr, /var.

3. ZFS-aware filtering in enhance_volume_with_cached_output: datasets
   on known system pools (boot-pool, rpool, zroot) are marked System +
   hidden, and datasets whose hierarchical name contains ix-applications
   / docker / containerd are hidden even if they mount somewhere
   outside the app root.

Hidden volumes are still tracked internally (stable fingerprints, can
still be used as locations) — they're just not shown in the default
volume list.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 01:34:01 -07:00
James Pine
f443063b0e fix HttpTransport to multiplex subscriptions over a single EventSource
Every useNormalizedQuery with a unique filter opened its own EventSource
to /events. Browsers cap HTTP/1.1 connections at 6 per origin, so the UI
exhausted the connection pool and RPC POSTs stalled indefinitely. The
daemon's /events stream is already a broadcast of every event, so we
share one EventSource across all subscribers and fan out in the client.

This only affected the web prod build (single origin); dev worked
because vite and sd-server were on different ports, and Tauri worked
because it uses IPC for subscriptions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 00:35:04 -07:00
James Pine
8beb815f84 bundle sd-cli into server release tarball
Build both sd-server and sd-cli in a single cargo invocation (shared
dependency compilation) and ship them together. Downloads of
sd-server-linux-<arch>.tar.gz now include the CLI so headless
deployments have a way to manage the daemon.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 22:46:00 -07:00
James Pine
e56621ed37 fix ZFS volume detection performance on systems with many datasets
Fetch `zfs list` output once and reuse across all ZFS volumes instead of
spawning a subprocess per volume, which caused 30s+ startup timeouts on
TrueNAS systems with many ZFS datasets.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 18:57:30 -07:00
James Pine
ef65e9b74b fix(web): polyfill crypto.randomUUID for non-HTTPS contexts
crypto.randomUUID is only available in secure contexts (HTTPS/localhost).
Add a polyfill so the web app works when served over HTTP on LAN.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 01:10:01 -07:00
James Pine
d313d5a12a fix: isolate @spacebot/api-client from web/server builds
Remove VoiceOverlay and Spacebot re-exports from @sd/interface barrel
export. Tauri imports them directly via subpath exports. Add importmap
shim in web index.html so externalized @spacebot/api-client doesn't
break at runtime. Add @sd/interface subpath exports for Spacebot and
VoiceOverlay.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 22:15:11 -07:00
James Pine
101e96f9e6 fix(windows): replace broken icon.ico, fix Obsidian SVG gradients
The icon.ico was a PNG renamed to .ico, breaking the Windows resource
compiler (error RC2175). Replace with a proper multi-size .ico. Also
fix decimal separators in Obsidian adapter SVG gradient transforms.

Co-Authored-By: slvnlrt <slvnlaurent@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 21:15:47 -07:00
James Pine
12669697bc better volume calc + tailscale plan 2026-04-14 21:05:48 -07:00
James Pine
198b4b5b65 fix: externalize @spacebot/api-client when spacebot unavailable
Both web and tauri builds fail in CI because @spacebot/api-client
isn't available. Conditionally externalize it when the spacebot repo
isn't present as a sibling directory.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 20:17:19 -07:00
James Pine
56230245c2 fix: use macos-26 for x86_64 darwin build (cross-compile)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 20:15:19 -07:00
James Pine
f79bbdceef fix: tauri CI builds + macOS runners + spaceui-link
- Add missing @vitejs/plugin-react-swc to tauri devDependencies
- Make spaceui/spacebot vite aliases conditional in tauri config
- Use package export paths for @spacedrive/tokens CSS imports
- Use regex aliases to prevent prefix-matching collisions
- Switch macOS release runners to macos-26 (GitHub-hosted)
- Fix spaceui-link to run bun link from workspace root

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 19:51:14 -07:00
James Pine
f68606abf2 fix(web): externalize @spacebot/api-client for web builds
Spacebot is a separate repo not available in CI. The VoiceOverlay
component that imports it is tauri-only, so externalize it for the
web/server build.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 19:11:49 -07:00
James Pine
f3d617cc4c fix: use package exports for @spacedrive/tokens CSS imports
Update CSS @import paths to use the package export map
(@spacedrive/tokens/theme, /css, /css/themes/*) instead of internal
source paths. Update vite aliases to map export paths back to source
files when spaceui is available locally.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 18:52:03 -07:00
James Pine
1d8dfa1bf5 fix(web): conditionally apply spaceui/spacebot vite aliases
Only apply @spacedrive/* and @spacebot/* vite aliases when the sibling
repos exist locally. In CI and production builds, packages resolve from
node_modules via npm. Revert spaceui checkout from release workflow.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 18:38:15 -07:00
James Pine
db5f362f12 fix(web): add missing @spacedrive/tokens dependency
The web app imports @spacedrive/tokens/src/css/theme.css in index.css
but didn't declare it as a dependency, causing the vite build to fail
in CI where spaceui isn't available as a sibling directory.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 18:17:05 -07:00
James Pine
37bb1b2f26 fix release CI + rustfmt + storage navigation
- Add bun setup and web frontend build steps to server-build job in
  release workflow (rust-embed needs apps/web/dist/ at compile time)
- Fix rustfmt violation in volume detection
- Fix storage dialog callback to use sdPath instead of location_id
- Navigate to explorer path after adding storage

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 17:58:40 -07:00
James Pine
1e58784482 fix volume detection + redundancy doc 2026-04-14 15:50:44 -07:00
Jamie Pine
b6c5937364 Update link from spacedrive.com to v2.spacedrive.com 2026-04-14 15:09:21 -07:00
Jamie Pine
a8f6e45dc5 Merge pull request #3047 from spacedriveapp/spacedrive-data
Archive System: External Data Source Indexing
2026-04-14 15:08:36 -07:00
James Pine
beab5a715b Add spaceui-link and spaceui-unlink justfile recipes
Matches the spacebot justfile pattern for local SpaceUI development.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 14:01:05 -07:00
Jamie Pine
c877cc243b disable spacebot window 2026-04-14 11:22:10 -07:00
Jamie Pine
8a7fc53cdf Fix ~750 TypeScript typecheck errors
- Add ReactComponent export to *.svg module declarations
- Fix SdPath imports to use generated types (device_slug, Cloud/Sidecar variants)
- Create useJobs barrel file for JobManager hooks
- Remove unused imports across ~65 files
- Add type annotations for implicit any params (d3, callbacks, map iterators)
- Remove stale @ts-expect-error directives in MeshViewer
- Add declare module for gaussian-splats-3d and qrcode
- Fix Location field names (path→sd_path, total_file_count→file_count, etc)
- Fix SdPath discriminated union narrowing (remove stale Local variant)
- Fix React 19 RefObject<T|null> vs RefObject<T> mismatches
- Fix null vs undefined mismatches throughout
- Add missing required fields to ApplyTagsInput, policy types, etc
2026-04-14 08:56:21 -07:00
Jamie Pine
730b177965 Fix remaining CI failures: rustfmt, TypeScript errors
- cargo fmt across all modified files
- Add Vite client types and Window.__SPACEDRIVE__ declaration
- Fix @sd/interface/platform import to @sd/interface
- Align @types/react versions between tauri and interface packages
- Remove unused imports/vars in useDropZone, DragOverlay, ContextMenuWindow
- Fix WebviewWindow.location references to use globalThis
- Exclude updater.example.ts from typecheck
2026-04-12 23:13:50 -07:00
Jamie Pine
acde17c088 Fix CI: rustfmt, xtask empty TARGET_TRIPLE, and package linking
- Sort `pub mod adapters` alphabetically in ops/mod.rs
- Wrap long line in volume/fs/refs.rs
- Handle empty TARGET_TRIPLE env var in xtask setup
- Replace `link:@spacedrive/*` with published `^0.2.3` versions
2026-04-12 20:33:43 -07:00
Jamie Pine
74cb8c46bb Merge main into spacedrive-data 2026-04-12 13:40:59 -07:00
Jamie Pine
6c5e656e6d spacedrive server work 2026-04-10 16:58:43 -07:00
Jamie Pine
efa8d07992 Rename @spaceui/* deps to @spacedrive/* 2026-04-04 21:35:20 -07:00
James Pine
3383fb7cdb Remove deprecated explorer sidebar (replaced by SpacesSidebar)
Delete Sidebar.tsx, SidebarItem.tsx, Section.tsx, LocationsSection.tsx
from the old explorer sidebar. The active sidebar is SpacesSidebar
which uses SpaceItem from @spaceui/primitives.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 03:49:55 -07:00
James Pine
adac2f3c45 Source sync jobs, voice overlay rewrite, TabBar migration, docs
- Refactor source sync to dispatch SourceSyncJob instead of inline sync
- Rewrite VoiceOverlay with audio recorder and TTS playback hooks
- Migrate TabBar to @spaceui/primitives
- Update SpacebotContext query invalidation
- Add SpaceUI section to CONTRIBUTING.md and README.md
- Update sources UI (Adapters, SourceDetail)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:12:52 -07:00
James Pine
907ab4e267 Add archive sources UI with full Rust integration
- Create core/src/data/ module with SourceManager wrapping sd-archive Engine
- Add Sources to GroupType and Source to ItemType enums
- Add default Sources group to new library creation
- Register source operations: create, list, get, delete, sync, list_items
- Register adapter operations: list, config, update
- Add bundled adapter sync from workspace adapters/ directory
- Add adapter update system with BLAKE3 change detection and backup/rollback
- Frontend: Sources home, source detail with virtualized list, adapters screen
- Frontend: SourcesGroup sidebar, SpaceGroup dispatch, spaceItemUtils
- Frontend: TopBar integration (path bar, search, sync, actions menu)
- Frontend: Tab title sync, adapter icon lookup hook
- Regenerate TypeScript types

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 03:24:53 -07:00
Jamie Pine
9f0685abc9 ui 2026-03-28 18:19:48 -07:00
Jamie Pine
f93a34ca73 remove sub modules 2026-03-27 00:38:27 -07:00
Jamie Pine
457833ee19 remove ui refs 2026-03-26 22:48:27 -07:00
James Pine
0c19c64a17 ui 2026-03-26 22:23:29 -07:00
James Pine
22020cf592 ui tweaks 2026-03-26 22:06:43 -07:00
James Pine
e36ee3a625 delete old ui 2026-03-26 21:10:31 -07:00
James Pine
1c3d7d2a4c ui 2026-03-26 21:09:37 -07:00
James Pine
edebc7423f yes 2026-03-26 19:48:47 -07:00
James Pine
a8b1f4d2a4 migration 2026-03-26 19:32:01 -07:00
James Pine
bb5076776d tailwind upgrade 2026-03-26 17:29:55 -07:00
Jamie Pine
be454a0b48 Merge pull request #3021 from StarbirdTech/fix/whitepaper-formatting
fix(whitepaper): text overflow in two-column layout
2026-03-26 12:28:27 -07:00
Jamie Pine
48c574c5ea Merge pull request #3041 from slvnlrt/delete-improvements
fix(delete): resolve Content paths, progress updates, DEL keybinds
2026-03-26 12:22:38 -07:00
Jamie Pine
03799060c7 Merge pull request #3040 from slvnlrt/windows-dark-titlebar
fix(windows,ui): dark titlebar via DWM API and sidebar hover effects
2026-03-26 12:22:05 -07:00
Jamie Pine
7b2ef810ae Merge pull request #3043 from slvnlrt/fix-location-watcher
fix(locations): register watcher on location add, canonicalize paths
2026-03-26 12:21:16 -07:00