Commit Graph

534 Commits

Author SHA1 Message Date
Oscar Beaumont
b6ea70b8d0 Hosted Locations - Part 1 (#1861)
* Hosted locations CRD

* Authorise + file upload demo

* Configurable path for testing perms
2024-01-08 06:10:26 +00:00
Oscar Beaumont
26f610014a Improve ephemeral walker (#1877)
* `non_indexed::walk` as `impl Stream`

* wip

* this is gooooood

* savepoint

* remove

* Batched stream

* `unsafe_streamed_query`

* nightmare nightmare nightmare nightmare nightmare nightmare

* JS Mutex

* cleanup

* proper error handling

* myCode satisfies Typescript

* Move to npm fork of rspc

* fixes

* rspc more crashy crashy

* Typescript is very disappointed
2024-01-02 06:09:44 +00:00
Brendan Allan
5371e9f961 [ENG-1512] Replace sync operation types with single type (#1900)
* replace multiple op types with crdt_operation

* cleanup
2023-12-21 05:58:03 +00:00
Ericson "Fogo" Soares
7c90bcb95b [ENG-1479] AI Prototype (#1845)
* First draft on image labeling

* Fixing execution providers for other OSs

* Better error handling and shutdown

* Working with shallow media processor

* bruh

* Fix warnings

* Now hooked to media processor job

* Link desktop app with libonnxruntime to avoid TLS error during startup

* Be able to change models on runtime
Revert to use labels table instead of tags

* A bug on a model-less inference

* Show AI labels on Inspector
 - Change yolo inference to use half precision
 - Add labels api to core

* Remove LD_PRELOAD

* Fix race condition on model executor shutdown

* Don't load all images in memory moron

* Embeed yolo model in prod build
 - Change yolo model path to new one relative to executable

* Disable volume watcher on linux, it was crashing the app
 - Invalidate labels when they are updated

* Rust fmt

* Minor changes

* Gate onnxruntime linking to the ai-models feature

* Add build script to sd-server to handle onnxruntime linking workaround

* Move AI stuff to its own crate and normalize deps

* Rust fmt

* Don't regenerate labels unless asked to

* Now blazingly fast

* Bad merge

* Fix

* Fix

* Add backend logic to download extra yolo models

* Add models api route
 - Add api call to get available model version
 - Add api call to change the model version

* Improve new model download logic
 - Add frontend to change image labeler model

* Fix new model downloader

* Fix model select width

* invalidate labels count after media_processor generates a new output

* Rename AI crate and first draft on download notifications

* fix types

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-12-19 09:28:57 +00:00
Brendan Allan
3cabc9c3a9 Basic actor manager (#1888)
* basic declared actor manager

* put actors in separate file

* clippy

* hopefully clean up some clippy warnings

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-12-13 17:49:22 +00:00
Brendan Allan
a9da6a0093 [ENG-1495] Pull new instance data when receiving cloud sync operations (#1882)
* add sd-cloud-api

* consume sd-cloud-api & leave instance writing to receiving

* module-ise api crate

* put back logging

* remove 'join cloud library'
2023-12-12 07:47:21 +00:00
Oscar Beaumont
801f819cdf Upgrade tracing (#1870)
* Upgrade tracing
Thank the lord it's finally here

* Fix excessive logs from mdns_sd & prisma

* fix excessive logs again
2023-12-07 10:22:26 +00:00
Brendan Allan
4fb9028452 Cloud Sync Basics (#1856)
* cloud operation models

* please clippy

* cloud sync table

* cloud ingest works very simply

* move stuff around + stop invalidation from triggering every 10ms

* cleanup

* fix ts

* remove localhost

* re-enable vendored openssl
2023-12-06 16:47:51 +00:00
Julian Braha
edbff8b398 Resolve warnings instead of suppressing them (#1848)
* Eliminate 'unused mut' warning and suppression

* Eliminate 'unreachable code' warning and suppression

* Fix formatting
2023-12-06 06:16:14 +00:00
Oscar Beaumont
89a7f735e5 [ENG-1400] Normalised caching (#1734)
* prototype

* `.normalise` helper + only `String` keys

* implement it for 'search.paths'

* redux devtools

* fix

* refactor backend

* wip: upgrade to rspc fork

* mega cursed

* Upgrade Specta-related stuff

* Upgrade Typescript

* Cache debug page

* bruh

* Fix optimistic library setting

* Cache clearing

* better timeout

* Fix tags page

* bit of cleanup

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-12-05 13:16:03 +00:00
Brendan Allan
f1b61dbc3d Cloud connection (#1842)
* actual cloud connection wow

* Preliminary cloud library joining

* remove dev overrides

* add back library caching

* re-enable vendored openssl
2023-11-30 07:39:39 +00:00
jake
c79c190c78 [ENG-1426] Apply clippy --fix with moderate lints (#1779)
* chore(clippy): fix

* chore: formatting

* fix: add cfg back to macos crate

* remove cfg to see if clippy works

* Restore macOS crate level conditional compilation

* More warns fixing

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-11-28 17:50:11 +00:00
Brendan Allan
908a13130c [ENG-1435] Saved Searches (#1810)
* saved search CRUD (not perfect)

* saved search settings page

* minor improvements

* fix search filter text apply

* serach in setting

* reduce new tab flicker

* fix tab delete index

* temporarily remove hover effect from applied filters

* fix types

* fix progress

* fix double-add for inOrNotIn

* no more saved searches settings page

* redirect on saved search delete

* cleaner

* fix filter checkbox double fire

* types

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-11-25 00:16:26 +00:00
jake
59ef3b94de [ENG-1095] HEIF* and PDF thumbnail orientation (#1815)
* fix horizontal pdf rendering

* don't rotate certain formats (heif*)

* return `u32` from scaling function

* add `avifs` as a valid extension

* add avifs as a valid MIME type

* re-categorize `avifs` as a video (sequence)

* Fix mime-types

* Add Hif extension
 - Cannon uses a special extension for heif in its cameras: https://github.com/digipres/digipres.github.io/blob/master/_sources/registries/fdd/fddXML/fdd000525.xml#L213-L221

* hif is actually a heic

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2023-11-24 22:07:25 +00:00
Ericson "Fogo" Soares
ea4913e028 Concurrency crash for ephemeral thumbs (#1821)
Solve
2023-11-24 16:31:45 +00:00
ameer2468
a7fed83556 [ENG-1413] Full disk access perms (#1791)
* fda wip

* clippy

* add tauri invoke fns for FDA

* fda wip

* clippy

* add tauri invoke fns for FDA

* wip

* fda wip

* clippy

* add tauri invoke fns for FDA

* wip

* wip

* wip fda

* remove imports

* hopefully improve FDA

* execute only on macos

* ts

* ts

* Update Platform.tsx

* Update AddLocationButton.tsx

* remove console log

* fix: fda and add unit tests

* temp commit for Jake

* add fda state and keybind handling (so the frontend is kept up to date)

* update FDA

* update imports

* testing purposes

* Jakes work

* fix fda checks

* work in progress (but not working)

* remove dead files

* attempt #2

* !!!temporarily enable devtools in prod

* remove alert

* show FDA screen but don't require it

* add an FDA button to general client settings

* Update AddLocationButton.tsx

* remove dead code

* unused dep

* old errors

* remove import

* dead code

* dead code + typesafety

* eslint

* remove fda dialog references

* remove mp4 vid

* hopefully fix onboarding for non-macos OSes

* shorter nav

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-11-23 20:54:45 +00:00
Ericson "Fogo" Soares
425c410c62 Thumbnailer channel with capacity 0 bug (#1808)
Solve
2023-11-23 18:10:44 +00:00
Ericson "Fogo" Soares
f662bf7594 [ENG-1398] Make thumbnailer parallelism configurable (#1780)
* Removing migrator

* Adding serde_repr dep

* Generalizing version manager

* Updating library config to use version manager

* Updating node config to use version manager

* Updating stuff to use new configs

* More async

* Library manager errors

* Node and thumbnailer preferences

* Small warning on prep

* Upgrading batch construction for thumbnailer

* Updating thumb version file to new version manager

* Configurable parallelism for thumbnailer

* Integration with frontend and refresh batch

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

---------

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>
2023-11-23 11:23:32 +00:00
Vítor Vasconcellos
f3ad0d7513 Pin rust to 1.73 (Workaround Internal Compiler Error) (#1796)
* Pin rust to 1.73
- Clippy fixes

* Don't install cargo-watch on CI
2023-11-17 22:47:06 +00:00
Jamie Pine
27f077ea0b [ENG-1269] Search options (#1561)
* search options start

* small progress

* more

* bunch of stuff

* semi functioning filters

* cleanup setup api

* progress

* remove filters

* hooked up to query epic moment

* fix

* move db stuff to specific modules

* in/notIn for some fields

* generate ts

* big gains

* working filter options for locations, tags and kind

* working search query

* perfect fixed filters

* saved searches lol

* merge error

* saved searches via api

* better routing

* [ENG-1338] Fix fresh Spacedrive install failing to start due to attempting to query a nonexistent Library (#1649)

Fix Spacedrive failing to start due to attempting to query a nonexistent Library
 - Rename useShoudRedirect to useRedirectToNewLocations
 - Improve behaviour for the immedite redirection after adding a new location

* Show hidden files false by default (#1652)

bool

* fix remove filter in list

* tweaks

* fix nav buttons

* unify MediaData search handling

* cleanup saved search writing

* Add left top bar portals for tags and search + fixed media view on tags

* added search to filter dropdown

* render cycle improvements

* hotfix

* wip

* Refactor with Brendan, but this is a WIP and the search query no longer works

Co-authored-by: Brendan Allan <Brendonovich@users.noreply.github.com>

* progress

* fix location/$id page

* fix tags too

Co-authored-by: Brendan Allan <Brendonovich@users.noreply.github.com>

* 3rd refactor lol

epic style

* half-done with enum-ification of SearchFilterArgs

* broken fixed filters but working inNotIn filters

* search name + extension kinda working

* hidden filter

* fixed filters working??

* deferred search value

* extensions works

* filtered search items mostly working

* tweaks

* stacked approach working for non-search filters

* move to Explorer/Search

* actually use filterArgs in queries

things actually work properly now

* added new icons from Mint

* goof

* cleanup types, filters and mutation logic

* actually use search value

* remove overview from sidebar

* don't shrink LibrariesDropdown ga

* remove overview from sidebar and default to /network

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Brendan Allan <Brendonovich@users.noreply.github.com>
2023-11-17 06:58:44 +00:00
Brendan Allan
031ac7b670 [ENG-1276] move assigned tags to top of tag assign list (#1768)
* move assigned tags to top of tag assign list

* sort tags by most recently assigned

* console.log

* error boundary
2023-11-14 09:35:50 +00:00
Carter
be004e2cf2 Clarify error that occurs when getting the hash of an on-demand file (#1722)
* Clarify Error

- Clarifies error that occurs when attempting to read the metadata from an on-demand file

* Make error more better

Changed 'Attempted' to 'Failed'

* Add comment and conditional compilation
2023-11-13 17:39:22 +00:00
Oscar Beaumont
5705f40aae [ENG-1333] Display active P2P listeners (#1644)
* wip

* share the actual port

* fixes

* wip: remove it

* cargo fmt

* thanks @niikeec
2023-11-13 07:02:03 +00:00
Oscar Beaumont
561eacfb6e [ENG-1201] P2P Error Handling (#1705)
* squash

* wip

* wip

* wip

* wip

* finish it up
2023-11-13 06:58:01 +00:00
Ericson "Fogo" Soares
88859026b1 [ENG-1360] Not always unlink objects from file_paths on update (#1751)
* Small warnings

* Introducing file and line number into tracing logs

* Update walker to account for existing object_ids

* Only unlinking at indexer for file_path with
object referencing more than one file_path

* Better filtering orphans file_paths

* Foreground thumbs generation for shallow media
processor

* Watcher tweaks to preserve existing object
Also fixing a media_data extraction/update bug
2023-11-09 13:47:38 +00:00
Oscar Beaumont
395e0ee3da [ENG-1344] Limit node name length (#1730)
limit length
2023-11-04 09:16:15 +00:00
Ericson "Fogo" Soares
f23e0b13c6 [ENG-1181] File System actions for Ephemeral Files (#1677)
* Backend side

* Rust fmt

* Removing uneeded duplicate files rspc route

* Create folder for ephemeral files

* Ephemeral delete files

* First draft on copy, cut and delete, still buggy

* Fixing copy function and updating async-channel dep

* Rename and some fixes
2023-11-03 17:06:34 +00:00
Brendan Allan
c91ccff37d [ENG-443] Allow folders to be tagged (#1712)
* allow folders to be tagged

* remove bruh

* custom useAssignItems hook

* fix types
2023-11-01 05:23:56 +00:00
Oscar Beaumont
53e4316143 [ENG-1369] Remove broadcast-stream from sd-p2p (#1704)
* let me create pr

* a whole lotta changes

* split `p2p_manager.rs` into smaller files

* the arcpocalypse is over

* minor generic cleanup

* wip removing 'MetadataManager'

* more wip

* wip: i am changing branch

* discovery2 -> discovery

* make it somewhat compile

* more wip

* wip: reassembling manager stream

* state more goodly

* wip

* more wip

* removing generic from sd_p2p::Manager

* reassemble networked libraries

* wip: hooking back up mDNS

* multi-flume wip

* contain bad code to a single file

* p2p_manager_actor + split handlers into file per operation

* cleanup after restructure

* cleaning up more

* wip: reenable resync

* wip: remote identity in connection payload

* track connected clients (required for `service.rs`)

* a big ass iterator

* working towards finishing `service.rs`

* service shutdown

* hook up listen channel in service

* fix address resolution

* merge nlm stuff into LibrariesService

* finish library to service mapping

* less footguns in p2p - seal `PeerId`

* fix previous pr

* p2p state rspc query

* send node events to the frontend

* minor

* wip

* more worky, less crashy

* make spacedrop work + debug state

* fix mdns expiry

* clippy

* other clippy

* remove broadcast type

* merge conflict issue
2023-10-31 04:27:25 +00:00
Oscar Beaumont
02775921ef [ENG-1318, ENG-1199, ENG-931, ENG-1046] Ability to disable mDNS discovery (#1620)
* let me create pr

* a whole lotta changes

* split `p2p_manager.rs` into smaller files

* the arcpocalypse is over

* minor generic cleanup

* wip removing 'MetadataManager'

* more wip

* wip: i am changing branch

* discovery2 -> discovery

* make it somewhat compile

* more wip

* wip: reassembling manager stream

* state more goodly

* wip

* more wip

* removing generic from sd_p2p::Manager

* reassemble networked libraries

* wip: hooking back up mDNS

* multi-flume wip

* contain bad code to a single file

* p2p_manager_actor + split handlers into file per operation

* cleanup after restructure

* cleaning up more

* wip: reenable resync

* wip: remote identity in connection payload

* track connected clients (required for `service.rs`)

* a big ass iterator

* working towards finishing `service.rs`

* service shutdown

* hook up listen channel in service

* fix address resolution

* merge nlm stuff into LibrariesService

* finish library to service mapping

* less footguns in p2p - seal `PeerId`

* fix previous pr

* p2p state rspc query

* send node events to the frontend

* minor

* wip

* more worky, less crashy

* make spacedrop work + debug state

* fix mdns expiry

* clippy

* other clippy

* remove old tests

* add tests back

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-10-30 08:24:04 +00:00
Oscar Beaumont
95f48ea575 [ENG-1367] TypeError: null is not an object (#1698)
strickter types pog
2023-10-30 00:43:18 +00:00
nikec
fe5df79175 [ENG-1378] Add gridGap to preferences (#1696)
Add gridGap to preferences
2023-10-29 14:19:15 +00:00
Utku
bf93d97874 Fix mobile onboarding (#1670)
* attempts

* expo stop using spaces pls

* attempt

* you used to work without this sir

* fixed

* use serde instead of specta

* remove `serde(default`) and update bindings

* fix mobile

* fix types

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-10-26 17:38:15 +00:00
Ericson "Fogo" Soares
033e61ac33 [ENG-1320 | ENG-1340] Separate thumbnails FS location by library | Make the thumbnails actor report back its progress (#1656)
* Renaming error type on ffmpeg subcrate

* Version manager overhaul

* Reworked thumbnail actor

* Updating sharding scheme

* New migration system for thumbnails

* Updating search to new actor

* Updating custom_uri to new thumbnail improvements

* Updating library to new thumbnail stuff

* LibraryId type alias

* Updating indexer to new thumbnail actor

* Updating watcher to new thumbnail actor

* Update location manager to use LibraryId type alias

* Updating location metadata to new LibraryId type alias

* New LocationPubId type alias

* Updating ephemeral walker to new thumbnail stuff

* Updating media processor to new thumbnail actor

* New thumbnailer actor state manager

* Introducing the concept of job phases

* Segregating the thumbnailer actor worker fn

* Fixes on job pausing

* Processing batches with progress reporting

* Updated actor

* Updated media processor

* Small tweaks

* Updating non indexed walker

* Changing a UI string
2023-10-26 04:48:29 +00:00
jake
cc72f54c3d [ENG-1210] Navigate to location after adding it (#1454)
* remove annoying log

* return location id on creation

* add checkbox to open new location once it's been added

* redirect if checkbox was true and a location id was provided, and update bindings

* add `new()` for `MissingFieldError`

* return location id on location relink

* working, clean redirecting

* accordion closing tag

* navigate to location after adding

* chore: remove erroneous `{' '}`

* multiple location redirect support

* oops missed check

* cleanup

* fix bad merge

---------

Co-authored-by: ameer2468 <33054370+ameer2468@users.noreply.github.com>
2023-10-20 21:01:13 +00:00
pr
5f417e7c1b Job text tabular-nums to reduce jumpiness (#1647)
job text tabular-nums to reduce jumpiness
& swap Scanning: and Found: text to also reduce jumpiness
2023-10-20 16:56:34 +00:00
Oscar Beaumont
758ee5bbe2 [ENG-1332] Fix regression causing P2P to be disabled by default (#1641)
lol, lmao, rofl
2023-10-20 08:57:19 +00:00
Ericson "Fogo" Soares
801473191a [ENG-1224] Duplicate folder duplicates files inside (#1635)
Properly handling directory renaming duplicates
2023-10-19 21:28:41 +00:00
Vítor Vasconcellos
34fe26e442 [ENG-1229] Show hidden files not working correctly (#1583)
* Ensure hidden prop is updated when file is modified

* Fix show hidden files in ephemeral locations

* Restore hidden prop for NonIndexedPathItem
2023-10-19 05:14:42 +00:00
Ericson "Fogo" Soares
7ea5a0ee5b [ENG-1308] Jobs shouldn't get stuck forever (#1625)
Look mom, unstuckable jobs!
2023-10-19 03:50:45 +00:00
nikec
ace3527bfc [ENG-1305] "Add Locations" onboarding flow (#1611)
* locations onboarding flow

* optional

* Backend for default location on library creation

* Rust fmt

* Enhancing error handling and introducing more resilience

* Removing .spacedrive metadata on library deletion

* Rust fmt again

* default to videos

---------

Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-10-19 03:18:36 +00:00
Oscar Beaumont
f67b7e807a [ENG-1291] enable/disable P2P & configurable port (#1609)
* it works pog

* UI for when networking is disabled

* Fix Clippy issues

* clippy tings
2023-10-18 03:52:54 +00:00
Oscar Beaumont
1b856848e8 [ENG-1310, ENG-1300] Spacedrop better testing + fix zero-sized files (#1606)
* unit tests save lives

* Support for zero-sized files

* fix: clippy

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-10-17 06:12:38 +00:00
Oscar Beaumont
a7ad24c641 [ENG-625] Spacedrop improvements (#1447)
* Fist fight with the abstraction

* Prevent opening multiple Spacedrop dialogues

* Infinite renders = infinite power

* Comment be gone

* Spacedrop multiple files

* correlate connection id's

* fixes

* thanks Typescript

* Clippy errors + fix cfg's from main
2023-10-17 04:31:42 +00:00
Jamie Pine
3709ab4883 [END-1309] Fix explorer query invalidation (#1604)
fix
2023-10-17 03:54:41 +00:00
Ericson "Fogo" Soares
bfe5a65949 [ENG-1267] Move thumbnails generation away from media processor (#1585)
* New vscode task to start developing

* Updating db in case of library updates just in case

* Done

* Forgot to remove some debug logs

* Rust fmt

* Saving thumbnails processing state on app shutdown

* bruh
2023-10-17 02:09:36 +00:00
Vítor Vasconcellos
65f13bf204 [ENG-1268] Fix custom indexer rule crashing location scanning (#1584)
Fix custom indexer rule crashing location scanning
2023-10-14 06:33:37 +00:00
Oscar Beaumont
39744b5c55 Reduce production logs + button on error page for logs (#1565)
* reduce prod logs

* Open logs dir button on error page
2023-10-13 08:50:01 +00:00
Brendan Allan
25fa1f0974 [ENG-1255] Feedback popover with new API (#1532)
* feedback popover with new api

* allow unauthed feedback

* add login information

* allow no auth

* fix hooks

* remove emoji on reset

* show cancel on left of login button
2023-10-13 08:08:07 +00:00
Ericson "Fogo" Soares
996361f081 [ENG-1160] Opening a location while indexing causes a lot of jobs to be spawned (#1554)
Solved, but missing a frontend error message
2023-10-13 02:47:13 +00:00