Commit Graph

621 Commits

Author SHA1 Message Date
Ericson "Fogo" Soares
8c3462a27a [ENG-1513] Better integration between Jobs and processing Actors (#1974)
* First draft on new task system

* Removing save to disk from task system

* Bunch of concurrency issues

* Solving Future impl issue when pausing tasks

* Fix cancel and abort

* Bunch of fixes on pause, suspend, resume, cancel and abort
Also better error handling on task completion for the user

* New capabilities to return an output on a task

* Introducing a simple way to linear backoff on failed steal

* Sample actor where tasks can dispatch more tasks

* Rustfmt

* Steal test to make sure

* Stale deps cleanup

* Removing unused utils

* Initial lib docs

* Docs ok

* Memory cleanup on idle

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-02-26 19:45:58 +00:00
Oscar Beaumont
4ef515d974 rspc over P2P (#2112)
* wip: rspc over p2p

* wip

* rspc over P2P

* Cleanup + error handling

* slight cleanup

* Using Hyper for HTTP streaming + websockets
2024-02-26 07:23:48 +00:00
Vítor Vasconcellos
f4e68497ed Fix core test and CI breaking (#2116)
Fix core test passing inverted arguments to sync_db_entry macro
2024-02-22 06:15:36 +00:00
Brendan Allan
bfd6e813c0 media data sync (#2102)
* basic sync operation backfill

* media data sync

* sync entry helpers

* fix sync generator

* nicer

* re-add key_id
2024-02-21 11:42:10 +00:00
Oscar Beaumont
a7d836a922 Fix P2P not working for libraries (#2031)
* P2P Debug route

* Remove legacy peer to peer pairing process

* Fix error typo

* Sync instances with cloud

* Upgrade deps + extended instance data

* Create instance with extended metadata

* Auto sync instances

* Actually `.await`

* bruh

* sync library info

* this isn't gonna work

* only sleep cloud receiver when no more messages (#1985)

* [ENG-1567] Fix renaming (#1986)

fix rename

* only sleep cloud receiver when no more messages

* use in memory instances during cloud receive (#1995)

* use in memory instances during cloud receive

* is_empty

---------

Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>

* fix type error

* wip

* make mdns mdns better

* rebuild state

* Add hooks + listeners + discovered state

* Split into crates

* wip fixing core + wip merging Spacetime into `sd-p2p2`

* `SmartLockGuard` + `Listener`

* Make `sd-core` compile

* Reenable all operation receivers

* Fix all broken code within `sd-core`

* minor fixes found in review

* Bring in `libp2p` + restructure `sd-p2p` for the gazillion-th time

* whoops

* Compile no matter the (runtime) cost

* fixing merge issues

* wip

* a

* b

* C

* Handle port betterer

* c

* Migrate node config

* a

* no crash on startup

* wip

* a

* jdfhskjfsg

* a

* fix discovery

* a bunch of fixes

* getting Spacedrop working

* I don't get why it no worky

* debug example

* a

* wip

* wip

* removing logging from stream impl

* wip: shit is fucked

* Redo quic integration  + Spacedrop working

* Fix shutdown - deadlocks + shutdown peers

* Add Prisma migrations

* Fix shutdown

* a

* fix

* cleanup

* The lord clippy hath spoken

* disable P2P settings for now

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>
2024-02-21 08:13:40 +00:00
Julian Braha
146838d19e Improve error handling by using decode::Error instead of io::Result (#2078)
Use decode::Error instead of io::Result
2024-02-20 20:06:05 +00:00
Brendan Allan
25f9f03010 Basic sync operation backfill (#2101)
* basic sync operation backfill

* no changes
2024-02-20 09:22:34 +00:00
Utku
b40fd0eb15 Bump version to 0.2.4 (#2080)
* bump

* cargo

* bump height so it doesn't block onboarding

* disable parallelism for image labeler

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-02-14 07:40:01 +00:00
Vítor Vasconcellos
cc94b57cb4 [ENG-1595] Fix some problems with the AI system (#2063)
* Fix some problems with the AI system
 - Fix downloading model using an internal rust string representation as path for the model file
 - Fix Linux loading onnx shared lib from a hardcoded path
 - Fix App should not crash when the AI system fails to start
 - Fix sd-server failing to start due to onnxruntime incorrect linking
 - Some extra clippy auto fixes

* Use latest ort

* Fix dangling sd_ai reference
 - Use entrypoint.sh to initilize container

* Fix server Dockerfile
 - Fix cargo warning

* Workaround intro video breaking onboarding for the web version

* Fix rebase
2024-02-12 17:45:17 +00:00
Brendan Allan
76de9d5fa4 sync support for labels (#2070)
* more sync support for file paths + saved searches

* sync support for labels

* update sync prisma generator to support more than tags

* workey

* don't do illegal db migration

* use name as label id in explorer
2024-02-09 13:20:51 +00:00
Brendan Allan
d3236ae735 More sync support for file paths + saved searches (#2067)
more sync support for file paths + saved searches
2024-02-09 08:17:04 +00:00
Brendan Allan
18ab7cc1a4 [ENG-1548] use in-memory instances when sending messages to cloud (#2057)
* use in-memory instances when sending messages to cloud

* comments
2024-02-06 09:13:47 +00:00
Utku
a409cc9f23 [ENG-1578] Missing localization & language detect and settings (#2037)
* i18n privacy screen

* more missing translations

* language detector

* Language Settings

* sort german

* bump version to 0.2.3
2024-02-02 22:16:01 +00:00
jake
ba3fc33335 Fix localStorage wiping on every launch (#2021)
* fix: wait until libraries have loaded before checking the count

* fix: include new core version in lockfile

* fix: moved value

* fix: check the `libraries/` data dir manually

* Clear localstorage via localStoarge.clear() (#2040)

only clear localstorage in desktop using js

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-02-02 15:55:20 +00:00
Oscar Beaumont
fdd1c5e1e4 Cloud + P2P (#1970)
* P2P Debug route

* Remove legacy peer to peer pairing process

* Fix error typo

* Sync instances with cloud

* Upgrade deps + extended instance data

* Create instance with extended metadata

* Auto sync instances

* Actually `.await`

* bruh

* sync library info

* this isn't gonna work

* only sleep cloud receiver when no more messages (#1985)

* [ENG-1567] Fix renaming (#1986)

fix rename

* only sleep cloud receiver when no more messages

* use in memory instances during cloud receive (#1995)

* use in memory instances during cloud receive

* is_empty

---------

Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>

* fix type error

* wip

* make mdns mdns better

* response

* remove renames

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>
2024-02-02 09:31:11 +00:00
Brendan Allan
2eaf75cb87 [ENG-1592] Correctly assign tab titles on desktop (#2030)
* correctly set tab titles

* use selectedTabIndex

* empty tabId

* prettier
2024-02-01 15:40:38 +00:00
Brendan Allan
bda3a78cf1 Add some documentation for the sync ingester (#2028) 2024-02-01 04:58:48 +00:00
Brendan Allan
d86fb21267 disable orphan remover (#2027) 2024-02-01 04:57:43 +00:00
Brendan Allan
74a72dbbe2 separate cloud sync into sd-core-cloud-sync (#2025)
* separate cloud sync into sd-core-cloud-sync

* ci
2024-01-31 08:16:54 +00:00
Utku
a51b92e817 0.2.2 (#2017)
bump
2024-01-29 17:39:10 +00:00
Brendan Allan
50e668dbba [ENG-1530] Display item count alongside labels (#2011)
* display count alonside labels

* format

* attempt i18n

* item_with_count
2024-01-29 13:10:40 +00:00
jake
3398b2cbcd [ENG-1540] Remove localStorage if no libraries are created (#1945)
* feat: clear localstorage on start if no libraries are found

* chore: add a check to see if the node is for desktop or server use

this will prevent desktop/app localstorage from being cleared if it's a server being hosted

* fix: add secondary localstorage directory for macos

for some reason the app/onboarding progress is kept in `Library/WebKit/Spacedrive` (at least in dev?) so it also needs clearing

* feat: delete cache directories also

* chore: iterate over paths

* feat: add support for windows `localStorage` by using `data_local_dir`

* docs/style: better comments and logs

* feat: remove both `AppData` local and roaming on Windows

* fix: mobile builds

* chore: tweak error message for windows
2024-01-29 12:32:48 +00:00
Utku
d9ef836349 Fix plugin (#1979)
* Fix plugin

* fix typo

* another typo

* typos
2024-01-23 20:11:13 +00:00
jake
6ed6583489 Fix app from not starting due to missing solid-refresh and bump version to 0.2.0 (#1972)
* deps: add `solid-refresh` to `interface`

* bump `core` and `desktop` to `0.2.0`
2024-01-23 09:35:54 +00:00
Vítor Vasconcellos
46a4b788e9 Port AppImage build to use appimage-builder (#1785)
* Initial port to appimage-builder

* Almost

* Fix appimage build on arm64

* Custom patch for external binaries run under appimage
 - Disable bubblewrap sandbox when running under appimage
 - Change cwd to APPDIR when running under appimage

* AppImage Works (for the first lunch, then it crashes with Stack Smash 😭)

* Fix stack smashing, now AppImage almos fully works \o/ (gstreamer is still broken)
 - Temporarily disable the volume watcher when running under appimage (Workaround for the stack smash error)
 - Wrap gnu lic version check for appimage under conditional compile for glibc targets
 - Add error handling for the justUpdatedCheck
 - Fix VITE_LANDING_ORIGIN being undefined

* On non glibc systems default to runtime/compat

* Use glibc version 2.8 instead of 0 for non-gnu systems

* Fix video playback not working due to broken GstRegistry

* Build and publish new AppImage release artifact
 - Fix model location when building deb
 - Improve model path resolution logic
 - Remove patchelf dependency from setup script
 - Fix incorrectly ignore gstreamer dependency in AppImage recipe

* Fix clippy complaining about `get_path_relative_to_exe`
 - Read GLIBC_FAKE_VERSION or use 2.8 for musl in appimage (while the code is there, this is not really supported for now)

* Remove appimage tauri target from release CI

* Remove setup-buildx-action, not relly needed

* typo fix

* Fix git describe command running on cwd instead of the repo root dir

* Attempt fix weird git permissions errors in CI+docker

* Pass CI env to docker appimage

* Only use git after installing it

* Pass target to appimage build script

* Fix permission after creating appimage

* -_-

* Swap envvar with github ci var

* Format

* Add instruction on how to manually build an AppImage

* Fix typos

* docs: add note about running podman with `--privileged` if there's a permission denied error

* docs: fix typo and link directly to appimage-building `README.md`

* refactor: streamline code and make it a bit cleaner

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2024-01-22 10:52:26 +00:00
Julian Braha
6be79278c2 Optimization: reuse AWS S3 client across procedure calls (#1967)
* Reuse aws s3 client across procedure calls

* Move into locations module
2024-01-22 02:18:56 +00:00
Julian Braha
e312222d83 Use PathBuf instead of String for paths (#1960)
* Use PathBuf instead of String for paths

* Remove unnecessary .into() for clippy
2024-01-20 15:04:48 +00:00
jake
24c2b55f12 [ENG-1546] Delete tag_on_object entries before the tag itself (so tags can be deleted) (#1944)
fix: delete `tag_on_object` entries before the tag itself
2024-01-16 17:58:30 +00:00
Jamie Pine
3f3815e33c [ENG-1511] Library Screens (#1903)
* init

* changes

* Now updating statistics once a minute

* More robust statistics updater

* Concurrency is hard

* improvements to stats

* refactor

* adjust setting back/forward padding so it matches top bar

* refactor sidebar

* rename

* setting up screens

* some changes

* Co-authored-by: Brendan Allan <Brendonovich@users.noreply.github.com>

* yes

* yes2

* refactored explorerItem.ts

* important explorer code shouldn't be thrown away in a util moment

* support for multiple thumbnails in ExplorerItem

* clippy

* move debug

* yes

* label filters

* ts

* comment out unconnected stuff

* added .mid for midi files

---------

Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-01-16 12:15:03 +00:00
Brendan Allan
ef46b2b5ff Use transaction in sync ingest (#1939) 2024-01-16 07:33:13 +00:00
Julian Braha
c4e62225d7 Remove deprecated is_background from JobReport (#1938) 2024-01-15 06:08:23 +00:00
Brendan Allan
8ebd5d5ffa [ENG-1488] Cloud sync message compression (#1922)
* CRDTOperationWithoutInstance

* almost there

* fully compress messages

* implement more of sd-cloud-api

* sd-cloud-api-ify

* landing -_-

* openssl

* clippy

* idk

* bruh

* wut

* ahhh right
2024-01-09 10:21:23 +00:00
Oscar Beaumont
fbac1e6fc5 [ENG-1508] Launch Spacedrop (#1893)
* Shit UI

* refactor a bit

* wip

* yeet

* farming the wry but it's stale

* Real-time hover event

* Hook with `Platform` + fix broken window-state plugin

* `DragAndDropDebug`

* Clippy

* revert Tauri v2 stuff

* minor

* probs not gonna work

* undo last commit

* a

* b

* c

* d

* e

* f

* g

* long shot

* 1

* no 7

* da hell

* large bruh moment

* lol

* zzzz

* SSH into CI

* yeet

* Tauri mouse position without new Wry

* go for gold

* Correctly lock `ort` version

* minor fixes

* debounce hover events

* WTF Tauri

* Replace DND hooks with goated versions

* wip frontend stuff

* fix ts

* disable library p2p stuff

* remove Spacedrop dialog + hook up backend

* `useOnDndLeave` working

* Close popover when drag outside

* Allow `openFilePickerDialog` for Spacedrop

* couple of fixes

* empty state

* smh

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-01-08 06:42:17 +00:00
Oscar Beaumont
c61efdcb67 Hosted Locations - Part 1 (#1861)
* Hosted locations CRD

* Authorise + file upload demo

* Configurable path for testing perms
2024-01-08 06:10:26 +00:00
Oscar Beaumont
cd13e9cf48 Improve ephemeral walker (#1877)
* `non_indexed::walk` as `impl Stream`

* wip

* this is gooooood

* savepoint

* remove

* Batched stream

* `unsafe_streamed_query`

* nightmare nightmare nightmare nightmare nightmare nightmare

* JS Mutex

* cleanup

* proper error handling

* myCode satisfies Typescript

* Move to npm fork of rspc

* fixes

* rspc more crashy crashy

* Typescript is very disappointed
2024-01-02 06:09:44 +00:00
Brendan Allan
21a279cecd [ENG-1512] Replace sync operation types with single type (#1900)
* replace multiple op types with crdt_operation

* cleanup
2023-12-21 05:58:03 +00:00
Ericson "Fogo" Soares
8af3a5f286 [ENG-1479] AI Prototype (#1845)
* First draft on image labeling

* Fixing execution providers for other OSs

* Better error handling and shutdown

* Working with shallow media processor

* bruh

* Fix warnings

* Now hooked to media processor job

* Link desktop app with libonnxruntime to avoid TLS error during startup

* Be able to change models on runtime
Revert to use labels table instead of tags

* A bug on a model-less inference

* Show AI labels on Inspector
 - Change yolo inference to use half precision
 - Add labels api to core

* Remove LD_PRELOAD

* Fix race condition on model executor shutdown

* Don't load all images in memory moron

* Embeed yolo model in prod build
 - Change yolo model path to new one relative to executable

* Disable volume watcher on linux, it was crashing the app
 - Invalidate labels when they are updated

* Rust fmt

* Minor changes

* Gate onnxruntime linking to the ai-models feature

* Add build script to sd-server to handle onnxruntime linking workaround

* Move AI stuff to its own crate and normalize deps

* Rust fmt

* Don't regenerate labels unless asked to

* Now blazingly fast

* Bad merge

* Fix

* Fix

* Add backend logic to download extra yolo models

* Add models api route
 - Add api call to get available model version
 - Add api call to change the model version

* Improve new model download logic
 - Add frontend to change image labeler model

* Fix new model downloader

* Fix model select width

* invalidate labels count after media_processor generates a new output

* Rename AI crate and first draft on download notifications

* fix types

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-12-19 09:28:57 +00:00
Brendan Allan
27f196c947 Basic actor manager (#1888)
* basic declared actor manager

* put actors in separate file

* clippy

* hopefully clean up some clippy warnings

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-12-13 17:49:22 +00:00
Brendan Allan
8feaa85e6d [ENG-1495] Pull new instance data when receiving cloud sync operations (#1882)
* add sd-cloud-api

* consume sd-cloud-api & leave instance writing to receiving

* module-ise api crate

* put back logging

* remove 'join cloud library'
2023-12-12 07:47:21 +00:00
Oscar Beaumont
f82186e174 Upgrade tracing (#1870)
* Upgrade tracing
Thank the lord it's finally here

* Fix excessive logs from mdns_sd & prisma

* fix excessive logs again
2023-12-07 10:22:26 +00:00
Brendan Allan
0acd0c5f13 Fix sync tests (#1866)
fix sync tests
2023-12-06 18:00:57 +00:00
Brendan Allan
554456748e Cloud Sync Basics (#1856)
* cloud operation models

* please clippy

* cloud sync table

* cloud ingest works very simply

* move stuff around + stop invalidation from triggering every 10ms

* cleanup

* fix ts

* remove localhost

* re-enable vendored openssl
2023-12-06 16:47:51 +00:00
Julian Braha
19d474d04b Resolve warnings instead of suppressing them (#1848)
* Eliminate 'unused mut' warning and suppression

* Eliminate 'unreachable code' warning and suppression

* Fix formatting
2023-12-06 06:16:14 +00:00
Oscar Beaumont
c32e60927b [ENG-1400] Normalised caching (#1734)
* prototype

* `.normalise` helper + only `String` keys

* implement it for 'search.paths'

* redux devtools

* fix

* refactor backend

* wip: upgrade to rspc fork

* mega cursed

* Upgrade Specta-related stuff

* Upgrade Typescript

* Cache debug page

* bruh

* Fix optimistic library setting

* Cache clearing

* better timeout

* Fix tags page

* bit of cleanup

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-12-05 13:16:03 +00:00
Brendan Allan
bc29daa60f Cloud connection (#1842)
* actual cloud connection wow

* Preliminary cloud library joining

* remove dev overrides

* add back library caching

* re-enable vendored openssl
2023-11-30 07:39:39 +00:00
jake
cf629931ac [ENG-1426] Apply clippy --fix with moderate lints (#1779)
* chore(clippy): fix

* chore: formatting

* fix: add cfg back to macos crate

* remove cfg to see if clippy works

* Restore macOS crate level conditional compilation

* More warns fixing

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-11-28 17:50:11 +00:00
Utku
e8cd0201d1 Bump versions (#1823)
release
2023-11-25 01:12:00 +00:00
Brendan Allan
d4190aab98 [ENG-1435] Saved Searches (#1810)
* saved search CRUD (not perfect)

* saved search settings page

* minor improvements

* fix search filter text apply

* serach in setting

* reduce new tab flicker

* fix tab delete index

* temporarily remove hover effect from applied filters

* fix types

* fix progress

* fix double-add for inOrNotIn

* no more saved searches settings page

* redirect on saved search delete

* cleaner

* fix filter checkbox double fire

* types

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-11-25 00:16:26 +00:00
jake
ea1e4d748a [ENG-1095] HEIF* and PDF thumbnail orientation (#1815)
* fix horizontal pdf rendering

* don't rotate certain formats (heif*)

* return `u32` from scaling function

* add `avifs` as a valid extension

* add avifs as a valid MIME type

* re-categorize `avifs` as a video (sequence)

* Fix mime-types

* Add Hif extension
 - Cannon uses a special extension for heif in its cameras: https://github.com/digipres/digipres.github.io/blob/master/_sources/registries/fdd/fddXML/fdd000525.xml#L213-L221

* hif is actually a heic

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2023-11-24 22:07:25 +00:00
Ericson "Fogo" Soares
ee17331e3e Concurrency crash for ephemeral thumbs (#1821)
Solve
2023-11-24 16:31:45 +00:00