Commit Graph

19 Commits

Author SHA1 Message Date
Brendan Allan
b7872421f4 Use transaction in sync ingest (#1939) 2024-01-16 07:33:13 +00:00
Brendan Allan
4962b1160b [ENG-1488] Cloud sync message compression (#1922)
* CRDTOperationWithoutInstance

* almost there

* fully compress messages

* implement more of sd-cloud-api

* sd-cloud-api-ify

* landing -_-

* openssl

* clippy

* idk

* bruh

* wut

* ahhh right
2024-01-09 10:21:23 +00:00
Brendan Allan
5371e9f961 [ENG-1512] Replace sync operation types with single type (#1900)
* replace multiple op types with crdt_operation

* cleanup
2023-12-21 05:58:03 +00:00
Ericson "Fogo" Soares
7c90bcb95b [ENG-1479] AI Prototype (#1845)
* First draft on image labeling

* Fixing execution providers for other OSs

* Better error handling and shutdown

* Working with shallow media processor

* bruh

* Fix warnings

* Now hooked to media processor job

* Link desktop app with libonnxruntime to avoid TLS error during startup

* Be able to change models on runtime
Revert to use labels table instead of tags

* A bug on a model-less inference

* Show AI labels on Inspector
 - Change yolo inference to use half precision
 - Add labels api to core

* Remove LD_PRELOAD

* Fix race condition on model executor shutdown

* Don't load all images in memory moron

* Embeed yolo model in prod build
 - Change yolo model path to new one relative to executable

* Disable volume watcher on linux, it was crashing the app
 - Invalidate labels when they are updated

* Rust fmt

* Minor changes

* Gate onnxruntime linking to the ai-models feature

* Add build script to sd-server to handle onnxruntime linking workaround

* Move AI stuff to its own crate and normalize deps

* Rust fmt

* Don't regenerate labels unless asked to

* Now blazingly fast

* Bad merge

* Fix

* Fix

* Add backend logic to download extra yolo models

* Add models api route
 - Add api call to get available model version
 - Add api call to change the model version

* Improve new model download logic
 - Add frontend to change image labeler model

* Fix new model downloader

* Fix model select width

* invalidate labels count after media_processor generates a new output

* Rename AI crate and first draft on download notifications

* fix types

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-12-19 09:28:57 +00:00
Brendan Allan
3cabc9c3a9 Basic actor manager (#1888)
* basic declared actor manager

* put actors in separate file

* clippy

* hopefully clean up some clippy warnings

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-12-13 17:49:22 +00:00
Brendan Allan
3abdb81e4b Fix sync tests (#1866)
fix sync tests
2023-12-06 18:00:57 +00:00
Brendan Allan
4fb9028452 Cloud Sync Basics (#1856)
* cloud operation models

* please clippy

* cloud sync table

* cloud ingest works very simply

* move stuff around + stop invalidation from triggering every 10ms

* cleanup

* fix ts

* remove localhost

* re-enable vendored openssl
2023-12-06 16:47:51 +00:00
jake
c79c190c78 [ENG-1426] Apply clippy --fix with moderate lints (#1779)
* chore(clippy): fix

* chore: formatting

* fix: add cfg back to macos crate

* remove cfg to see if clippy works

* Restore macOS crate level conditional compilation

* More warns fixing

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-11-28 17:50:11 +00:00
Julian Braha
bf543f45d0 Remove unused dependencies in src-tauri and p2p (#1726)
* Remove unused dependency: window-shadows

* Remove unused dependencies: rmp-serde and arc-swap

* Update Cargo.lock

* remove all unused dependencies

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2023-11-04 04:56:53 +00:00
Oscar Beaumont
94d8f61de9 Update Rust dependencies (#1314)
* Update most of Rust dependencies

* downgrade notify

* a

* fix Linux?

* Android is painful

* update build.gradle

* The lord Clippy has spoken

* Bruh

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2023-09-12 14:03:25 +00:00
Vítor Vasconcellos
cb6a356287 [ENG-1027] Fix build on Windows (#1271)
* Fix Vite relativeAliasResolver on Windows
 - Add Strawberry Perl installation to setup-system.ps1 (stop rust-analyzer from crashing on Windows)
 - Remove unused imports from desktop windows crate
 - Add todo for missing sync test match cases

* Use path.join instead of doing string concatenation with path.sep by hand
2023-08-30 03:26:13 +00:00
Oscar Beaumont
d758977d82 Improve Sync + P2P Integration (#1265)
* Big bruh moment

* whoops

* Less stackoverflowy debug

* stuff

* Fix flawed P2P mDNS instance advertisements

* do sync when connecting with peer

* Sync after pairing

* resync_part2 all the time

* Invalidate all the things

* Invalidate whole React Query on sync event

* emit_messages_flag

* emit_messages_flag

* Backend feature flags + "emitSyncEvents" feature

* Patch `confirm` type cause Tauri cringe

* clippy

* idk but plz work

* bruh

* Fix ComLink bug

* remove log

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-29 17:54:58 +00:00
Ericson "Fogo" Soares
4a3acfa1c8 [ENG-835] Duplicating an empty directory doesn't duplicate nested empty directories (#1213)
* Bunch of warnings

* Solving the issue

* sd_core_sync::New

* dumb

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-13 18:34:46 +00:00
Brendan Allan
8f1afbf164 Isolate sync p2p logic (#1198)
* remove p2p from sd-core-sync and isolate p2p handling

* re-enable tests and stop being silly

* move sync protocol to modules

* don't emit messages

* cleanup

* formatting
2023-08-11 08:32:20 +00:00
Oscar Beaumont
ae3d4c1c37 Library "states" continuation (#1180)
* Remove `NodeServices`

* minor

* fix

* cleanup services

* do the shuffle

* `LoadedLibrary` -> `Library`
Yeah this is probs better

* fixes

* bruh

* lol

* get_all_libraries -> get_all

* job::manager::Actor

* `ctx` to `node` for consistency

* _

* cleanup

* do arc higher up

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-07 09:26:10 +00:00
Brendan Allan
00e4aa9e8f Sync ingesting (#1181)
* sync + ingest refactor

* fix Event enum description

* actually do sync over network

* re-enable heif

* remove comment
2023-08-07 07:52:43 +00:00
Oscar Beaumont
9e919f9909 Library "state" (#1160)
* `Library` -> `LoadedLibrary`

* Move `ThumbnailRemoverActor` onto `NodeServices`
It should either be on `Library` or `NodeServices` none of this in-between stuff.

* small > big files

* prune imports

* isolate cringe

* it helps to test the code

* `Vec` -> `HashMap` - faster lookups

* mpscrr

* minor fixes

* fix language

* `!Sync` :(

* lets goooooo

* removed a comma

* fixes

* fix desktop

* lol, no one uses it

* Clippy
2023-08-05 14:42:21 +00:00
Oscar Beaumont
6c9e08540f Networked sync (#1133)
* temp remove discriminator from tunnel

* i'm lost :(

* Working `Tunnel` + move sync stuff back into p2p

* sync module

* `NetworkedSyncManager`

* Move sync stuff off `P2PManager`

* Test `SyncMessage`

* Library edit + delete hooks

* stable `identity` column

* fix

* plz work

* p2p is hard

* sync identities post pairing

* Remove `p384`

* a couple improvements

* comment out ReqRes

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-02 11:49:50 +00:00
Brendan Allan
043b607ad4 CRDTOperation receiving (#1122)
* operation receive + compare

* cleanup + deduplication

* operation receive + compare

* cleanup + deduplication

* sync route + operation grouping

* tag assign sync

* proper relation support in sync debug page

* migration

* separate core-sync + utils crates

* separate p2p event loop from manager

* cleanup library handling

* clippy

* feature gate sync messages properly

* make migration not add required field
2023-07-24 15:26:00 +00:00