Commit Graph

22 Commits

Author SHA1 Message Date
Ericson "Fogo" Soares
2d19fad2eb [ENG-1793] Introduce error handling for sd-core-sync crate (#2556)
* Error handling for sd-core-sync crate
Also a bunch of tweaks and fixes

* Update core.ts

* Auto format

* Comment out background_processing_percentage on frontend too
 - Update rust version in contributing

* Trying to avoid data loss on actor stop

* Stronger linter on sync generator

* Stronger lint on sync subcrate

* Clean back tasks.json file

* TS typechecking

* Autoformat

* Add more verbose errors for Cloud REST API

* Removing some comments

Copilot is fun sometimes lol

* Properly stopping actors

* Fix ingest stop

* Racing on stop for actors

* Error conversion from merge with main

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Arnab Chakraborty <11457760+Rocky43007@users.noreply.github.com>
2024-07-16 22:30:35 +00:00
Ericson "Fogo" Soares
c0d736453e [ENG-1775] Scan location using new jobs (#2476)
* First draft on task system usage, still missing job system

* Scan location roughly working, a ton of stuff to fix yet

* Updating some deps due to crashes and bugs

* Exposing non critical errors to frontend

* Getting active job reports from job system

* Using boxed opaque type to avoid a downcast issue with generics

* Task system issues discovered on race conditions

* Enable debug

* Fix job report in the job manager

* Fix race condition on steal tasks

* Fixed race condition on task suspend

* Some fixes on job progress reporting and save

* Fixed many race conditions and a hard deadlock
Also some progress report polishing

* Ignore .ts and .mts video files for now

* Some better logs

* bruh

* Internal deadlocks and excess of communication in the task system
- Also better logs

* Bunch of fixes and optimizations

* WIP at fixing file identifier

* Fixed file identifier job
- still need to work on its progress report frontend

* A bunch of polishing

* Fixed some bugs and did more polishing

* Cleanup

* Bridging old and new job systems

* A ton of fixes

* A bunch of bugs related to shutdown and resume

* Indexer and watcher bugs

* Log normalizing

* Fixing CI

* Change error! to warn! on non critical errors log

* Fix redirect to new location

* Type annotation

* Bogus merge resolution on cargo lock
2024-06-17 21:30:57 +00:00
Brendan Allan
ded0779718 [ENG-1776] Delete cloud ops after they've been ingested (#2512)
* delete cloud ops after they've been ingested

* give wait_tx to p2p sync ingest

* remove Ingested event

* add sync docs for setting relation fields

* Update core/crates/sync/README.md

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2024-05-31 06:54:22 +00:00
Vítor Vasconcellos
d4f74aecd0 Fix CI (#2461)
Use native-tls-vendored for reqwest
 - Update prisma-client-rust
 - Fix mock_instance test
 - Clippy + fmt
 - Improve error handling when uninstall i386 libs in setup-system action
2024-05-08 05:04:25 +00:00
Vítor Vasconcellos
d7827f50a7 Fix core test (#2386)
* Fix core test

* Import CompressedCRDTOperations

---------

Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2024-04-24 23:37:38 +00:00
Brendan Allan
e1561bb394 ignore update after delete in sync (#2327) 2024-04-17 04:57:45 +00:00
Brendan Allan
e0401a1910 [ENG-1722] Numeric sync model IDs (#2298)
* numeric sync model ids

* migration

* fix test compilation
2024-04-11 03:46:30 +00:00
Ericson "Fogo" Soares
b2bee3d223 [ENG-1628] Write new indexer with the task system (#2161)
* Moving file-path-helper to a sub-crate on core

* Parallel walker tested and working

* Change inner core crate names to sd-core-* naming scheme

* Moving stuff around

* Save and Update tasks for the new indexer job

* Some initial drafts on the new job system

* More drafts on job system

* Changing walker task to a more explicit state machine
Also drafting more of job system

* More drafting on job resume

* Completed the draft on job system inner workings

* New job context abstraction to decouple library stuff from job system

* Properly use composition on task dispatcher

* First draft on indexer job

* Job serialization

* Handling ancestors in the distributed walker for indexer

* Saving computed directories sizes on a location to db

* Enable a way to check if jobs are running in a location

* Progress reporting on indexer job

* Reorganizing modules

* Shallow indexer

* Rust fmt

* Attempting windows CI fix

* Attempting windows CI fix again

* Attempting windows CI fix again
2024-04-08 14:32:44 +00:00
Brendan Allan
1609310d6d [ENG-1674] fix request drop test and add write_ops test (#2252)
fix request drop test and add write_ops test
2024-03-27 06:40:57 +00:00
Vítor Vasconcellos
d6f899a93d Fix a couple of tests after #2218 (#2219)
- Fix some rust warnings
2024-03-19 02:23:42 +00:00
Brendan Allan
95786c1cbb Fix sync tests (#2194)
* Fix sync tests

* fix build
2024-03-13 05:34:14 +00:00
Vítor Vasconcellos
f4e68497ed Fix core test and CI breaking (#2116)
Fix core test passing inverted arguments to sync_db_entry macro
2024-02-22 06:15:36 +00:00
Brendan Allan
bfd6e813c0 media data sync (#2102)
* basic sync operation backfill

* media data sync

* sync entry helpers

* fix sync generator

* nicer

* re-add key_id
2024-02-21 11:42:10 +00:00
Brendan Allan
8ebd5d5ffa [ENG-1488] Cloud sync message compression (#1922)
* CRDTOperationWithoutInstance

* almost there

* fully compress messages

* implement more of sd-cloud-api

* sd-cloud-api-ify

* landing -_-

* openssl

* clippy

* idk

* bruh

* wut

* ahhh right
2024-01-09 10:21:23 +00:00
Brendan Allan
554456748e Cloud Sync Basics (#1856)
* cloud operation models

* please clippy

* cloud sync table

* cloud ingest works very simply

* move stuff around + stop invalidation from triggering every 10ms

* cleanup

* fix ts

* remove localhost

* re-enable vendored openssl
2023-12-06 16:47:51 +00:00
jake
cf629931ac [ENG-1426] Apply clippy --fix with moderate lints (#1779)
* chore(clippy): fix

* chore: formatting

* fix: add cfg back to macos crate

* remove cfg to see if clippy works

* Restore macOS crate level conditional compilation

* More warns fixing

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-11-28 17:50:11 +00:00
Vítor Vasconcellos
952fcd8c36 [ENG-1027] Fix build on Windows (#1271)
* Fix Vite relativeAliasResolver on Windows
 - Add Strawberry Perl installation to setup-system.ps1 (stop rust-analyzer from crashing on Windows)
 - Remove unused imports from desktop windows crate
 - Add todo for missing sync test match cases

* Use path.join instead of doing string concatenation with path.sep by hand
2023-08-30 03:26:13 +00:00
Oscar Beaumont
c50741dd79 Improve Sync + P2P Integration (#1265)
* Big bruh moment

* whoops

* Less stackoverflowy debug

* stuff

* Fix flawed P2P mDNS instance advertisements

* do sync when connecting with peer

* Sync after pairing

* resync_part2 all the time

* Invalidate all the things

* Invalidate whole React Query on sync event

* emit_messages_flag

* emit_messages_flag

* Backend feature flags + "emitSyncEvents" feature

* Patch `confirm` type cause Tauri cringe

* clippy

* idk but plz work

* bruh

* Fix ComLink bug

* remove log

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-29 17:54:58 +00:00
Ericson "Fogo" Soares
3f468d8950 [ENG-835] Duplicating an empty directory doesn't duplicate nested empty directories (#1213)
* Bunch of warnings

* Solving the issue

* sd_core_sync::New

* dumb

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-13 18:34:46 +00:00
Brendan Allan
262a0dd922 Isolate sync p2p logic (#1198)
* remove p2p from sd-core-sync and isolate p2p handling

* re-enable tests and stop being silly

* move sync protocol to modules

* don't emit messages

* cleanup

* formatting
2023-08-11 08:32:20 +00:00
Brendan Allan
1eb3b0e55a Sync ingesting (#1181)
* sync + ingest refactor

* fix Event enum description

* actually do sync over network

* re-enable heif

* remove comment
2023-08-07 07:52:43 +00:00
Oscar Beaumont
3521b4e25d Networked sync (#1133)
* temp remove discriminator from tunnel

* i'm lost :(

* Working `Tunnel` + move sync stuff back into p2p

* sync module

* `NetworkedSyncManager`

* Move sync stuff off `P2PManager`

* Test `SyncMessage`

* Library edit + delete hooks

* stable `identity` column

* fix

* plz work

* p2p is hard

* sync identities post pairing

* Remove `p384`

* a couple improvements

* comment out ReqRes

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-02 11:49:50 +00:00