Commit Graph

22 Commits

Author SHA1 Message Date
Ericson "Fogo" Soares
8147f78a55 [ENG-1793] Introduce error handling for sd-core-sync crate (#2556)
* Error handling for sd-core-sync crate
Also a bunch of tweaks and fixes

* Update core.ts

* Auto format

* Comment out background_processing_percentage on frontend too
 - Update rust version in contributing

* Trying to avoid data loss on actor stop

* Stronger linter on sync generator

* Stronger lint on sync subcrate

* Clean back tasks.json file

* TS typechecking

* Autoformat

* Add more verbose errors for Cloud REST API

* Removing some comments

Copilot is fun sometimes lol

* Properly stopping actors

* Fix ingest stop

* Racing on stop for actors

* Error conversion from merge with main

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Arnab Chakraborty <11457760+Rocky43007@users.noreply.github.com>
2024-07-16 22:30:35 +00:00
Ericson "Fogo" Soares
bdc242a852 [ENG-1775] Scan location using new jobs (#2476)
* First draft on task system usage, still missing job system

* Scan location roughly working, a ton of stuff to fix yet

* Updating some deps due to crashes and bugs

* Exposing non critical errors to frontend

* Getting active job reports from job system

* Using boxed opaque type to avoid a downcast issue with generics

* Task system issues discovered on race conditions

* Enable debug

* Fix job report in the job manager

* Fix race condition on steal tasks

* Fixed race condition on task suspend

* Some fixes on job progress reporting and save

* Fixed many race conditions and a hard deadlock
Also some progress report polishing

* Ignore .ts and .mts video files for now

* Some better logs

* bruh

* Internal deadlocks and excess of communication in the task system
- Also better logs

* Bunch of fixes and optimizations

* WIP at fixing file identifier

* Fixed file identifier job
- still need to work on its progress report frontend

* A bunch of polishing

* Fixed some bugs and did more polishing

* Cleanup

* Bridging old and new job systems

* A ton of fixes

* A bunch of bugs related to shutdown and resume

* Indexer and watcher bugs

* Log normalizing

* Fixing CI

* Change error! to warn! on non critical errors log

* Fix redirect to new location

* Type annotation

* Bogus merge resolution on cargo lock
2024-06-17 21:30:57 +00:00
Brendan Allan
735e80ad4d [ENG-1776] Delete cloud ops after they've been ingested (#2512)
* delete cloud ops after they've been ingested

* give wait_tx to p2p sync ingest

* remove Ingested event

* add sync docs for setting relation fields

* Update core/crates/sync/README.md

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2024-05-31 06:54:22 +00:00
Vítor Vasconcellos
0f405caf24 Fix CI (#2461)
Use native-tls-vendored for reqwest
 - Update prisma-client-rust
 - Fix mock_instance test
 - Clippy + fmt
 - Improve error handling when uninstall i386 libs in setup-system action
2024-05-08 05:04:25 +00:00
Vítor Vasconcellos
2c777e53f1 Fix core test (#2386)
* Fix core test

* Import CompressedCRDTOperations

---------

Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2024-04-24 23:37:38 +00:00
Brendan Allan
a315dd632d ignore update after delete in sync (#2327) 2024-04-17 04:57:45 +00:00
Brendan Allan
40fa3380e5 [ENG-1722] Numeric sync model IDs (#2298)
* numeric sync model ids

* migration

* fix test compilation
2024-04-11 03:46:30 +00:00
Ericson "Fogo" Soares
6b760b0b2a [ENG-1628] Write new indexer with the task system (#2161)
* Moving file-path-helper to a sub-crate on core

* Parallel walker tested and working

* Change inner core crate names to sd-core-* naming scheme

* Moving stuff around

* Save and Update tasks for the new indexer job

* Some initial drafts on the new job system

* More drafts on job system

* Changing walker task to a more explicit state machine
Also drafting more of job system

* More drafting on job resume

* Completed the draft on job system inner workings

* New job context abstraction to decouple library stuff from job system

* Properly use composition on task dispatcher

* First draft on indexer job

* Job serialization

* Handling ancestors in the distributed walker for indexer

* Saving computed directories sizes on a location to db

* Enable a way to check if jobs are running in a location

* Progress reporting on indexer job

* Reorganizing modules

* Shallow indexer

* Rust fmt

* Attempting windows CI fix

* Attempting windows CI fix again

* Attempting windows CI fix again
2024-04-08 14:32:44 +00:00
Brendan Allan
902ed24ffd [ENG-1674] fix request drop test and add write_ops test (#2252)
fix request drop test and add write_ops test
2024-03-27 06:40:57 +00:00
Vítor Vasconcellos
16354b0f72 Fix a couple of tests after #2218 (#2219)
- Fix some rust warnings
2024-03-19 02:23:42 +00:00
Brendan Allan
79530f1e4e Fix sync tests (#2194)
* Fix sync tests

* fix build
2024-03-13 05:34:14 +00:00
Vítor Vasconcellos
6a32752243 Fix core test and CI breaking (#2116)
Fix core test passing inverted arguments to sync_db_entry macro
2024-02-22 06:15:36 +00:00
Brendan Allan
c533d12df0 media data sync (#2102)
* basic sync operation backfill

* media data sync

* sync entry helpers

* fix sync generator

* nicer

* re-add key_id
2024-02-21 11:42:10 +00:00
Brendan Allan
4962b1160b [ENG-1488] Cloud sync message compression (#1922)
* CRDTOperationWithoutInstance

* almost there

* fully compress messages

* implement more of sd-cloud-api

* sd-cloud-api-ify

* landing -_-

* openssl

* clippy

* idk

* bruh

* wut

* ahhh right
2024-01-09 10:21:23 +00:00
Brendan Allan
4fb9028452 Cloud Sync Basics (#1856)
* cloud operation models

* please clippy

* cloud sync table

* cloud ingest works very simply

* move stuff around + stop invalidation from triggering every 10ms

* cleanup

* fix ts

* remove localhost

* re-enable vendored openssl
2023-12-06 16:47:51 +00:00
jake
c79c190c78 [ENG-1426] Apply clippy --fix with moderate lints (#1779)
* chore(clippy): fix

* chore: formatting

* fix: add cfg back to macos crate

* remove cfg to see if clippy works

* Restore macOS crate level conditional compilation

* More warns fixing

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-11-28 17:50:11 +00:00
Vítor Vasconcellos
cb6a356287 [ENG-1027] Fix build on Windows (#1271)
* Fix Vite relativeAliasResolver on Windows
 - Add Strawberry Perl installation to setup-system.ps1 (stop rust-analyzer from crashing on Windows)
 - Remove unused imports from desktop windows crate
 - Add todo for missing sync test match cases

* Use path.join instead of doing string concatenation with path.sep by hand
2023-08-30 03:26:13 +00:00
Oscar Beaumont
d758977d82 Improve Sync + P2P Integration (#1265)
* Big bruh moment

* whoops

* Less stackoverflowy debug

* stuff

* Fix flawed P2P mDNS instance advertisements

* do sync when connecting with peer

* Sync after pairing

* resync_part2 all the time

* Invalidate all the things

* Invalidate whole React Query on sync event

* emit_messages_flag

* emit_messages_flag

* Backend feature flags + "emitSyncEvents" feature

* Patch `confirm` type cause Tauri cringe

* clippy

* idk but plz work

* bruh

* Fix ComLink bug

* remove log

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-29 17:54:58 +00:00
Ericson "Fogo" Soares
4a3acfa1c8 [ENG-835] Duplicating an empty directory doesn't duplicate nested empty directories (#1213)
* Bunch of warnings

* Solving the issue

* sd_core_sync::New

* dumb

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-13 18:34:46 +00:00
Brendan Allan
8f1afbf164 Isolate sync p2p logic (#1198)
* remove p2p from sd-core-sync and isolate p2p handling

* re-enable tests and stop being silly

* move sync protocol to modules

* don't emit messages

* cleanup

* formatting
2023-08-11 08:32:20 +00:00
Brendan Allan
00e4aa9e8f Sync ingesting (#1181)
* sync + ingest refactor

* fix Event enum description

* actually do sync over network

* re-enable heif

* remove comment
2023-08-07 07:52:43 +00:00
Oscar Beaumont
6c9e08540f Networked sync (#1133)
* temp remove discriminator from tunnel

* i'm lost :(

* Working `Tunnel` + move sync stuff back into p2p

* sync module

* `NetworkedSyncManager`

* Move sync stuff off `P2PManager`

* Test `SyncMessage`

* Library edit + delete hooks

* stable `identity` column

* fix

* plz work

* p2p is hard

* sync identities post pairing

* Remove `p384`

* a couple improvements

* comment out ReqRes

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-02 11:49:50 +00:00