Commit Graph

23 Commits

Author SHA1 Message Date
Vítor Vasconcellos
01d61631d7 Merge remote-tracking branch 'origin/main' into eng-1828-migration-to-new-cloud-api-system 2024-08-15 18:16:13 -03:00
Vítor Vasconcellos
6672c6cffc Remove unused code and update rust deps (#2662)
* Remove deps-generator app
 - It is broken for quite a while and not being used

* Remove unused code and unused deps

* auto format

* Restore incorrectly removed files
 - Remove aws deps and test code using it

* Replace winapi-util with windows-rs
 - Adjust some deps versions

* Autoformat

* Update rand, remove unused dep

* Fix image crate due to changes in pdf-renderer
 - Dont use default features in reqwest
 - Use native-tls tauri

* Remove incorrect notice about blake3
2024-08-15 17:09:03 +00:00
Arnab Chakraborty
139ba6e1a4 [ENG-1854] Migrate to Tauri v2 Release Candidate (#2654)
* Migrate to Tauri v2 Release Canidate

* Formatting

* Update pnpm

* Update all tauri deps (js and rust) to Release Candidate 2
 - A lot of misc fixes due to changes in the new versions of tauri, rspc and specta

* Recreate pnpm-lock

* Fix pnpm-lock

* Formatting

* Use 10 chars SHA for git deps
 - Update a couple of rust dependencies
 - Add taplo for toml auto formatting
 - Adjust some formatting configurations
 - Updated prettier and its plugins

* Fix formatter settings

* Minor improvements to Cargo.toml format

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-08-14 18:22:46 +00:00
Arnab Chakraborty
019b157d9f Merge branch 'rocky/tauri-v2-rc' into eng-1828-migration-to-new-cloud-api-system 2024-08-13 13:51:52 +03:00
Vítor Vasconcellos
8b3574d7a6 Use 10 chars SHA for git deps
- Update a couple of rust dependencies
 - Add taplo for toml auto formatting
 - Adjust some formatting configurations
 - Updated prettier and its plugins
2024-08-12 17:49:31 -03:00
Arnab Chakraborty
46a400880b Migrate all cloud routes to the new API system
All cloud routes are now migrated to the new system in the backend.

The old routes are still available, as the frontend has not been modified to use the new ones yet.
2024-08-07 17:55:22 +03:00
Ericson "Fogo" Soares
8147f78a55 [ENG-1793] Introduce error handling for sd-core-sync crate (#2556)
* Error handling for sd-core-sync crate
Also a bunch of tweaks and fixes

* Update core.ts

* Auto format

* Comment out background_processing_percentage on frontend too
 - Update rust version in contributing

* Trying to avoid data loss on actor stop

* Stronger linter on sync generator

* Stronger lint on sync subcrate

* Clean back tasks.json file

* TS typechecking

* Autoformat

* Add more verbose errors for Cloud REST API

* Removing some comments

Copilot is fun sometimes lol

* Properly stopping actors

* Fix ingest stop

* Racing on stop for actors

* Error conversion from merge with main

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Arnab Chakraborty <11457760+Rocky43007@users.noreply.github.com>
2024-07-16 22:30:35 +00:00
Vítor Vasconcellos
405fb023d8 Clean up and update rust dependencies (#2544)
* Clean up and update rust dependencies

* Attempt to fix Windows Error
2024-06-10 19:44:00 +00:00
Oscar Beaumont
b015763a6f More P2P docs (#2492)
* Remove relay

* restructure p2p

* wip

* cleanup webrtc

* split up P2P docs

* wip

* more wip

* the fork has moved

* finish local network discovery

* Document the relay system

* be less stupid

* a

* remote ip from deploy script

* remove debug from deploy script

* Explain relay setup and usage

* Physical pain

* fix

* error handling for relay setup

* Listeners Relay state + merge it into NLM state

* `node_remote_identity`

* redo libraries hook

* toggle relay active in settings

* Dedicated network settings page

* Stablise P2P debug page

* warning for rspc remote

* Linear links in docs

* fix p2p settings switches

* fix typescript errors on general page

* fix ipv6 listener status

* discovery method in UI

* Remove p2p debug menu on the sidebar

* wip

* lol

* wat

* fix

* another attempt at fixing library hook

* fix

* Remove sync from sidebar

* fix load library code

* I hate this

* Detect connections over the relay

* fix

* fixes

* a

* fix mDNS

* a bunch o fixes

* a bunch of state management fixes

* Metadata sync on connection

* skill issue

* fix markdown

* Clippy cleanup

* Backport #2380

* Update interface/locales/en/common.json

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/local-network-discovery.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/local-network-discovery.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/overview.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/overview.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/transport-layer.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/local-network-discovery.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* a

* Cleaning binario section

* cleanup Docker message

* idk

* Idempotent listeners

* Manual peers working????

* minor fixes

* crazy idea - don't panic in the event loop

* fixes

* debug

* debug

* LAN badge in network settings

* Use `dns_lookup` instead of `tokio::net::lookup_host`

* fix

* bruh sandwich

* proper dialing

* a

* remove logs

* fix

* Small cleanup

* manual peers state on connected device

* a

* Fix manual discovery state + give it a badge

* Clippy improvements

* flip discovery priority

* Add `addrs` to debug query

* connection candidates in debug

* Fix state

* Clippppppppppppy

* Manual discovery badge

* Flesh out ping example

* Usage guide

* `sd_p2p_proto` examples

* More discovery docs

* More docs work

* docs docs docs and more docs

* PONG

* rename

---------

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>
2024-05-30 21:48:12 +08:00
Vítor Vasconcellos
0f405caf24 Fix CI (#2461)
Use native-tls-vendored for reqwest
 - Update prisma-client-rust
 - Fix mock_instance test
 - Clippy + fmt
 - Improve error handling when uninstall i386 libs in setup-system action
2024-05-08 05:04:25 +00:00
Ericson "Fogo" Soares
6b760b0b2a [ENG-1628] Write new indexer with the task system (#2161)
* Moving file-path-helper to a sub-crate on core

* Parallel walker tested and working

* Change inner core crate names to sd-core-* naming scheme

* Moving stuff around

* Save and Update tasks for the new indexer job

* Some initial drafts on the new job system

* More drafts on job system

* Changing walker task to a more explicit state machine
Also drafting more of job system

* More drafting on job resume

* Completed the draft on job system inner workings

* New job context abstraction to decouple library stuff from job system

* Properly use composition on task dispatcher

* First draft on indexer job

* Job serialization

* Handling ancestors in the distributed walker for indexer

* Saving computed directories sizes on a location to db

* Enable a way to check if jobs are running in a location

* Progress reporting on indexer job

* Reorganizing modules

* Shallow indexer

* Rust fmt

* Attempting windows CI fix

* Attempting windows CI fix again

* Attempting windows CI fix again
2024-04-08 14:32:44 +00:00
Brendan Allan
3afc3bd34f fix cloud sync and operations (#2218)
* fix cloud sync and operations

* remove some logs

* cleanup
2024-03-18 08:51:16 +00:00
Oscar Beaumont
7dc0880f8b P2P Metadata Cache + Sidebar (#2208)
* wip

* wip

* Render peers in sidebar

* clippy

* stringify on server

* don't stringify

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-03-14 09:12:29 +00:00
Oscar Beaumont
27c6570c2f Rename P2P crate (#2207)
`sd_p2p2` to `sd_p2p`
2024-03-14 04:38:31 +00:00
Brendan Allan
63b17adb10 switch sync to msgpack instead of json (#2164)
* switch sync to msgpack instead of json

* use rmpv

* more rmpv

* use msgpack in cloud

* receive operations as msgpack
2024-03-06 10:15:29 +00:00
Ericson "Fogo" Soares
68ca2382fd [ENG-1627] Move all jobs and actors to an "Old" namespace (#2157)
* Bunch of new warns due to a stronger clippy config

* Moving old processing stuff to a new namespace
Also fixing a bunch of typos through the entire codebase.

* Rustfmt
2024-03-04 18:57:35 +00:00
Brendan Allan
54af5a822f fix doAdd cloud api and provide operations count (#2154) 2024-03-01 10:41:47 +00:00
Oscar Beaumont
519b1b6c46 Fix P2P not working for libraries (#2031)
* P2P Debug route

* Remove legacy peer to peer pairing process

* Fix error typo

* Sync instances with cloud

* Upgrade deps + extended instance data

* Create instance with extended metadata

* Auto sync instances

* Actually `.await`

* bruh

* sync library info

* this isn't gonna work

* only sleep cloud receiver when no more messages (#1985)

* [ENG-1567] Fix renaming (#1986)

fix rename

* only sleep cloud receiver when no more messages

* use in memory instances during cloud receive (#1995)

* use in memory instances during cloud receive

* is_empty

---------

Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>

* fix type error

* wip

* make mdns mdns better

* rebuild state

* Add hooks + listeners + discovered state

* Split into crates

* wip fixing core + wip merging Spacetime into `sd-p2p2`

* `SmartLockGuard` + `Listener`

* Make `sd-core` compile

* Reenable all operation receivers

* Fix all broken code within `sd-core`

* minor fixes found in review

* Bring in `libp2p` + restructure `sd-p2p` for the gazillion-th time

* whoops

* Compile no matter the (runtime) cost

* fixing merge issues

* wip

* a

* b

* C

* Handle port betterer

* c

* Migrate node config

* a

* no crash on startup

* wip

* a

* jdfhskjfsg

* a

* fix discovery

* a bunch of fixes

* getting Spacedrop working

* I don't get why it no worky

* debug example

* a

* wip

* wip

* removing logging from stream impl

* wip: shit is fucked

* Redo quic integration  + Spacedrop working

* Fix shutdown - deadlocks + shutdown peers

* Add Prisma migrations

* Fix shutdown

* a

* fix

* cleanup

* The lord clippy hath spoken

* disable P2P settings for now

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>
2024-02-21 08:13:40 +00:00
Oscar Beaumont
a30deaa144 Cloud + P2P (#1970)
* P2P Debug route

* Remove legacy peer to peer pairing process

* Fix error typo

* Sync instances with cloud

* Upgrade deps + extended instance data

* Create instance with extended metadata

* Auto sync instances

* Actually `.await`

* bruh

* sync library info

* this isn't gonna work

* only sleep cloud receiver when no more messages (#1985)

* [ENG-1567] Fix renaming (#1986)

fix rename

* only sleep cloud receiver when no more messages

* use in memory instances during cloud receive (#1995)

* use in memory instances during cloud receive

* is_empty

---------

Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>

* fix type error

* wip

* make mdns mdns better

* response

* remove renames

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>
2024-02-02 09:31:11 +00:00
Brendan Allan
c45f6d7bfa separate cloud sync into sd-core-cloud-sync (#2025)
* separate cloud sync into sd-core-cloud-sync

* ci
2024-01-31 08:16:54 +00:00
Brendan Allan
4962b1160b [ENG-1488] Cloud sync message compression (#1922)
* CRDTOperationWithoutInstance

* almost there

* fully compress messages

* implement more of sd-cloud-api

* sd-cloud-api-ify

* landing -_-

* openssl

* clippy

* idk

* bruh

* wut

* ahhh right
2024-01-09 10:21:23 +00:00
Brendan Allan
3cabc9c3a9 Basic actor manager (#1888)
* basic declared actor manager

* put actors in separate file

* clippy

* hopefully clean up some clippy warnings

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-12-13 17:49:22 +00:00
Brendan Allan
a9da6a0093 [ENG-1495] Pull new instance data when receiving cloud sync operations (#1882)
* add sd-cloud-api

* consume sd-cloud-api & leave instance writing to receiving

* module-ise api crate

* put back logging

* remove 'join cloud library'
2023-12-12 07:47:21 +00:00