Commit Graph

354 Commits

Author SHA1 Message Date
Oscar Beaumont
54f757bf1f Job system improvements v2 (#707)
* Add `JobInitData` trait + change `StatefulJob::name` to a constant

* `StatefulJob::new`

* Move `invalidate_query` into `StatefulJob::finalize`

* Cleanup `spawn_job` signature

* Remove `queue_job` and run jobs from the `StatefulJob::finalize` method

* `StatefulJob::queue_jobs`

* `JobManager::ingest` return `Result`

* Remove `jobs.isRunning`

* Invalidation system direct push + batching invalidate

* Look ma, only a single clippy warning!!!!

* Error handling for JobManager

* Rust fmt

* Introducing Job hierarchy to enable job enqueuing

* Rust fmt again 🙄

* core.ts

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-04-15 14:43:04 +08:00
Brendan Allan
3f3a5be557 Revert "Give FilePath a pub_id (#691)"
This reverts commit aef277af4e.
2023-04-14 10:50:37 +08:00
jake
d8ea2c172c update to ffmpeg 6.0.0 (#697) 2023-04-13 14:10:23 +00:00
Brendan Allan
aef277af4e Give FilePath a pub_id (#691)
* checkpoint - indexer parent ids need fixing

* use pub_id for file paths

* Removing LastFilePathIdManager and fixing some loose ends

---------

Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2023-04-12 21:58:23 -07:00
Ericson "Fogo" Soares
0359fd59d1 ENG-462 Object size is 0 when it shouldn't be (#702)
* Fixing file size not being collected from system

* Rust fmt
2023-04-13 01:06:27 +00:00
Vítor Vasconcellos
bee9c0b350 ENG-460 Location creation fails for some writable directories (#696)
Remove read-only check when creating a location
- Revert location creation if writing the `.spacedrive` file or updating the library location manager fails
2023-04-12 12:51:32 +00:00
nikec
096741a8fd [Desktop] Renamable files (#675)
* Add rename file option

* make Ericson's changes

* Prevent arrow selecting when renaming, start renaming with Enter option

* fix rename

---------

Co-authored-by: Jamie Pine <32987599+jamiepine@users.noreply.github.com>
Co-authored-by: Jamie Pine <ijamespine@me.com>
2023-04-11 20:43:19 -07:00
Oscar Beaumont
5bc234ae87 [ENG-414] P2P Shutdown (#688)
* `Event::Shutdown` in P2P system

* Mdns shutdown broadcast + await shutdown

* Fix mdns service name & prevent poll after shutdown
2023-04-11 15:05:50 +00:00
Oscar Beaumont
b1d619c42f Add MetadataManager into sd-p2p (#686)
* Add `MetadataManager` into sd-p2p

* Add `update_metadata` helper to P2P manager

* Praise Clippy our lord and saviour

* change mdns advertisement log level

* Casually broke Linux
2023-04-11 10:20:14 +08:00
Vítor Vasconcellos
b8eba88e90 QuickPreview Component (Needs test on MacOS) (#665)
* Add QuickPreview Component
 - Improve the handling of Range requests
 - Implement logic to answer HEAD and OPTIONS methods
 - Handle CORS pre-flight requests
 - Expand accepted file types
 - Improve error handling of invalid Range requests

* Fix linter errors
 - Add `use std::cmp::min` to custom_uri (Required on MacOS & Windows)
 - Improve logic for retrieving file information in QuickPreview.tsx

* More linter errors

* Simplify `QuickPreview` by extracting the logic for choosing the file preview tag to a `FilePreview` component
 - Fix the typo in `QuickPreview` props name
 - Remove the unused `handleMedia` ref
 - Move the remaining `QuickPreview` logic to the `transitions` callback
 - Simplify the `cors` return type in `custom_uri.rs`

* Refactor range handling in `handle_file` function
 - Move range handling logic to the initialization of the `range` variable
 - Replace `if let` with `match` to reduce code duplication
 - Don't export FilePreview
 - Export QuickPreviewProps

* Fix typo in `RangeNotSatisfiable` error message
 - Remove redundant variables

* Fixing cas_id generation on watcher
Some improvements on watcher file creation

* Rust fmt

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: Jamie Pine <ijamespine@me.com>
2023-04-05 21:15:13 -07:00
Brendan Allan
5d1c8a3017 fix syncing (#679) 2023-04-05 19:47:10 -07:00
Vítor Vasconcellos
a2fcdd68cf Fix cas_id sample hashing logic (#672) 2023-04-04 13:59:19 +00:00
Brendan Allan
5575575875 fix core 2023-04-04 15:12:09 +08:00
Brendan Allan
98aa82b30e some more sync impls (#669)
a lil more sync impls
2023-04-03 22:45:00 -07:00
Brendan Allan
3792cf1034 fix ts (#661)
* fix ts

* eslint
2023-03-31 00:22:54 -07:00
Ericson "Fogo" Soares
2609f69282 Eng 372 location awareness improvements (#612)
* Complying with a pedantic Clippy

* Solving duplicated directories events
When creating a directory through MacOS's Finder, for some reason
FSEvents receives 2 Create Folder events that we have to handle

* Handling moving to trash bin and restoring on Mac
Still missing the feature to restore a directory and its children

* Now handling creation of empty files on MacOS

* Enabling restore of directories and its children

* Now working: moving a directory to another
inside the same location

* Now Indexer also remove file_paths not on fs

* Enabling multiple file moves on location watcher

* Fix Windows conditional compilation issues

* Fixing cas_id generation and bumping some deps

* Many Windows specific improvs and some refactors

* Rust fmt

* Using conditional compilation on extract inode function

* Linux fixes and some MaterializedPath improvements

* Rust fmt again

* Introducing tick behavior on location watchers

* Making LastFilePathIdManager more atomic

* Some vscode launch configs for lldb debugger

* Simplifying some lifetimes

* Making all watchers more consistent
2023-03-31 04:59:33 +00:00
Brendan Allan
c13d9c1ffb Sync debugging (#643)
* sync no networkey

* sync technically wired up

* fix p2p endianness for sync payload

* some sync debug stuff

* feature gate message production

* bump lockfile

* execute queries when sync messages disabled

* maybe fix ci?

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2023-03-29 00:38:48 -07:00
Vítor Vasconcellos
79d8e2871a Fix adding Locations that already have a .spacedrive file (#627)
* Handle RPC errors when adding Locations
 - Add custom error codes for NeedRelink and AddLibraryToMetadata errors in backend
 - Handle errors in AddLocationDialog form
 - Show custom error message for above errors and ask user to confirm action
 - Handle user action confirmation with appropriate RPC method (`locations.relink`/`locations.addLibrary`)

* Abstract openDirectoryPickerDialog logic to a function
 - Add 'Unkown Error' when code doesn't match a know one

* Fix bugs in logic to handle `locations.create` errors
 - Fix minor issue: AddLocationButton opened the dialog in desktop even when the user cancelled the file picker
 - Replace ternary with object for custom form error messages to allow for easier expansion
 - Break onSubmit logic up into functions
 - Remove incorrect `locations.indexer_rules.delete` call
 - Fix Checkbox element state not updating on external form state changes

* Adjust AddLocationDialog style to match other Dialogs in the app
 - Reset exception code at the end of `confirmAfterError`

* Rework how backend errors are handled when adding Locations
 - Change NeedRelink and AddLibraryToMetadata errors to use 409 (Conflict) status code and a static message.
 - Only enable custom submit logic in AddLocationDialog for 409 (Conflict) errors that have a known custom error form message.
 - Differentiate previous errors based on their messages instead of status code in AddLocationDialog onSubmit.
 - Add @hookform/error-message dependency for improved form validation error rendering.

* Invalidate library.list to ensure it is shown on sidebar

* Improvements to AddLocationDialog
 - Remove uncessary scope block and make useEffect depend directly on `form`
 - Move `form.handleSubmit` from `onSubmit` call to `handleSubmit` definition
 - Change basic Checkbox to RadixCheckbox and adapt onChange logic to onCheckedChange
 - Some simple styling improvements to the error message
2023-03-28 07:09:52 +00:00
Oscar Beaumont
0cafbf8aea Sync over p2p (#606)
* sync no networkey

* sync technically wired up

* fix p2p endianness for sync payload

* some sync debug stuff

* feature gate message production

* bump lockfile

* execute queries when sync messages disabled

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-03-21 06:03:08 -07:00
Jamie Pine
69f972d410 [Refactor] CAS id generation code simplify
Credit: ChatGPT 4
"In this simplified version, the code calculates the sample interval based on the file size and then uses a single loop to handle both the small and large file cases. The last sample is always taken from the end of the file."
2023-03-17 20:11:08 -07:00
Utku
ca6ec598d4 Onboarding, Spacedrop & Location Settings Screen & Styled API (#596)
* fix wrong current lib logic

* add delete lib dialog to LibraryGeneralSettings

* add delete lib to mobile LibraryGeneralSettings too

* onboarding screens

* move zxcvbn to @sd/client

* get started screen and bloom

* merge fix

* move generatePassword back to interface

* add useZodForm to mobile and match react-hook-form versions

* new lib screen

* Implement styled api

* create lib screen and some tweaks

* password input

* fix password meter comp

* new library style tweaks

* Fix remove password bug (interface)

* master password screen

* privacy screen

* creating lib screen

* hexagons are cool

* Expo 48

* keyboard handling

* fix P2P on IOS

* fix types

* asset script

* new icons

* Spacedrop screen

* Fix mobile asset imports

* fix import cycle warning

* Edit Location Settings screen and style changes on other setting screens

* fix library creating bug? hopefully lol

* move PasswordMeter to interface

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2023-03-17 02:00:02 +00:00
Jamie Pine
0c65f48d5e [ENG-441] Explorer size toggle (#611)
explorer size toggle + light theme icons start + radix checkbox
2023-03-16 18:59:06 -07:00
jake
f487db254b [ENG-429] Crypto organization/refactor (#607)
* update comment

* key manager feature gating

* update deps & more feature gating

* fix keyring feature gating

* add an `exhaustive_read` function for crypto/stream

* restructure STREAM module

* move tests+shared utils

* clean up stream decryption

* further cleanup

* impl to `GenericArray` for `Nonce`

* update examples

* update refs & `use`s

* fix `Nonce` -> `GenericArray` conversions

* better `Protected` conversions + remove `Password` type

* a work of art

* finishing touches

* some API changes

* rename `StreamX` to `X`

* fix everything else

* separate `primitives` from `types`

* update imports & fix build
2023-03-15 16:09:36 +00:00
Oscar Beaumont
47da733cfd Eng 432 refactor mdns service/state (#604)
clean up mdns state and refactor advertisement system
2023-03-13 18:05:46 +08:00
Oscar Beaumont
9719940552 Typesafe stream handling (#603)
* typesafe stream handling
Now it's impossible for your to respond to a broadcast stream. Previously this would just fail due to the TCP connection having been closed by the sender.

* remove connected_peers state from `SpaceTime`

* use `OutEvent` instead of `tokio::spawn` for `SpaceTime` events

* Polling for events > tokio::mpsc

* extend max length of sync messages to ~4GB + broadcast done message

* Fix "actOS" operating system decode error
2023-03-13 16:58:35 +08:00
Ericson "Fogo" Soares
12a437ffb6 [ENG-371] Subpath shallow re indexing (#599)
* Introducing sub path indexing for IndexerJob

* Introducing shallow indexer job

* Some feature flags to avoid warnings at pnpm prep

* Many fixes at shallow indexer job

* Changing how we implement the bookkeeping for file_path ids
Now we're account for independent locations, and also integrating
better with the LibraryContext, instead of using a static global

* Making indexer job reentrant

* Introducing shallow file identifier job
And a bunch of minor refactors

* Rust fmt

* Removing light scan from getExplorerData query
Light scan is a mutation, so we can call it on useEffect function from
the Explorer component in the frontend, when a location_id or the explorer
path changes

* Handling job early finish on init for identifier

* Only invalidate query if we have orphan paths

* Introducing ShalowThumbnailerJob

* Clippy warnings about `into_iter()`

* Naming scheme for Prisma's selects and includes

* Invalidating getExplorerData at thumbnailer

* Small mistakes lol

* Some nitpicks with pnpm prep

* Rust fmt

* Changing indexer's walk log to `trace!`

* Not deleting all file_paths on location fullRescan

* TS typecheck

* Removing `file_path` selection just with id
2023-03-10 16:25:58 +00:00
Oscar Beaumont
7588442d5d Basic Spacedrop working with rspc! (#601) 2023-03-09 06:23:17 -08:00
Jamie Pine
93fecd6492 [ENG-408] Dynamic Object icons (#600)
* improved object icons

* remove redundant control flow

* added support for base icon overrides by extension
+ deleted old icons folder and moved official icons into there

* fixed folder icon error

* removed temp nodes example

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-03-09 11:08:10 +00:00
Oscar Beaumont
2518e98ad5 [ENG-407] Spacedrop backend (#598)
* format Rust

* Spacedrop a string

* Praise thee Clippy, lord of the Rust

* add protobuf to Mac and Linux CI

* plz GH Actions have Chocolatey
2023-03-09 11:27:45 +08:00
Oscar Beaumont
e696db186b P2P Core (#597)
* implement space time

* I love it when my code works
2023-03-09 03:18:33 +08:00
Brendan Allan
25b4b72cf1 rename LibraryContext to Library (#595) 2023-03-07 01:15:20 -08:00
Brendan Allan
828ea8c498 CI Cache Factory (#593)
* introduce cache factory

run cache factory on all branches

add names

use single quotes

add setup and prisma generation

add pnpm install

don't install cargo-watch in ci

make composite setup script

use env var for windows ci

undo all powershell stuff

* better windows ci

* don't install cargo-watch in ci

* only run cache factory on main

* i'm silly

* debug only ci

* check runner instead of matrix in setup action

* don't save cache output most of the time

* no more os-specific features

* cache entire workspace

* only cache on main again

* test instead of buidl in cache factory

* smaller ci + testing

* don't individually test each crate

* remove unnecessary needs

* smaller formatting and more clippy

* undo rustfmt changes

* disable tests
2023-03-06 06:10:55 -08:00
Brendan Allan
2a09c7214b generate + use ModelSyncData enum (#592)
* generate + use ModelSyncData enum

* cache prisma clients on sync-generator
2023-03-04 22:12:37 -08:00
Brendan Allan
3a86b6c2d8 convert location + file path to unique shared models (#587) 2023-03-03 00:02:33 -08:00
Brendan Allan
c2d8c2f072 More sync impls (#586)
tags and stuff
2023-03-02 21:23:10 -08:00
Ericson "Fogo" Soares
ae41b1a0f4 Fix unique constraint on file path table (#585)
* Complying with a pedantic Clippy

* Some task runners for vscode

* Solving duplicated directories events
When creating a directory through MacOS's Finder, for some reason
FSEvents receives 2 Create Folder events that we have to handle

* Damn rustfmt
2023-03-02 02:36:49 +00:00
Jamie Pine
963584f065 [ENG-379] Explorer item resizer (#580)
* added item resizer
fixed explorer store bug
refactored file image component

* better sizing for videos

* fixed inspector width issue

* remove console.log

* added column titles to list view + extra details

* moved util

* remove imports

* Update packages/interface/src/components/explorer/FileColumns.tsx

Co-authored-by: Brendan Allan <brendonovich@outlook.com>

* address issues

* fix extension in file list name

* Update packages/interface/src/components/explorer/FileColumns.tsx

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-02-25 06:16:57 +00:00
Jamie Pine
fab84b7f7a [ENG-363] Spacedrop UI + Misc Improvements (#568)
* begin spacedrop ui + misc ui improvements

* better 404 xox

* Update extensions.rs

I think I prefer Container

* added DragRegion component,  ot tested cuz im on my fone

* Update DragRegion.tsx

fix import

* added dummy drop items

* better dummy data

* added clouds & search bar

* added action buttons to spacedrop items

* customize subtle button

* added support for apng, thanks luka big pants

* use relative path in sidebar

* use BYTES const

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-02-24 08:12:21 +00:00
Brendan Allan
f4ff6c6a99 Replace Location.local_path with path (#571)
* replace Location.local_path with path

* Using more references to avoid unneeded moves and removing unneeded errors variants

* remove unnecessary stuff

* location id checks

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-02-20 05:14:47 +00:00
jake
c21743f44e [ENG-361] Crypto crate docs and tests (#572)
* add hashing tests

* add encryption/decryption tests

* remove `reason`

* add file header tests (preview media deserialization is broken)

* fix keyslot reading bug

* add sd-crypto testing to ci

* add tests/constants for all hashing algorthms and param levels

* add blake3-kdf tests

* use `const` arrays for storing expected output

* test for `5MiB` `encrypt_streams` and `decrypt_streams`

* add invalid/mismatched nonce tests

* update `primitives` docs

* remove erroneous `,`

* grammar tweaks

* add errors to `#[should_panic]`

* cleanup `stream` tests

* cleanup hashing tests a little

* function docs
2023-02-16 11:42:30 +00:00
Oscar Beaumont
60d5bc4afe improvements to spacedrive:// custom URI protocol (#550)
* fix `spacedrive://` custom protocol on Windows
(hopefully)

* custom protocol using `http::Response` + fix broken web

* import patches before App on web

* use `http::Request` for input to `handle_custom_uri`

* break  into dedicated file + error handling

* serving files via custom protocol

* cargo fmt because vscode did cringe

* lru cache to reduce video chunk request time

* add  helper to JS

* clippy be like

* remove duplicate Open buttons in context menu

* fix Linux 🙏

* no shot

* fix Windows custom URI passing (hopefully)

* better fix for custom uri on Linux

* upgrade Tauri for  feature

* switch url replacement order

* prevent React dev tools script being added in prod to desktop

* remove React devtools from html

* upgrade Tauri; required upgrading rspc, Axum, PCR

* pass typecheck + less cringe bigint

* clippy is love, clippy is life

* Typecheck plz

* fix bigint to number conversion

* use httpz + localhost server for Linux

* clippy be right

* Remove console.log

* [wip] proper auth

* fix Linux sidebar padding

* Secure Axum server with random

* Extracting app setup specific to linux to a different file

* remove outdated comment

* Some tweaks on cursom_uri.rs

* file_path_with_location doesn't need to be a named include

* fix typo

* factually wrong comment

* Change `unwrap` to `expect`

* bruh

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-02-14 13:27:11 +08:00
Jamie Pine
c07a42b7c8 [ENG 239] Onboarding Flow & Location Settings (#529)
* begin better onboarding

* added input and altered text

* better router & text + database icon

Co-authored-by: maxichrome <maxichrome@users.noreply.github.com>

* work on privacy screen + radio buttons

* fix video extension bug and alter screens

* add pending schema and location manager helper

* functional onboarding

* added secure temp store and started creating library loading screen

* fix secure temp keystore + api

* better onboarding

* added location settings and some overview concept, all WIP

* fix switch

* prep

* fix location router

* added backend settings

* attempted to fix form

* begin indexer rules editor, plus tweaks

* indexer rules coming soon

* fix onboarding img size

* cleanup

* clone is needed here, but clippy no like

* sike

* whole bunch of fixes

* clippy + ts

* Removing some TODOs from api/libraries.rs and fixing db size calculation

* moved object kind to client, added half functionality for appearance settings

* fix RadioGroup helper

* fix type issues

* cargo fmt

* fix creating library error handling + invalidate location list on update

* forgot to switch back to onError

* Invalidating getStatistics query on library creation and introducing the concept of waiting for a job on FileCopierJob

* F* cargo fmt

* fix RadioGroup interactivity

* wipe all migrations

* put back COLLATE NOCASE on extension columns

* update core.ts

* remove unused device component

* fix typeerror in mobile

---------

Co-authored-by: maxichrome <maxichrome@users.noreply.github.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-02-10 14:08:13 -08:00
jake
c4658815ad [ENG-357] Automatically mount associated key during decryption (#564)
* auto-mount associated key during decryption based on content salt

* use iterator directly

* cleanup
2023-02-09 10:07:53 +00:00
jake
08059ae700 [ENG-358] Malicious client protection for the KM (#562)
* add `queue_check`, `mount_check` and `unlock_check`

* make `unlock_check` marginally more readable

* clippy

* clean up & rename functions

* remove unlock check on `keys.list`

* use american spelling of `favorite`

* use `then_some` (thanks clippy)

* more american spelling
2023-02-09 09:30:26 +00:00
Brendan Allan
23fbb1e408 Move location online status out of database (#560)
* move location online status out of database

* remove useLibrarySubscription

* remove old code
2023-02-07 05:58:47 -08:00
Brendan Allan
b901a72208 Merge branch 'main' of https://github.com/spacedriveapp/spacedrive 2023-02-07 21:26:06 +08:00
Brendan Allan
4602690751 damn i'm stupid 2023-02-07 21:00:27 +08:00
jake
e5b45d8f49 [ENG-355] Keychain integration (and some typesafety) (#558)
* update crypto MSRV

* rename `keychain` to `keyring`

* make a start on the keymanager unlock refactor/keychain integration

* update routes

* update bindings

* add const identifiers

* add UI/front-end support for unlocking KM with OS keychains

* remove SK from lib creation dialog

* update query name

* add keyring functions

* attempt to update `change_master_password()` to use the keychain

* cleanup, fix master password change ui, better secret key in keyring detection

* cleanup TS a little

* add route for getting secret key from keyring

* update bindings

* update var names + show secret key in keys settings

* add `react-qr-code` and option to view the secret key (if it's in the OS keyring)

* allow copying SK to clipboard

* add `key_type` so we're not reliant on specific UUIDs for root/verification key handling

* clippy

* fix mobile typecheck

* fix typecheck, fix typo and tweak balloon hash parameters

* minor cleanup + typo fix

* use newtype structs

* WIP type refactoring (major readability boost!)

* update `use`

* add tokio `sync` feature

* too many structs? idk

* more cleanup

* add `generate` and `Nonce`

* `Nonce` and `Key` typesafety (beautiful)

* clippy + cleanup

* update code & examples

* fix bug & remove `ProtectedVec` as it looked out of place

* use `Key`

* add a query invalidation to make the UI extremely responsive

* ci pls work

* remove `keyringHasSk` route
2023-02-07 12:03:12 +00:00
Brendan Allan
b9089bf065 update msrv to 1.67 2023-02-01 12:21:34 +08:00
Brendan Allan
bd37ca0d14 remove TryInto usage in sync manager 2023-02-01 11:42:43 +08:00