Commit Graph

53 Commits

Author SHA1 Message Date
Oscar Beaumont
f821bb9a0f Stop App Freezing (#1257)
* Drop Tauri custom URI handler
Me no likely but it has to be done.

* fix app startup with location

* fix "Add Location" button on web

* Serve correct content range

* Backport changes from 08ba4f91

* none of my homies like panics

* minor fixes

* fmt with new Rust version
2023-08-31 06:54:40 +00:00
Oscar Beaumont
d758977d82 Improve Sync + P2P Integration (#1265)
* Big bruh moment

* whoops

* Less stackoverflowy debug

* stuff

* Fix flawed P2P mDNS instance advertisements

* do sync when connecting with peer

* Sync after pairing

* resync_part2 all the time

* Invalidate all the things

* Invalidate whole React Query on sync event

* emit_messages_flag

* emit_messages_flag

* Backend feature flags + "emitSyncEvents" feature

* Patch `confirm` type cause Tauri cringe

* clippy

* idk but plz work

* bruh

* Fix ComLink bug

* remove log

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-29 17:54:58 +00:00
Oscar Beaumont
7bf0c0ae4a [ENG-994] Minor P2P fixes (#1238)
* mdns advertisement on detection

* fix bug in P2P discovery

* Add `P2PEvent::ExpirePeer`

* React properly

* Consistent tracing versions

* better logger config

* Reduce excessive logging from P2P

* Fix panic log hook

* Remove `dbg`

* Fix bug in thumbnailer remover

* wip: Connected peers in UI

* `Sync` is large pain

* init services after library load

* improve error handling of logger

* Sync protocol shutdown

* Add "updater" feature

* fix commands bindings

* nlm state debug query

* nlm debug status

* Make TS happy

* Add `RemoteIdentity` to libraries debug info

* Improve debug data presentation

* Among us level sus

* minor fix panic hook

* Fix EOF issue

* fix instance not being added on pairing

---------

Co-authored-by: Utku <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2023-08-27 06:41:26 +00:00
Vítor Vasconcellos
1b28343304 Fix Server CI running out of disk space (#1222) 2023-08-16 21:01:39 +00:00
Vítor Vasconcellos
c86a728a1a Update server Dockerfile due to crate name change (#1211)
* Update server Dockerfile due to crate name change

* Forgot that the bin name changed too
2023-08-12 21:00:16 +00:00
Brendan Allan
00e4aa9e8f Sync ingesting (#1181)
* sync + ingest refactor

* fix Event enum description

* actually do sync over network

* re-enable heif

* remove comment
2023-08-07 07:52:43 +00:00
Vítor Vasconcellos
417d5d3be3 [ENG-811] Fix server Dockerfile + create CI workflow to build and publish docker image (#1012)
* Fix server docker build failing due to apt locking error
 - Replace ffmpeg dep with only the libav* and related required lib (reduce size)
 - Add warning to tauri build command when DMG background is missing
 - Minor rust fmt

* Enable assets feature for server Dockerfile
 - Fix web app not using the correct address in prod
 - Add build step for web assets in server Dockerfile

* Enable repository to be defined in server's Dockerfile
 - Enable server's release workflow to specify which repository to build to allow external PRs to work

* Fix pnpm docker cache not working
2023-06-25 15:16:11 +00:00
Ericson "Fogo" Soares
e693c7a542 [ENG-381] Update context menufs actions to support multiple source (#927)
* Generalizing filesystem jobs to accept multiple files at once

* Some small changes and fixing front

* Enable rename route to replicate Finder's behavior

* Assign tag to multiple objects at once

* Remove from recents accepting multiple files

* Fixing some warnings

* Adding multiple files feature to Open and Open With

* Conditional stuff for macos

* Generating commands.ts and other minor warnings

* Rust fmt

* TS typecheck

* Rust format and TS typecheck

* Requested changes

* Requested changes

---------

Co-authored-by: Utku <74243531+utkubakir@users.noreply.github.com>
2023-06-14 23:00:28 +00:00
Vítor Vasconcellos
1c7855ded6 [ENG-630] Windows ffmpeg + libheif custom build (#871)
* Initial Windows ffmpeg + libheif custom build

* Add build steps for most of ffmpeg deps

* FFmpeg deps and libheif

* Fix libheif build

* Fix libvpx and dlfcn + attempt to fix rav1e

* Rework the whole ffmpeg-windows build system
 - New system based on https://github.com/BtbN/FFmpeg-Builds
 - Add new ffmpeg-windows workflow
 - Rename macos ffmpeg workflow
 - Adapt macos setupt script due to above name change

* Forgot to update update the workflow name

* Strip all libs from debug symbols

* Add docs

* Add libde265 deps, required by libheif
 - Make x265, svtav1 and dav1d as shared deps (used by both ffmpeg and libheif)

* Add missing libheif to Linux setup script

* Fix libx265 build script

* Forgot to point x265 ninja install to the correct directory

* Remove libaom and libsvt-av1
 - dav1d and rav1d are our default AV1 decoders/encoders
 - Quote subshell executions
 - Make libweb shared

* Forgot to remove libaom and libsvt-av1 build steps

* Fix typo

* Try force webp to link against static libs

* Revert libwebp to a static build

* Dumb typo

* Modify windows script to download our ffmpeg build (WIP)

* Fix dlls output folder structure

* Fix dumb mistake

* Remove unused ffbuild_enabled

* Enable core's heif feature on Windows
 - Fix windows ffmpeg build not including the headers
 - Fix windows setupt script incorrect download loagic
 - Implement build_arg to pass which repo ref to use when cloning

* Fix windows setup script

* Fix workflow artifact path
 - Make ffmpeg-windows dockerfile respect the FFMPEG_VERSION env

* Fix Windows setup script incorrect logic for downloading ffmpeg builds

* Error out when workflow_runs is empty

* Fix dumb mistake

* Manually define ffmpeg version for windows script

* Fix ffmpeg windows build extract logic

* Fix prop access in windows setup script

* Revert back to a web request because nightly.link does a redirect before serving the artifact content

* Fix windows setup script

* Do not use nightly.link in Github CI

* Fix windows setup script

* Should finally fix window setup script
 - Update ffmpeg-windows deps
 - Should fix ffmpeg-windows failing to build due to mingw changes in new base image

* Fix libxz failing to build due to doxygen

* Fix windows setup-script not executing till the end

* Fix LASTEXITCODE not defined

* Remove libjxl, deps are not being compiled

* Fix dll and lib copy logic

* Move final copy dll logic to external script

* Use main for libjxl

* Change brotli from stable to main
 - Attempt to fix libjxl

* Attempt fix lib copy again

* Split copy_dll logic to avoid cache burst in docker

* Missing file

* Change how to export build files from shared deps

* Replace rsync with cp

* Fix copy

* Fix dir not existing

* Fix pkgconfig

* Remove superflous files from exported ffmpeg for windows
 - Adjust dav1d to not build tools and examples
 - Adjust windows setup-script to point linker to the libs directory

* Fix dav1d meson config args

* Fix dumb mistake

* WORK PLZ

* Fix .lib file location
 - Strip all dlls

* Formatter
2023-06-10 15:23:37 +00:00
Brendan Allan
fd236a1b57 [ENG-679] Reserve ids for built in indexer rules (#909)
* indexer rules pub ids

* should work?

* better migrator

* errors

* debugging

* maybe?

* double migrate

* please

* maybe fix?

* update lockfile

* SD_ACCEPT_DATA_LOSS message

* put tracing back

* dumb

* fix system indexer rule ui
2023-06-06 14:42:52 +00:00
Oscar Beaumont
049966d848 [ENG-674] fix invalidation (#891)
* reenable the invalidate requests system

* a tester for the invalidate system

* watch locations in Library::load

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-05-31 08:55:46 +00:00
Oscar Beaumont
e4696c7303 [ENG-667] Fix Clippy CI (#887)
* plz just work

* no shot

* just disable Tauri notifications

* Tauri is weird AF
2023-05-30 17:11:37 +00:00
Oscar Beaumont
5f71e0c1f6 [ENG-546] Improve error handling (#802)
* log to disk

* remove some unwraps

* panicless

* some p2p error handling

* clippy moment

* Fix `<ErrorBoundary />`

* open logs button

* 39 to 0

* fix types

* update deps and comment out broken tests

* clippy

* more clippy

* upgrade rimraf - https://github.com/isaacs/rimraf/issues/259

* regen broken lockfile

* update `notify` and update `commands.ts`

* more clippy (pls work)

* allow deprecated for p2p (hopefully temporary)

* a

* upgrade deps for p2p

* do betterer

* do it correctly

* remove unused import

* improve core startup error + bc keypair

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: brxken128 <77554505+brxken128@users.noreply.github.com>
2023-05-30 09:24:05 +00:00
Oscar Beaumont
a20f6262fd [ENG-659] Spacedrive On Jamie's NAS (#866)
* serve dem

* 0% chance of working

* With a side of Vite plz

* no CI for now

* Clippy plz work

* undo

* bruh

* plz work
2023-05-29 10:47:59 +00:00
jake
1aa8826398 SPDX license and workspace inheritence for re-used metadata (#833)
use workspace inheritence for license, edition and repo URL
2023-05-20 01:52:55 +00:00
Oscar Beaumont
c2666f7f63 Sync over p2p (#606)
* sync no networkey

* sync technically wired up

* fix p2p endianness for sync payload

* some sync debug stuff

* feature gate message production

* bump lockfile

* execute queries when sync messages disabled

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-03-21 06:03:08 -07:00
Oscar Beaumont
a9fceae819 improvements to spacedrive:// custom URI protocol (#550)
* fix `spacedrive://` custom protocol on Windows
(hopefully)

* custom protocol using `http::Response` + fix broken web

* import patches before App on web

* use `http::Request` for input to `handle_custom_uri`

* break  into dedicated file + error handling

* serving files via custom protocol

* cargo fmt because vscode did cringe

* lru cache to reduce video chunk request time

* add  helper to JS

* clippy be like

* remove duplicate Open buttons in context menu

* fix Linux 🙏

* no shot

* fix Windows custom URI passing (hopefully)

* better fix for custom uri on Linux

* upgrade Tauri for  feature

* switch url replacement order

* prevent React dev tools script being added in prod to desktop

* remove React devtools from html

* upgrade Tauri; required upgrading rspc, Axum, PCR

* pass typecheck + less cringe bigint

* clippy is love, clippy is life

* Typecheck plz

* fix bigint to number conversion

* use httpz + localhost server for Linux

* clippy be right

* Remove console.log

* [wip] proper auth

* fix Linux sidebar padding

* Secure Axum server with random

* Extracting app setup specific to linux to a different file

* remove outdated comment

* Some tweaks on cursom_uri.rs

* file_path_with_location doesn't need to be a named include

* fix typo

* factually wrong comment

* Change `unwrap` to `expect`

* bruh

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-02-14 13:27:11 +08:00
brxken128
a3b1c8f5b8 tell everything to use workspace tokio + add tokio git patch 2023-01-25 14:36:29 +00:00
Ericson "Fogo" Soares
bcbcd260d4 ENG 223 Location Awareness (#468)
* Introducing location online/offline checks and location relink

* Some initial drafts for location watchers

* Location metadata relink and add library

* Many improvements at job system
Now using prisma batching at identifier job
Removing blocking I/O from extension subcrate
Implementing lazy generation of thumbnails
New current directory identifier job to be used on light rescans

* Some optimizations on identifier and object validator jobs

* merge jamie's identifier PR

* fully repaired identifier job

* properly hooked up object kind

* inspector fix

* fix video badge

* small improvements to libraries settings

* identifier and inspector improvements

* fix feature flags and hook up context menu location utilities

* BETTER CONTEXT MENU x100

* test-files

* style tweaks

* new icon designs

* manifest

* fix thumbnails on web

* media data

* New Location Watcher and some minor fixes

* disable broken media_data extractor, wip

* wip

* function name fix

* Fixing pnpm prep and some warnings

* Solving a race condition beetween indexer job and FS event handlerSome other minor warnings

* Generating thumbnails on watcher

* Remove event handler on watcher

* Some initial works on modify events and other small fixes

* File update event

* Trying to be more generic with used events and some tests to validate our assumptions

* Turning on location metadata file

* Introducing core unit tests on CI pipeline

* Submiting new unit test assumptions to validate on windows CI

* Fixing unit tests

* Fixing unit tests again

* Fixing unit tests

* Fixing unit tests for macos

* Fixing unit tests for macos again

* New structure for platform dependent event handling
Implementing event handlers for Linux and MacOS

* minor fixes + rustfmt + clippy

* Windows event handling

* Introducing a feature gate to only use location watching on desktop app for now

* Putting more stuff behind feature gates to avoid warnings

* Adding feature to cargo test on CI

* Changing some debug logs to trace logs and removing Jamie specific stuff

* Make location removal from manager less async

* fix build when "location-watcher" feature disabled

* fix types + clippy

* make location manager non-static

* remove uses of `to_string_lossy`

* more invalidate_query calls

* Expose `library_ctx` directly to avoid needless clones

* New materialized_path handling for directories

* Removing cascade delete between file_path and object
- Some other minor stuff

* remove unused `CurrentDirFileIdentifierJob`

Co-authored-by: Jamie Pine <ijamespine@me.com>
Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-31 00:53:24 +08:00
Brendan Allan
b713c896a0 More Sync Stuff (#438)
* broken sync example

* move to react

* filter owned ops more

* update deps

* working sync

* relation operations

* fix pnpm lock

* please rustfmt

* fix api

* compare strings properly

* please clippy

* feature gate logging

* use bleeding rspc in example

* use bleeding rspc

* use generated rspc client
2022-10-27 23:12:28 -07:00
Brendan Allan
723058e060 merge main into update-rspc-and-more 2022-10-05 08:58:03 +08:00
Brendan Allan
ed06e3051e [ENG-84] Sync library (#394)
* new sql lib stuff

* add sync stuff + rename all crates

* build sd-core

* add sync/example/dist to source

* fix sync example in monorepop

* appease clippy

* update lockfile

* update commit hooks

* fix typescript

* fix typescript build

* please rustfmt
2022-10-04 05:25:12 -07:00
Brendan Allan
e3f92ab4ec please rustfmt 2022-10-04 20:07:05 +08:00
Oscar Beaumont
60e228ec94 upgrade rspc 2022-10-02 01:16:11 +08:00
Ericson "Fogo" Soares
6fd620087b Video thumbnails (#376)
* Preparing some scaffolding for video thumbnails

* Implemented thumbnail generation for videos
* Propagating errors of `Node` creation
* Using ffmpeg feature gate

* Introducing ffmpegthumbnailer-rs as a subcrate on core

* - rename to thumbnailer
- fix explorer thumbnail bug
- add more supported video types
- re-fix explorer performance

* remove nested licence

Co-authored-by: Jamie Pine <ijamespine@me.com>
2022-09-24 01:21:13 -07:00
Brendan Allan
6e07435a15 Use PCR for migrations (#385)
* use pcr migrations branch

* use 0.6.2 branch with migrations

* use latest prisma stuff

* allow force reset of db in dev

* remove .spacedrive file

* update rspc in apps/server

* use rspc 0.0.5 in all crates

* add os to prisma client cache key

* add runner os to clippy prisma cache
2022-09-22 00:50:21 -07:00
maxichrome
4673867ad1 Merge remote-tracking branch 'origin/main' into use-rust-shortcuts 2022-09-15 02:15:40 -05:00
maxichrome
119b8176f2 Merge remote-tracking branch 'origin/main' into use-rust-shortcuts 2022-09-14 15:59:59 -05:00
Oscar Beaumont
dd04625a4f properly shutdown the core when using server 2022-09-14 19:02:11 +08:00
maxichrome
2c3eed2359 ignore server library data 2022-09-13 18:49:45 -05:00
Oscar Beaumont
c7441d1d64 Rustify mobile (#361)
* Refactor navigation flow & types

* Remove drawer screen wrapper

* Remove DrawerItem + cleanup

* Switch to JS Stack Nav & header animations

* [WIP] Spacedrive core on Android & IOS

* Update Podfile and cleanup Contributing guide.

* Remove @sd/core from mobile

* File Modal

* Prettify File Modal & Add date-fns

* IOS subscriptions

* Update package versions

* Custom header for stack screens

* android subscriptions

* Animate Drawer button & template for Search screen

* Search header

* Fix Search icon being weird

* Merge branch 'main' into rustify-mobile

* fix rspc dep + setup script for mobile

* Less margin on header

* Move shared assets & drawer logo for mobile

* support for IOS simulator

* add type safe rspc hooks to mobile

* Cleanup PR & Update packages

* Updated bindings from main

* Update lefthook.yml

* Remove `tag` folder from core
The `tag` folder came back from the dead. Maybe it got confused in merge conflict?

* update pnpm lockfile + fix tsc errors

* fix asset import

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2022-08-29 04:59:09 -07:00
Oscar Beaumont
ea92af57f8 Move Rust backend to rspc (#345)
* move Rust backend to rspc

* move server to Axum + remove parts of old bridge

* move frontend over to @rspc/client

* move core to rspc + update deps

* fix Typescript errors + upgrade deps

* document invalidate_query! macro

* general cleanup + upgrade to PCR 0.6.0

* prisma error handling

* upgrade to rspc 0.0.4

* update vite-plugin-ssr

* fix typescript

* fix builds

* put landing page app name back

* hardcode ffmpeg version on Windows

* rename 'command' to 'mutation' to line up with react-query terminology

* upgrade rspc to v0.0.5 + fix types

* use shared ffmpeg binaries

* general Typescript cleanup

* fix clippy workflow failing due to tauri proc-macro

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2022-08-03 08:36:03 -07:00
Ericson "Fogo" Soares
85e5eec993 Resumable Jobs + Lefthook Integration (#344)
* Introducing Lefthook for git hooks automation
* TypeScript typechecking and linting
* Rust formatting and linting
* Spellchecking (also corrected spell errors in some files)
* Check links in md files

* Introducing resumable jobs
* Abstractions to pause and resume jobs automatically when application exits and is started
* Changing database to use Bytes for UUID fields
* Changing uuid fields on core to use uuid::Uuid instead of String
* Updating some dependencies and introducing msg_pack serialization to save job state on database

* Fixing some clippy warnings

* Fixing a regression introduced on identifier job, doing too much db accesses concurrently
2022-07-27 00:06:34 -07:00
Oscar Beaumont
d7794bad4e make the Socket have named fields for clarity 2022-07-12 19:33:04 +08:00
Oscar Beaumont
7e9d036aee allow events to work from web 2022-07-11 16:38:02 +08:00
Oscar Beaumont
c685ce5fe9 Library manager (#258) 2022-07-10 19:05:24 -07:00
Jamie Pine
2cc3f3d95d Location Settings (#302)
* Delete locations

+ Show online status

* style tweaks

* tweaks

* location rescan button

* fix location delete db locking bug

* opting to remove self referencial relation on file_path

* correct query

* consolodate migration

* consolodate migrations
2022-06-24 06:26:45 -07:00
maxichrome
59bc2ee9db Merge remote-tracking branch 'origin/main' into jamie-unsorted-changes 2022-05-26 03:22:36 -05:00
Oscar Beaumont
01610e008e Update package.json license field to GPL-3.0-only (#184) 2022-05-24 14:00:01 -05:00
maxichrome
f5e0666d11 Format existing code with new conventions 2022-05-22 23:24:43 -05:00
maxichrome
a0d0938001 format using tabs 2022-05-22 23:07:35 -05:00
Jamie Pine
9870d58b35 much 2022-05-10 09:03:20 -07:00
Oscar Beaumont
d834fe3b2c feature gated p2p + update deps + pushed bindings 2022-04-27 17:22:26 +08:00
Harry Hopkinson
d7cbba7a12 Linted Files according to the .prettierrc. 2022-04-24 21:16:55 +01:00
Jamie Pine
0287561f48 wip 2022-04-24 09:50:22 -07:00
Oscar Beaumont
a8696d8faa @sd/server in Docker and deploy to Kubernetes 2022-04-24 14:55:20 +08:00
Oscar Beaumont
bdc074b0d9 added Dockerfile for sdserver 2022-04-23 23:41:11 +08:00
Jamie Pine
93d80233d7 server transport via websocket
Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2022-04-23 06:50:17 -07:00
Jamie Pine
40cc780ec1 added unique file identifier 2022-04-23 01:12:35 -07:00
Jamie Pine
4c4325619f server start, landing tweaks 2022-04-20 23:35:48 -07:00