Commit Graph

11 Commits

Author SHA1 Message Date
Vítor Vasconcellos
405fb023d8 Clean up and update rust dependencies (#2544)
* Clean up and update rust dependencies

* Attempt to fix Windows Error
2024-06-10 19:44:00 +00:00
Vítor Vasconcellos
e797b02e65 Media metadata extraction & Thumbnailer rework (#2285)
* initial ffprobe commit

* Working slim down version ffprobe

* Auto format ffprobe and deps source

* Remove show_pixel_formats logic
- Fix do_bitexact incorrect check in main after last changes
- Fix some clangd warning

* Remove show_* and print_format options and their respective logic

* Rework ffprobe into simple_ffprobe
- Simplify ffprobe logic into a simple program that gather and print a media file metadata

* Reduce the amount of ffmpeg log messages while generating thumbnails

* Fix completly wrong comments

* mend

* Start modeling ffmpeg extracted metadata on schema
 - Start porting ffprobe code to rust
 - Rename some references to media_data to exif_data

* Finish modeling media info data
 - Add MediaProgram, MediaStream, MediaCodec, MediaVideoProps, MediaAudioProps, MediaSubtitleProps to Schema
 - Fix simple_ffproble to use its custom print_codec, instead of ffmpeg's impl

* Add relation between MediaInfo and FilePath
 - Remove shared properties from MediaInfo and related structs
 - Implement Iterator for FFmpegDict

* Fix and update schema

* Data models and start populating MediaInfo in rust

* Finish populating media info, chapters and program

* Improve FFmpegFormatContext data raw pointer access
 - Implement stream data gathering

* Impl FFmpegCodecContext, retrieve codec information
 - Improve some unsafe pointer uses
 - Impl from FFmpegFormatContext to MediaInfo conversion

* Fix FFmpegDict Drop

* Fix some crago warnings

* Impl retrieval of video props
 - Fix C char* to Rust String convertion

* Impl retrieval of audio and subtitle props
 - Fill props for MediaCodec

* Remove simple_ffprobe now that the Rust impl is done

* Fix schema to match actually retrieved media info
 - Fix import some FFmpeg constants instead of directly using values

* Rework movie_decoder
 - Re-implement create_scale_string and add support anamorphic video
 - Improve C pointer access for FFmpegFormatContext and FFmpegCodecContext
 - Use newer FFmpeg abstractions in movie_decoder

* Fix incorrect props when initializing MovieDecoder

* Remove unecessary lifetimes

* Added more native wrappers for some FFmpeg native objects used in movie_decoder

* Remove FFmpegPacket
 - Some more improvements to movie_decoder

* WIP

* Some small fixes

* More fixes
Rename movie_decoder to frame_decoder
Remove more references to film_strips

* fmt

* Fix duplicate migration for job error changes

* fix rebase

* Solving segfaults, fuck C lang

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Update rust to version 1.77
 - Pin rust version with rust-toolchain.toml
 - Change from dtolnay/rust-toolchain to IronCoreLabs/rust-toolchain for rust-toolchain support
 - Remove unused function and imports
 - Replace most CString uses with new c literal string

* More segfault solving and other minor fixes

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix ffmpeg rotation filter breaking portrait video thumbnails #2150
 - Plus some other misc fixes

* Auto format

* Retrieve video/audio metadata on frontend

* Auto format

* First draft on ffmpeg data save on db

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix some incorrect changes to prisma schema

* Some fixes for the FFmpegData schema
 - Expand logic to save FFmpegData to db

* A ton of things

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Integrating ffmpeg media data in jobs and API

* Rspc can't BigInt

* 🙄

* Add initial ffmpeg metadata entries to Inspector
 - Fix ephemeral metadata api to match the files metadata api call

* Fix Inspector not showing ffmpeg metadata

* Add bitrate, start time and chapters video metadata to Inspector
- Fix backend BigInt conversion incorrectly using i32 instead of u32
- Change FFmpegFormatContext/FFmpegMetaData bit_rate to i64
- Rename byteSize to humanizeSize
- Expand humanizeSize logic to allow handling bits and Binary units
- Move capitalize to @sd/client utils

* Solving some issues

* Fix ffmpeg probe getting incorrect stream id and breaking database unique constraint
 - Fix humanizeSize breaking when receiving floating numbers
 - Fix incorrect equality in StatCard
 - Fix unhandled error in Dialog when trying to remove an unknown dialog

* fmt

* small improvements
 - Remove some unecessary recursion_limit directive
 - Remove unused app_image releated functions
 - Fix metadata query enabled flag

* Add migration for ffmpeg media data

* Fix cypress test

* Requested changes

* Implement feedback
 - Update locale keys for all languages
 - Add pnpm command to update all language keys

* Fix thumb reactivity in non indexed locations

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>
2024-05-09 02:20:28 +00:00
Ericson "Fogo" Soares
6b760b0b2a [ENG-1628] Write new indexer with the task system (#2161)
* Moving file-path-helper to a sub-crate on core

* Parallel walker tested and working

* Change inner core crate names to sd-core-* naming scheme

* Moving stuff around

* Save and Update tasks for the new indexer job

* Some initial drafts on the new job system

* More drafts on job system

* Changing walker task to a more explicit state machine
Also drafting more of job system

* More drafting on job resume

* Completed the draft on job system inner workings

* New job context abstraction to decouple library stuff from job system

* Properly use composition on task dispatcher

* First draft on indexer job

* Job serialization

* Handling ancestors in the distributed walker for indexer

* Saving computed directories sizes on a location to db

* Enable a way to check if jobs are running in a location

* Progress reporting on indexer job

* Reorganizing modules

* Shallow indexer

* Rust fmt

* Attempting windows CI fix

* Attempting windows CI fix again

* Attempting windows CI fix again
2024-04-08 14:32:44 +00:00
Arnab Chakraborty
3bd1622e93 [MOB-23] Mobile Hardware Information for Overview Page (#2106)
* wip for iDevices

* Working HardwareModel Info for iOS

* wip

* Merge 'main' into 'mob-hw-info-overview'

* Half-Working `get_volume()`

* Objective c bridge to talk to FS

* Working objc bridge

The bridge works now, and we can now access the iOS file system using the native objective-c APIs instead for proper values, including on the simulator.

* Isolate `icrate` for `ios` deployments only

* Working Stats for Android

* Clean Up + `pnpm format`

* Fix to FSInfoResult Type

Due to the RNFS fork change, I had to change the types to make it so it doesn't fail building and CI.

* iOS Device Name Fix
2024-03-06 06:46:22 +00:00
Vítor Vasconcellos
393a907b57 Update github actions due to nodejs 16 deprecation (#2107)
* Update github actions due to nodejs 16 deprecation
 - Replace archived actions-rs/clippy-check with maintained fork actions-rs-plus/clippy-check
 - Replace redhat-actions/push-to-registry with updated fork Eusebiotrigo/push-to-registry
 - Point redhat-actions/buildah-build and softprops/action-gh-release to current master to fix nodejs deprecation

* Build the correct ios core rust arch for CI runs

* Build ios app for the same arch as the host in Mobile CI

* Some changes to try and make cache-factory faster and avoid failing so much

* Add trigger to run cache-factory on pull requests when there are changes to itself

* Attempt to fix sed usage on macOS

* Don't treat warning as errors

* Fix windows

* Fix windows 2

* Use target ad cache key for rust to differentiate between macOS x86_64 and arm64

* Use faster/better linkers to compile for macOS, Linux and Windows

* Fix missing shell in action

* Fix typo

* Fix missing shell in action 2

* Fix mold download
 - Replace bsdtar with plain tar

* Fix permission denied when extracting mold

* Remove zld

* Don't restore cache for rustfmt
 - Remove target symlink to C:/ in an attempt to speed-up windows CI

* Fix typo

* Restore target symlink on windows
 - Removing it didn't make CI faster

* Run Mobile on macos-14
2024-02-21 09:27:40 +00:00
ameer2468
c2aa4cc292 [MOB-52] Tabs animation (#2020)
* tabs animation

Update mobile-ci.yml

rive app requires macos 13.1

Update mobile-ci.yml

Update mobile-ci.yml

rive requires 14.0 ios deployment target

attempt to set deployment target to ios 14.0 for rive app

update target to 14.0

* downgrade packages for expo compatibility, fix test, touchable change to avoid ripple effect

* notes
2024-01-31 12:25:18 +00:00
Vítor Vasconcellos
2e6e00bc6d [ENG-1184, ENG-1286, ENG-1330] Rework native dependencies (+ deb fixes) (#1685)
* Almost working

* Downgrade libplacebo
 - FFMpeg 6.0 uses some now removed deprecated functions

* Use -Oz for zimg

* Fix CI script to run the new ffmpeg build script

* Fix heif step name + Ignore docker cache while building in CI

* Fix Opencl build on linux

* Fix adding incorrect -target argument to linker
 - Update zig for windows target

* Disable opengl for ffmpeg, it only uses it as an outdev, not for processing
 - Disable opengl and directx for libplacebo, ffmpeg only supports vulkan when using it
 - Add WIN32_LEAN_AND_MEAN to global cflags to optimize windows api usage
 - Fix 99-heif.sh incorrect bsdtar flag

* Remove WIN32_LEAN_AND_MEAN from global CFLAGS as that was breaking OpenCL build
 - Fix Dockerfile step for cleaning up the out dir
 - Improve licensing handling

* x86_64 windows and linux builds are working

* Fix aarch64 build for windows and linux

* Fix symbol visibility in linux builds
 - Fix soxr failing to download due to sourcefourge
 - Only patch zimg on windows targets
 - Tell cmake to hide libheif symbols

* Fix Linux .so rpath
 - Add lzo dependency
 - Publish source for the built libs
 - Add warning for missing nasm in tauri.mjs
 - Remove ffmpeg install from setup.sh
 - Add download logic for our linux ffmpeg bundle in preprep.mjs

* Remove jobs, docker doesn't support this

* Fix typing

* Change ffmpeg references to native deps
 - Rename FFMpeg.framework to Spacedrive.framework
 - Centralize the macOS native deps build with the windows and linux one
 - Change the preprep script to only download our native deps
 - Remove old macOS ffmpeg build scripts

* Compress native deps before creating github artifact
 - The zip implementation for github artifact does not mantain symlinks and permissions
 - Remove conditional protoc, it is now always included

* Don't strip dylibs, it was breaking them
 - Only download macOS Framework for darwin targets
 - Fix preprep script
 - Improve README.md for native-deps
 - Fix not finding native-deps src

* Attempt to fix macOS dylib

* Fix macOS dylibs
 - Replace lld.ld64 with apple's own linker
 - Add stages for building apple's compiler tools to use instead of LLVM ones

* Ensure sourced file exists

* All targets should build now
 - Fix environment sourcing in build.sh
 - Some minor improvements to cc.sh
 - Fix incorrect flag in zlib.sh
 - Improve how -f[...] flags are passed to compiler and linker
 - Add more stack hardening flags

* We now can support macOS 11.0 on arm64

* Improve macOS Framework generation
 - Remove installed unused deps
 - Improve cleanup and organization logic in Dockerfile last step
 - Move libav* .dll.a to .lib to fix missing files in windows target
 - Remove apple tools from /srv folder after installation to prevent their files from being copied by other stage steps
 - Create all the necessary symlinks for the macOS targets while building
 - Remove symlink logic for macOS target from preprep.mjs

* Remove native-deps from spacedrive repo
 - It now resides in https://github.com/spacedriveapp/native-deps
 - Modify preprep script to dowload native-deps from new location
 - Remove Github API code from scripts (not needed anymore)
 - Add flock.mjs to allow running tauri.mjs cleanup as soon as cargo finishes building in linux

* Handle flock not present in system
 - Allow macOS to try using flock

* Fix preprep on macOS

* Add script that patch deb to fix errors and warnings raised by lintian

* Fix ctrl+c/ctrl+v typo

* Remove gstreamer1.0-gtk3 from deb dependencies

* eval is evil

* Handle tauri build release with an explicit target in fix-deb.sh

* Preserve environment variables when re-executing fix-deb with sudo

* Only execute fix-deb.sh when building a deb bundle

* Improvements fix-deb.sh

* Improve setup.sh (Add experiemental alpine support)
2023-11-17 19:20:14 +00:00
Utku
bf93d97874 Fix mobile onboarding (#1670)
* attempts

* expo stop using spaces pls

* attempt

* you used to work without this sir

* fixed

* use serde instead of specta

* remove `serde(default`) and update bindings

* fix mobile

* fix types

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-10-26 17:38:15 +00:00
Brendan Allan
49cc098f32 [ENG-927, ENG-735, ENG-766] Fix Updater & Tauri 1.5 (#1361)
* custom updater with toasts

* new state management + updated router route

* tauri-specific update route

* ref

* update in prod only

* change 'Install' to 'Update'

* fix tsconfig

* desktop tauri

* remove tauri patch

* tauri 1.5

* tauri 1.5

* use tauri script

* native-deps

* Rework preprep and tauri script to better support tauri 1.5

* Update to tauri 1.5.1
 - Update workspace and apps/desktop dependencies
 - Fix mustache import, @types/mustache is not compatible with ES imports
 - Replace arm64 with aarch64 in machineID, they should be treated the same and this simplyfies the code

* Fix tauri updater not building due to missing key
 - Fix dmg background not being found
 - Generate an adhoc key for tauri updater with it is enabled and the user is doing a prod build

* Fix ctrl+c/ctrl+v typo

* Normalie @tanstack/react-query version through workspace
 - Use undici in scripts instead of global fetch
 - Fix typecheck

* Fix linux prod and dev builds
 - Improve error handling in tauri.mjs

* Normalize dev deps in workspace
 - Improve linux shared libs setup

* Fix CI and server docker

* Fix windows
 - Remove superfluous envvar

* Attempt to fix server, mobile, deb and release updater

* Attempt to fix deb and mobile again
 - Fix type on deb dependency
 - Enable release deb for aarch64-unknown-linux-gnu

* Github doesn't have arm runners
 - Fix typo in server Dockerfile

* Publish deb and updater artifacts

* remove version from asset name

* update commands

* log release

* Some logs on updater errors

* show updater errors on frontend

* fix desktop ui caching

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-10-10 07:30:56 +00:00
Vítor Vasconcellos
d75a6e9b12 [ENG-1054] libheif for Linux (plus fixes) + Webp update (#1405)
* Move postinstall script to a preprep script
 - Fix libheif crate failing to build with our libheif
 - Rework CI due to postinstall to preprep changes

* Linux heif build script + Update webp

* Fix ctrl+c/ctrl+v bug

* Improve libheif linux script
 - Add support for linux aarch64
 - Add CI workflow to build libheif for linux
 - Some other misc fixes

* Fix libheif CI requires sudo

* Fix wrong path for libheif build.rs override in Windows

* Fix wrong path manipulations in libheif build script

* 🤦

* Use ubuntu-latest in libheif action
 - Specify glib version in target triple to support old distros
 - Fix libheif artifact publishing

* Fix artifact upload path again

* Add musl support for libheif
 - Remove unused files from libheif artifact
 - Add setup logic for libheif in postinstall script

* Build libheif for linux as a shared lib

* Fix meson not building the correct arch
 - Add logic to get git branch from githubs CI env vars

* libheif finnaly works on linux
 - Make spacedrive binary rpath point to where appimage and deb expects our libs to be
 - Add some logic to tauri.js to convince tauri to bundle our shared libs
 - Work-around appimage bundling step breaking sometimes
 - Add logic to handle sigint in tauri.js to ensure we clean up after ourselves
 - Rename postinstall.mjs to setup.mjs
 - Add logic to setup.mjs to point our dev build to our shared libs in linux

* Fix windows dekstop dev
 - Rename setup.mjs to preprep.mjs

* test cache-factory

* Fix preprep script not parsing the cross compilation target triple and always using the host info to download dependencies
 - Fix action env vars not being correctly passed
 - Remove possibility to pass multiple targests to rust action

* Don't compile mobile crates on desktop targets

* Remove cache-factory pull_request trigger

* remove patched tauri cli

* Use git plumbing command to get remote branch name
 - Fallback to reading .git/HEAD if remote name was not retrieved

* fix type

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-10-03 17:31:33 +00:00
Oscar Beaumont
ae4c3d0e6b Fix Android bridge + drop IOS and Android projects (#1336)
* Goodbye Android Project, you won't be missed

* fix ignore ignoring Kotlin code

* Backport IOS `Podfile` + move Rust crates into `modules/sd-core`

* Workaround for issue in OpenSSL build script

* Fix IOS

* Migrate IOS bridge into Expo Native Module

* Optimistic update for library creation

* useFrameworks static

* add key to infoplist

* more keys

* fix `Podfile`

* Remove IOS project from repo

* fix mobile ci

* woops wrong place

* clean package.json

* Make Android's Rust code build again

* Add back in `if_watch` patch for IOS

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-09-13 10:07:26 +00:00