Commit Graph

45 Commits

Author SHA1 Message Date
Vítor Vasconcellos
b48da0965a [ENG-270, ENG-213] Stabilize release pipeline (#768)
* Attempt at building our own ffmpeg dylibs for macOS
 - Create dockerfile to setup osxcross and required host dependencies for cross-compiling ffmpeg for macOS
 - Create script to setup required macOS dependencies and build ffmpeg for both x86_64 and aarch64

* Improve dockerfile to better use docker's cache
 - Attempt at fixing arm64 build (still not there yet)

* ARM64 sucessfully compiles \o/
 - Use clang for everything (e.g. linker, assembler, ...)
 - Remove -lc++ because it is now handled internally thanks to a patch
 - Apply 3 open PRs to osxcross as external patches to solve some of our problems
 - PR 180 reduces the macports dependencies size by prefering non universal libs for x88
 - PR 314 adds the -lc++ flag for macOS SDKs >= 11.1
 - PR 379 fix macports pulling incorrects deps for SDK11 and adds support for SDK13 (future proofing, as it does nothing for us rigth now)

* Fix relink and copy logic in ffmpeg build script
 - Ensure ffmpeg and dependencies dylibs are correctly relinked and copied over to the output directory
 - Copy includes from ffmpeg and dependencies to allow for dev building against these dylibs
 - Set up symlinks for all dylibs to the common `lib` directory
 - Remove superfluous linker and compiler options
 - Add `-headerpad_max_install_names` linker option to accommodate the required relink changes
 - Remove `--disable-fft` for the arm64 build, as it doesn't seem to be required nowadays
 - Allow `swresample` and `avdevices` to be compiled as they seem to be expected to exist by `ffmpeg-sys-next`
 - Disable ffmpeg program compilation
 - Fix dylibs id to match their new expected location
 - Fix ffmpeg dylibs inter-linkage not being updated
 - Add custom patch for osxcross macports to load already downloaded files from cache
 - Remove the requirement for specifying the darwin version in the Dockerfile
 - Enable docker cache for macports
 - Move arm min macOS version ARG to the top

* Attempt at building ffmpeg with github actions
 - Split osxcross setup into it's own Dockerfile to allow exporting it as a image to Dockerhub
 - Create a github action to build ffmpeg
 - Add osxcross patch for PR 372 for future proofing (Add support for macOS SDK 13.1)

* Remove incompatible step name

* Change FFMPEG_VERSION to string to avoid yaml weird type casting

* Start to use our ffmpeg build in the macOS
 - Adjust the macOS setup script to download our ffmpeg build, also replace installing protobuf compiler from brew with grabbing it from it's own repository
 - Some misc improvement to the *nix setup script
 - Fix ffmpeg build script not copying static libs (for when we may want to statically link ffmpeg)
 - Fix ffmpeg build script not taking into acount possible libraries that are multiple paths deep in /lib
 - Update CI to actions/checkout@v3, v2 is deprecated
 - Update ffmpeg build CI to not xz the artifacts, as github already zips it anyway

* Fix macOS setup not finding the ffmpeg artifact due to a jq script mistake
 - Comment out embeeding static libs in ffmpeg artifact for now, as they take too much space
 - Remove incorrect dockerfile settings from editorconfig

* Fix protobuf download in macOS
 - Fix Framework directory creation not following script's dir but cwd
 - Improve download logic to avoid being rate limited on failures
 - Return to using xz for ffmpeg build, as it saves quite a bit of space

* Replace `OSTYPE` with `$(uname)` for a more standardized way to check which os is running
 - Fix mobile check using `==` instead of `=` in test command

* Fix syntax error in bash 3.x (which is the default in macOS)
 - Fix jq not downloading all deps and being placed in the incorrect directory

* Fix incorrect install_name_tool arguments
 - Remove leading './' when referencing ffmpeg libs during build

* Fix protoc not being executable
 - Fix some error messages

* jq throw input when it is invalid
 - Enable debug log in setup script when running in CI

* Add a dev.js build script for desktop app
 - Remove dll copy from setup-system.ps1, now done by dev.js
 - Minor rework to build.js to improve signals and stdio handling

* Configure cargo to use FFMPEG_DIR and PROTOC envvars on macOS

* Pass GITHUB_TOKEN to macOS setup script in CI

* Replace simple curl with gh_curl in github request

* Fix some ctrl+c/ctrl+v typos

* Add missing ffmpeg deps
 - Fix missing symlinks for ffmpeg libs in the output_dir/libs
 - Change arm64 minimum macOS target to 11.2 due to libbrotli only being available precompiled to this version
 - Add more options to ffmpeg configure in an attempt to reduce its size + improve performance

* Rework ffmpeg build script to create a FFMpeg.framework instead of loose dylibs
 - Disable unused postproc and enabled missing required protocols in ffmpeg configuration
 - Adjust Dockerfile and setup-script to handle the new FFMpeg.framework

* Fix setup script incorrect linking logic for FFMpeg.framework

* Forgot to remote the leading path in the lib/header linking logic for FFMpeg.framework

* Enable size optimization for ffmpeg
 - Remove Frameworks directory before setting it up in setup-system.sh

* Revert lib id to use `@executable_path/../Frameworks/` instead of `@rpath`
 - `@rpath` breaks dev builds
 - Add logic to create less versioned versions of dylibs in setup-system script, due to it being required for compiling ffmpeg-sys-next (not required during runtime, just for compilation)
 - Add scripts/deps to gitignore

* Fix ffmpeg build script not copying unversioned symlinks to FFMpeg.framework
 - Remove unecessary unversioned symlink logic from setup-system
 - Add new build scripts for dev and prod that setup the environment and tauri to correctly point to FFMpeg.framework
 - Rework windows setup-system script to follow the same logic as in macOS setup-system

* Attempt at fixing windows setup script
 - Add powershell editorconfig

* Attempt to fix Windows CI

* Fix some mistakes in the build scripts
 - Replace toml lib with something more used and maybe better?
 - Attempt to fix Windows CI

* Fix some more problems with the build script

* Another attempt at fixing windows CI

* Some more fixes for the windows setup script

* Add build.js step to release CI
 - Implement patch only mode for build.js when running under CI
 - Implement appeding extra env in cargo config in env.js
 - Only run windows dlls copy in dev in env.js
 - Another attempt at fixing windows CI

* Maybe this will make the windows CI happy?

* Windows CI why dont you work, whyyy?

* Try to apease the Windows CI Gods
 - Disable updater build in macOS, due to tauri-apps/tauri#3933

* Fix build.js extra env vars
 - Uncomment parts of setup-system.ps1 to check if it keeps working

* Uncomment another section and see if windows CI will complain

* uncomment some more of setup-system.ps1

* Use Start-Process for robocopy instead of executing it directly
 - Only raise an error if robocopy exists with a status code >=8
 - Revert build.js macOS specific bundles change
 - Disable updater for now

* Remove updater from cargo.toml

* Add -PassThru to Start-Process to ensure we get and Process obj back

* Comment out more of updater

* Improve documentation for both of the docker containers
 - Fix a harcoded install_name_tool call in the ffmpeg-build-macos.sh script

* Configure tauri to copy ffmpeg dlls to windows prod version
 - Fix windows dlls copy on dev
 - Stop replacing tauri.conf.json, except on CI, for prod builds and use a tauri.conf.patch.json for any required changes
 - Don't unset macOSPrivateApi on prod build, we need it for the transparent background
 - Fix dev.js not exiting when the spawned command was over

* Fix windows not find protoc
 - Fix missing import in env.js

* Fix Windows Dlls copy logic for prod build
 - Fix dev script failing due to missing env
 - Implement error handler to spawned process

* Format + Fix pnpm-lock

* Fix video thumb generation failing on Windows due to bad path to CString conversion logic

* Fix mobile build
 - Greatly improve windows setup-script error handling
 - Install LLVM with winget instead of downloading exe from github

* Improve CI a bit
 - Fix mobile CI not using github token to avoid being rate limited

* Fix pnpm not accesible to actions/setup-node

* Skip pnpm check on CI

* Fix pnpm skip not working
 - Fix pnpm cache check missing a step

* Only lock action to major versions

* CI really doesn't like running robocopy directly

* Attempt to build our own patched tauri bundler

* Trigger new workflow

* Fix attempt for patched tauri action tests
 - Disable cache for patched tauri action
 - Enable package publishing for patched tauri action

* Attempt fix patched tauri publish

* Remove commented code from patched tauri workflow
 - Replace deprecated `actions-rs/toolchain` with `dtolnay/rust-toolchain`

* Fix url patching logic for publishing step of patched tauri package

* Fix prepublishOnly command for patched tauri package

* Adjust patched tauri package package.json

* Also patch the specific os/arch sub packages

* Instruct napi to stop doing github releases

* Patch package.json befor build

* Re-enable updater
 - Patch tauri cli to workaround a tauri bug that prevented use from bundling frameworks in the updater
 - Update tauri/api to 1.3 and tauri/cli 1.3.1
 - Add target triple in the release artifacts name
 - Use ubuntu-20.04 instead of ubuntu-latest
 - Configure release workflow to build to all supported platforms
 - Replace `tauri-apps/tauri-action` with simply calling `pnpm desktop build`
 - Simplify tauri-patched-cli-js to only build the required macOS binaries
 - Unify build.js and dev.js in a tauri.js build script, which now acts as an all around wrapper for the tauri/cli

* Restore incorrectly removed step from `tauri-patched-cli-js`

* Disable auto-run of tauri-patched-cli-js
 - Update setup-system.sh with the latest tauri-patched-cli-js artifacts

* Forgot to mkdir bin

* Fix pnpm-lock.yaml not being up to date
 - Remove pnpm deps caching

* Fix variable name conflict

* Enable the updater build conditionally
 - Enable release workflow to test it

* Fix artifacts not being published
 - Change macOS on arm64 minimumSystemVersion to 11.2
 - Commented out rust cache to check if this is the culprit to the low memory problem on windows runner
 - Add some debug log to tauri.js

* Fix release workflow artifact uploading

* Specify which bundles to build on each platform
 - Attempt fix apple arm build
 - Don't include internal deb archives in linux updater artifact

* Attempt to fix apple arm build
 - Fix dmg background not applying

* Fix incorrect semver usage

* Patch swift-rs to see if that fix apple arm64 release build

* Update swift-rs submodule
 - Remove unused deps lodash.merge

* Configure so only release workflow runs without cache
 - Improve tauri.js logs

* Ignore `crates/swift-rs`

* Revert "Ignore `crates/swift-rs`"
- Remove `crates/swift-rs` before running `cargo fmt`

This reverts commit 851bd84373.

* Github CI input/output are always string
 - Attempt to fix Windows Clippy

* Fix CI syntax error

* Fix mobile CI
 - Disable pnpm deps caching
 - Disable rust targets caching on Windows
 - Configure cache factory to run for all targets

* Remove `crates/swift-rs`

* Attempt fix Mobile CI
 - Enable cache factory run in pull_request to test it
 - Specific more path for the CI run to ignore
 - Specify path that will trigger the cache factory run
 - Some cleanup in the setup-pnpm action

* Remove restore-cache options
 - Add prefix key to cache-factory
 - Fix LLVM cache-hit check
 - Add libheif to FFMpeg.framework

* Remove submodule clone from actions

* Add fake deps to ffmpeg build dockerfile
 - Fake install deps that aren't really necessary to build ffmpeg
 - Fail ffmpeg build script if it can't find a macports deps
 - Copy libheif headers to Framework

* Remove automatic trigger for relase and pull request trigger for cache-factory actions

* Remove pnpm setup from Cache factory
 - Cache factory is exclusivly for rust deps caching, no need to setup Node and pnpm with it

* Re-enabled fail-fast in Github CI
2023-05-15 18:02:56 +00:00
Utku
173fb421e6 bump pnpm (#798)
* bump pnpm

* bump ci pnpm version

* almost forgot a pnpm ci lol

* Update pnpm version installed during windows setup-script

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2023-05-05 22:14:49 +00:00
Vítor Vasconcellos
164fcc34d3 Fix Windows CI + format (#767)
* Attempt to fix Windows CI:
- Ensure version 15 of LLVM is installed in Windows CI (due to` ffmpeg-sys-next`)
- Use `GITHUB_TOKEN` in setup-system.ps1 to avoid rate-limiting by github api in Windows CI
- Add `yaml` settings to `.editorconfig`
- Create a symbolic link from `.prettierignore` to `.eslintignore`
- Move `eslint` root to package root config
- Enable `fileNesting` to reduce dotfile clutter in vscode
- Pass `--strict` to apps/releases `lint` command to avoid making interactive queries and hanging `pnpm lint`
- Format with `prettier` and `eslint`
- Remove empty interface/hooks/useMediaQuery.ts

* Replace unmaintained `actions-rs/toolchain` with `dtolnay/rust-toolchain`
 - Change setup-system.ps1 to avoid installing pnpm in CI
 - Change `pnpm/action-setup` to lock only o major versions of the action
2023-04-27 07:32:15 +00:00
Brendan Allan
6c7083cfb1 Update server (#744)
* release workflow

* tauriScript

* pnpm deps

* unify rust cache keys

* bruh

* basic manual updater stuff

* move updater override

* use existing publishing action

* use debug builds for desktop

* cache release builds too

* use release builds instead

* add workflow dispatch trigger

* remove branch from release workflow trigger
2023-04-21 10:02:31 -07:00
Utku
e83d26372e Roadmap (#692)
* update roadmap

* add android & ios to footer

* Limit CI to run only when needed

* fix paths-ignore syntax

* update key manager description
2023-04-12 02:36:16 +00:00
Utku
abeebcfd88 Mobile CI with Tests (#631)
* stuff

* stuff (but for android)

* test mobile ci

* move pnpm up

* install ndk

* add ios & cleanup

* onboarding flow test

* test ci

* fixes, cleanup, caches

* why you do this cargo

* fix pnpm-lock

* add path to build rust script?

* ci is fun

* yolo

* fix broken flow..

* fix pnpm

* probably not gonna work

* test x2

* use real branch of pcr

* android emulator and try ios x2

* Use react native architectures, instead of all.

* override architecture to speed up android build

* protoc & build android on macos too

* fix java ndk

* android gradle

* disable ios for now

* use simulator sdk & debug configuration on ios build

* cleanup

* avd test

* fix avd settings.

* only build for x86_64 on ci

* Fix ios build

* Add IOS testing

* maestro script

* ios release build & wait for library creation

* clean up and disable android for now

* fix pnpm-lock

* Add concurrency to cancel previous runs

* fix pnpm-lock

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-04-03 08:12:28 +00:00
Brendan Allan
828ea8c498 CI Cache Factory (#593)
* introduce cache factory

run cache factory on all branches

add names

use single quotes

add setup and prisma generation

add pnpm install

don't install cargo-watch in ci

make composite setup script

use env var for windows ci

undo all powershell stuff

* better windows ci

* don't install cargo-watch in ci

* only run cache factory on main

* i'm silly

* debug only ci

* check runner instead of matrix in setup action

* don't save cache output most of the time

* no more os-specific features

* cache entire workspace

* only cache on main again

* test instead of buidl in cache factory

* smaller ci + testing

* don't individually test each crate

* remove unnecessary needs

* smaller formatting and more clippy

* undo rustfmt changes

* disable tests
2023-03-06 06:10:55 -08:00
jake
c21743f44e [ENG-361] Crypto crate docs and tests (#572)
* add hashing tests

* add encryption/decryption tests

* remove `reason`

* add file header tests (preview media deserialization is broken)

* fix keyslot reading bug

* add sd-crypto testing to ci

* add tests/constants for all hashing algorthms and param levels

* add blake3-kdf tests

* use `const` arrays for storing expected output

* test for `5MiB` `encrypt_streams` and `decrypt_streams`

* add invalid/mismatched nonce tests

* update `primitives` docs

* remove erroneous `,`

* grammar tweaks

* add errors to `#[should_panic]`

* cleanup `stream` tests

* cleanup hashing tests a little

* function docs
2023-02-16 11:42:30 +00:00
Brendan Allan
842ea113e7 update name of eslint job 2023-01-22 16:14:59 +08:00
Brendan Allan
3e4ca13766 simplify eslint configs + add CI job 2023-01-22 16:01:15 +08:00
Ericson "Fogo" Soares
3b142c6180 ENG 223 Location Awareness (#468)
* Introducing location online/offline checks and location relink

* Some initial drafts for location watchers

* Location metadata relink and add library

* Many improvements at job system
Now using prisma batching at identifier job
Removing blocking I/O from extension subcrate
Implementing lazy generation of thumbnails
New current directory identifier job to be used on light rescans

* Some optimizations on identifier and object validator jobs

* merge jamie's identifier PR

* fully repaired identifier job

* properly hooked up object kind

* inspector fix

* fix video badge

* small improvements to libraries settings

* identifier and inspector improvements

* fix feature flags and hook up context menu location utilities

* BETTER CONTEXT MENU x100

* test-files

* style tweaks

* new icon designs

* manifest

* fix thumbnails on web

* media data

* New Location Watcher and some minor fixes

* disable broken media_data extractor, wip

* wip

* function name fix

* Fixing pnpm prep and some warnings

* Solving a race condition beetween indexer job and FS event handlerSome other minor warnings

* Generating thumbnails on watcher

* Remove event handler on watcher

* Some initial works on modify events and other small fixes

* File update event

* Trying to be more generic with used events and some tests to validate our assumptions

* Turning on location metadata file

* Introducing core unit tests on CI pipeline

* Submiting new unit test assumptions to validate on windows CI

* Fixing unit tests

* Fixing unit tests again

* Fixing unit tests

* Fixing unit tests for macos

* Fixing unit tests for macos again

* New structure for platform dependent event handling
Implementing event handlers for Linux and MacOS

* minor fixes + rustfmt + clippy

* Windows event handling

* Introducing a feature gate to only use location watching on desktop app for now

* Putting more stuff behind feature gates to avoid warnings

* Adding feature to cargo test on CI

* Changing some debug logs to trace logs and removing Jamie specific stuff

* Make location removal from manager less async

* fix build when "location-watcher" feature disabled

* fix types + clippy

* make location manager non-static

* remove uses of `to_string_lossy`

* more invalidate_query calls

* Expose `library_ctx` directly to avoid needless clones

* New materialized_path handling for directories

* Removing cascade delete between file_path and object
- Some other minor stuff

* remove unused `CurrentDirFileIdentifierJob`

Co-authored-by: Jamie Pine <ijamespine@me.com>
Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-31 00:53:24 +08:00
Utku Bakir
fcbc92203b Bump node to lts (v18) 2022-12-14 12:43:02 +03:00
Utku
17865a6c96 Mobile - Settings & Tags (#479)
* Sync tailwind configs

* Switch to new colors part 1

* new colors part 2

* switched to new colors

* settings screens template

* settings progress

* Setting header titles

* Refactor settings icon

* Fix tsconfig relative path issue

* Move mobile eslint config to @sd/config

* Add no-restricted-imports rule to eslint

* rename TextInput to Input & tailwind font sizing

* General Settings screen + Card component

* Library settings screen & delete lib dialog

* autoform hook

* Mini tweaks and new packages

* Exclude android & ios from search

* Add Switch (Input)

* Library General settings screen

* Refactor settings + Switch and Switch Container

* Locations screen & delete and rescan locations

* Rename folder + small tweaks

* Make things Swipeable

* Create Tag Dialog with color picker

* Upgrade to Expo 47

* More tag stuff

* fix pnpm lock

* regen pnpm-lock

* Change CI node version to 16

* Move `isVideo` into shared package

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-13 15:18:03 +08:00
Brendan Allan
008b0eb72b separate fetch & check 2022-10-05 17:20:40 +08:00
Brendan Allan
49d4dbb0dd Improve CI Speed (#398)
* use different cache

* cache prisma client in rustfmt

* cleanup prettier config

* build server image in sub-action

* server image + prisma client custom actions

* remove lefthoook checks that are in ci

* fix build-server-image

* rename Clippy job

* rename build core job

* use custom rust caching action

* fix rust cache shared key

* cleanup custom action job names

* remove platform check in clippy ci

* setup pnpm in clippy ci

* update mobile package name

* make ios compiling conditional

* use pnpm cache

* run clippy in release

* force use release for all jobs
2022-10-04 18:45:54 -07:00
Brendan Allan
74b42bf45c [ENG-84] Sync library (#394)
* new sql lib stuff

* add sync stuff + rename all crates

* build sd-core

* add sync/example/dist to source

* fix sync example in monorepop

* appease clippy

* update lockfile

* update commit hooks

* fix typescript

* fix typescript build

* please rustfmt
2022-10-04 05:25:12 -07:00
Brendan Allan
aafba4a7bf run rustfmt in ci 2022-10-04 20:03:49 +08:00
Brendan Allan
8a7cf83f46 use proper workflow syntax 2022-09-28 11:42:05 +08:00
Brendan Allan
0f8d6b75e9 expand prisma client cache key 2022-09-28 11:39:04 +08:00
Brendan Allan
69a6de65d5 remove needs: typescript from rust ci 2022-09-23 13:53:56 +08:00
Brendan Allan
18079ba378 Use PCR for migrations (#385)
* use pcr migrations branch

* use 0.6.2 branch with migrations

* use latest prisma stuff

* allow force reset of db in dev

* remove .spacedrive file

* update rspc in apps/server

* use rspc 0.0.5 in all crates

* add os to prisma client cache key

* add runner os to clippy prisma cache
2022-09-22 00:50:21 -07:00
voletro
afd33b2d4c Remove Action for LLVM, use action that is already built in. Set GH env variables. 2022-07-06 12:28:28 +10:00
Oscar Beaumont
6ee4974068 remove Cargo frozen flag 2022-07-05 23:55:44 +08:00
voletro
b783c32a04 Added headless mode, added LLVM installer to ci. 2022-07-05 11:21:42 +10:00
Oscar Beaumont
825fd8ecf5 fix clippy workflow (hopefully) 2022-06-20 11:27:37 +08:00
Oscar Beaumont
0c370f5373 remove frozen flag from cargo fetch 2022-06-19 01:52:03 +08:00
Oscar Beaumont
461c7b20d8 Fix CI (#242) 2022-06-14 22:27:03 +08:00
Lars Gyrup Brink Nielsen
8c33edf903 ci: update pnpm/action-setup to v2.2.2 2022-06-03 10:38:42 +02:00
Lars Gyrup Brink Nielsen
380de162ea ci: update actions/setup-node to v3 2022-06-03 10:20:34 +02:00
Lars Gyrup Brink Nielsen
bf0b48a08e ci: update to actions/checkout v3 2022-06-03 10:19:59 +02:00
Harry Hopkinson
f08e30d509 Change pnpm -r exec tsc to pnpm typecheck 2022-05-15 10:35:12 +01:00
Oscar Beaumont
8a2e9efdd6 add workflow_dispatch to CI flow 2022-05-04 20:59:20 +08:00
Brendan Allan
88e9428f3e ci: fix 2022-05-01 18:36:59 +08:00
Brendan Allan
84a91f1cce ci: use sharedKey 2022-05-01 17:46:44 +08:00
Brendan Allan
adeea160c4 ci: use original cache action + build server 2022-05-01 17:36:23 +08:00
Brendan Allan
fcf27e0cf5 ci: make nicer 2022-05-01 16:23:09 +08:00
Brendan Allan
4e3109ce84 don't build prisma-client-rust separately 2022-05-01 13:12:57 +08:00
Brendan Allan
7c6501da38 use pnpm 7 2022-05-01 12:33:42 +08:00
Brendan Allan
4f81d32e08 tsconfig refactor 2022-05-01 12:17:55 +08:00
Brendan Allan
7c4911f0e0 ci: don't build + publish server on pr 2022-05-01 09:59:39 +08:00
Brendan Allan
ef7e915151 use brendonovich/rust-cache 2022-05-01 09:48:35 +08:00
Brendan Allan
1b720ea873 Merge remote-tracking branch 'upstream/main' into new-ci 2022-05-01 09:36:12 +08:00
Harry Hopkinson
d6bb002586 Linted Files according to the .prettierrc. 2022-04-24 21:16:55 +01:00
Oscar Beaumont
44e254dff3 @sd/server in Docker and deploy to Kubernetes 2022-04-24 14:55:20 +08:00
Brendan Allan
64b872715c .yml supremacy 2022-04-23 23:29:03 +08:00