Files
spacedrive/.github/scripts/ffmpeg-macos/ffmpeg-build-macos.sh
Vítor Vasconcellos b48da0965a [ENG-270, ENG-213] Stabilize release pipeline (#768)
* Attempt at building our own ffmpeg dylibs for macOS
 - Create dockerfile to setup osxcross and required host dependencies for cross-compiling ffmpeg for macOS
 - Create script to setup required macOS dependencies and build ffmpeg for both x86_64 and aarch64

* Improve dockerfile to better use docker's cache
 - Attempt at fixing arm64 build (still not there yet)

* ARM64 sucessfully compiles \o/
 - Use clang for everything (e.g. linker, assembler, ...)
 - Remove -lc++ because it is now handled internally thanks to a patch
 - Apply 3 open PRs to osxcross as external patches to solve some of our problems
 - PR 180 reduces the macports dependencies size by prefering non universal libs for x88
 - PR 314 adds the -lc++ flag for macOS SDKs >= 11.1
 - PR 379 fix macports pulling incorrects deps for SDK11 and adds support for SDK13 (future proofing, as it does nothing for us rigth now)

* Fix relink and copy logic in ffmpeg build script
 - Ensure ffmpeg and dependencies dylibs are correctly relinked and copied over to the output directory
 - Copy includes from ffmpeg and dependencies to allow for dev building against these dylibs
 - Set up symlinks for all dylibs to the common `lib` directory
 - Remove superfluous linker and compiler options
 - Add `-headerpad_max_install_names` linker option to accommodate the required relink changes
 - Remove `--disable-fft` for the arm64 build, as it doesn't seem to be required nowadays
 - Allow `swresample` and `avdevices` to be compiled as they seem to be expected to exist by `ffmpeg-sys-next`
 - Disable ffmpeg program compilation
 - Fix dylibs id to match their new expected location
 - Fix ffmpeg dylibs inter-linkage not being updated
 - Add custom patch for osxcross macports to load already downloaded files from cache
 - Remove the requirement for specifying the darwin version in the Dockerfile
 - Enable docker cache for macports
 - Move arm min macOS version ARG to the top

* Attempt at building ffmpeg with github actions
 - Split osxcross setup into it's own Dockerfile to allow exporting it as a image to Dockerhub
 - Create a github action to build ffmpeg
 - Add osxcross patch for PR 372 for future proofing (Add support for macOS SDK 13.1)

* Remove incompatible step name

* Change FFMPEG_VERSION to string to avoid yaml weird type casting

* Start to use our ffmpeg build in the macOS
 - Adjust the macOS setup script to download our ffmpeg build, also replace installing protobuf compiler from brew with grabbing it from it's own repository
 - Some misc improvement to the *nix setup script
 - Fix ffmpeg build script not copying static libs (for when we may want to statically link ffmpeg)
 - Fix ffmpeg build script not taking into acount possible libraries that are multiple paths deep in /lib
 - Update CI to actions/checkout@v3, v2 is deprecated
 - Update ffmpeg build CI to not xz the artifacts, as github already zips it anyway

* Fix macOS setup not finding the ffmpeg artifact due to a jq script mistake
 - Comment out embeeding static libs in ffmpeg artifact for now, as they take too much space
 - Remove incorrect dockerfile settings from editorconfig

* Fix protobuf download in macOS
 - Fix Framework directory creation not following script's dir but cwd
 - Improve download logic to avoid being rate limited on failures
 - Return to using xz for ffmpeg build, as it saves quite a bit of space

* Replace `OSTYPE` with `$(uname)` for a more standardized way to check which os is running
 - Fix mobile check using `==` instead of `=` in test command

* Fix syntax error in bash 3.x (which is the default in macOS)
 - Fix jq not downloading all deps and being placed in the incorrect directory

* Fix incorrect install_name_tool arguments
 - Remove leading './' when referencing ffmpeg libs during build

* Fix protoc not being executable
 - Fix some error messages

* jq throw input when it is invalid
 - Enable debug log in setup script when running in CI

* Add a dev.js build script for desktop app
 - Remove dll copy from setup-system.ps1, now done by dev.js
 - Minor rework to build.js to improve signals and stdio handling

* Configure cargo to use FFMPEG_DIR and PROTOC envvars on macOS

* Pass GITHUB_TOKEN to macOS setup script in CI

* Replace simple curl with gh_curl in github request

* Fix some ctrl+c/ctrl+v typos

* Add missing ffmpeg deps
 - Fix missing symlinks for ffmpeg libs in the output_dir/libs
 - Change arm64 minimum macOS target to 11.2 due to libbrotli only being available precompiled to this version
 - Add more options to ffmpeg configure in an attempt to reduce its size + improve performance

* Rework ffmpeg build script to create a FFMpeg.framework instead of loose dylibs
 - Disable unused postproc and enabled missing required protocols in ffmpeg configuration
 - Adjust Dockerfile and setup-script to handle the new FFMpeg.framework

* Fix setup script incorrect linking logic for FFMpeg.framework

* Forgot to remote the leading path in the lib/header linking logic for FFMpeg.framework

* Enable size optimization for ffmpeg
 - Remove Frameworks directory before setting it up in setup-system.sh

* Revert lib id to use `@executable_path/../Frameworks/` instead of `@rpath`
 - `@rpath` breaks dev builds
 - Add logic to create less versioned versions of dylibs in setup-system script, due to it being required for compiling ffmpeg-sys-next (not required during runtime, just for compilation)
 - Add scripts/deps to gitignore

* Fix ffmpeg build script not copying unversioned symlinks to FFMpeg.framework
 - Remove unecessary unversioned symlink logic from setup-system
 - Add new build scripts for dev and prod that setup the environment and tauri to correctly point to FFMpeg.framework
 - Rework windows setup-system script to follow the same logic as in macOS setup-system

* Attempt at fixing windows setup script
 - Add powershell editorconfig

* Attempt to fix Windows CI

* Fix some mistakes in the build scripts
 - Replace toml lib with something more used and maybe better?
 - Attempt to fix Windows CI

* Fix some more problems with the build script

* Another attempt at fixing windows CI

* Some more fixes for the windows setup script

* Add build.js step to release CI
 - Implement patch only mode for build.js when running under CI
 - Implement appeding extra env in cargo config in env.js
 - Only run windows dlls copy in dev in env.js
 - Another attempt at fixing windows CI

* Maybe this will make the windows CI happy?

* Windows CI why dont you work, whyyy?

* Try to apease the Windows CI Gods
 - Disable updater build in macOS, due to tauri-apps/tauri#3933

* Fix build.js extra env vars
 - Uncomment parts of setup-system.ps1 to check if it keeps working

* Uncomment another section and see if windows CI will complain

* uncomment some more of setup-system.ps1

* Use Start-Process for robocopy instead of executing it directly
 - Only raise an error if robocopy exists with a status code >=8
 - Revert build.js macOS specific bundles change
 - Disable updater for now

* Remove updater from cargo.toml

* Add -PassThru to Start-Process to ensure we get and Process obj back

* Comment out more of updater

* Improve documentation for both of the docker containers
 - Fix a harcoded install_name_tool call in the ffmpeg-build-macos.sh script

* Configure tauri to copy ffmpeg dlls to windows prod version
 - Fix windows dlls copy on dev
 - Stop replacing tauri.conf.json, except on CI, for prod builds and use a tauri.conf.patch.json for any required changes
 - Don't unset macOSPrivateApi on prod build, we need it for the transparent background
 - Fix dev.js not exiting when the spawned command was over

* Fix windows not find protoc
 - Fix missing import in env.js

* Fix Windows Dlls copy logic for prod build
 - Fix dev script failing due to missing env
 - Implement error handler to spawned process

* Format + Fix pnpm-lock

* Fix video thumb generation failing on Windows due to bad path to CString conversion logic

* Fix mobile build
 - Greatly improve windows setup-script error handling
 - Install LLVM with winget instead of downloading exe from github

* Improve CI a bit
 - Fix mobile CI not using github token to avoid being rate limited

* Fix pnpm not accesible to actions/setup-node

* Skip pnpm check on CI

* Fix pnpm skip not working
 - Fix pnpm cache check missing a step

* Only lock action to major versions

* CI really doesn't like running robocopy directly

* Attempt to build our own patched tauri bundler

* Trigger new workflow

* Fix attempt for patched tauri action tests
 - Disable cache for patched tauri action
 - Enable package publishing for patched tauri action

* Attempt fix patched tauri publish

* Remove commented code from patched tauri workflow
 - Replace deprecated `actions-rs/toolchain` with `dtolnay/rust-toolchain`

* Fix url patching logic for publishing step of patched tauri package

* Fix prepublishOnly command for patched tauri package

* Adjust patched tauri package package.json

* Also patch the specific os/arch sub packages

* Instruct napi to stop doing github releases

* Patch package.json befor build

* Re-enable updater
 - Patch tauri cli to workaround a tauri bug that prevented use from bundling frameworks in the updater
 - Update tauri/api to 1.3 and tauri/cli 1.3.1
 - Add target triple in the release artifacts name
 - Use ubuntu-20.04 instead of ubuntu-latest
 - Configure release workflow to build to all supported platforms
 - Replace `tauri-apps/tauri-action` with simply calling `pnpm desktop build`
 - Simplify tauri-patched-cli-js to only build the required macOS binaries
 - Unify build.js and dev.js in a tauri.js build script, which now acts as an all around wrapper for the tauri/cli

* Restore incorrectly removed step from `tauri-patched-cli-js`

* Disable auto-run of tauri-patched-cli-js
 - Update setup-system.sh with the latest tauri-patched-cli-js artifacts

* Forgot to mkdir bin

* Fix pnpm-lock.yaml not being up to date
 - Remove pnpm deps caching

* Fix variable name conflict

* Enable the updater build conditionally
 - Enable release workflow to test it

* Fix artifacts not being published
 - Change macOS on arm64 minimumSystemVersion to 11.2
 - Commented out rust cache to check if this is the culprit to the low memory problem on windows runner
 - Add some debug log to tauri.js

* Fix release workflow artifact uploading

* Specify which bundles to build on each platform
 - Attempt fix apple arm build
 - Don't include internal deb archives in linux updater artifact

* Attempt to fix apple arm build
 - Fix dmg background not applying

* Fix incorrect semver usage

* Patch swift-rs to see if that fix apple arm64 release build

* Update swift-rs submodule
 - Remove unused deps lodash.merge

* Configure so only release workflow runs without cache
 - Improve tauri.js logs

* Ignore `crates/swift-rs`

* Revert "Ignore `crates/swift-rs`"
- Remove `crates/swift-rs` before running `cargo fmt`

This reverts commit 851bd84373.

* Github CI input/output are always string
 - Attempt to fix Windows Clippy

* Fix CI syntax error

* Fix mobile CI
 - Disable pnpm deps caching
 - Disable rust targets caching on Windows
 - Configure cache factory to run for all targets

* Remove `crates/swift-rs`

* Attempt fix Mobile CI
 - Enable cache factory run in pull_request to test it
 - Specific more path for the CI run to ignore
 - Specify path that will trigger the cache factory run
 - Some cleanup in the setup-pnpm action

* Remove restore-cache options
 - Add prefix key to cache-factory
 - Fix LLVM cache-hit check
 - Add libheif to FFMpeg.framework

* Remove submodule clone from actions

* Add fake deps to ffmpeg build dockerfile
 - Fake install deps that aren't really necessary to build ffmpeg
 - Fail ffmpeg build script if it can't find a macports deps
 - Copy libheif headers to Framework

* Remove automatic trigger for relase and pull request trigger for cache-factory actions

* Remove pnpm setup from Cache factory
 - Cache factory is exclusivly for rust deps caching, no need to setup Node and pnpm with it

* Re-enabled fail-fast in Github CI
2023-05-15 18:02:56 +00:00

346 lines
10 KiB
Bash
Executable File

#!/usr/bin/env bash
# This script builds ffmpeg for macOS using osxcross.
# This script is heavly influenced by:
# https://github.com/FFmpeg/FFmpeg/blob/ea3d24bbe3c58b171e55fe2151fc7ffaca3ab3d2/configure
# https://github.com/GerardSoleCa/macports-ports/blob/6f646dfaeb58ccb4a8b877df1ae4eecc4650fac7/multimedia/ffmpeg-upstream/Portfile
# https://github.com/arthenica/ffmpeg-kit/blob/47f85fa9ea3f8c34f3c817b87d8667b61b87d0bc/scripts/apple/ffmpeg.sh
# https://github.com/zimbatm/ffmpeg-static/blob/3206c0d74cd129c2ddfc3e928dcd3ea317d54857/build.sh
set -euox pipefail
if [ "$#" -ne 2 ]; then
echo "Usage: $0 <target-arch> <macos-version>" >&2
exit 1
fi
if [ -z "$MACOSX_DEPLOYMENT_TARGET" ]; then
echo "You must set MACOSX_DEPLOYMENT_TARGET first." >&2
exit 1
fi
ARCH="$1"
MACOS_VERSION="$2"
set -- # Clear command line arguments
if [ "$ARCH" = "x86_64" ]; then
TARGET_CPU="x86_64"
TARGET_ARCH="x86_64"
set -- --enable-x86asm
elif [ "$ARCH" = "aarch64" ]; then
TARGET_CPU="armv8"
TARGET_ARCH="aarch64"
set -- --enable-neon --enable-asm
else
echo "Unsupported architecture: $ARCH" >&2
exit 1
fi
# Get darwin version and build compiler triple
DARWIN_VERSION="$(basename "$(realpath "$(command -v "oa64-clang")")" | awk -F- '{print $3}')"
TRIPLE="${ARCH}-apple-${DARWIN_VERSION}"
# Check macOS clang exists
if ! CC="$(command -v "${TRIPLE}-clang" 2>/dev/null)"; then
echo "${TRIPLE}-clang not found" >&2
exit 1
fi
export CC
# Get osxcross root directory
_osxcross_root="$(dirname "$(dirname "$CC")")"
# Check macports root exists
_macports_root="${_osxcross_root}/macports/pkgs/opt/local"
if ! [ -d "$_macports_root" ]; then
echo "macports root not found: $_macports_root" >&2
exit 1
fi
# Check SDK exists
_sdk="${_osxcross_root}/SDK/MacOSX${MACOS_VERSION}.sdk"
if ! [ -d "$_sdk" ]; then
echo "Invalid MacOS version: $MACOS_VERSION" >&2
exit 1
fi
# Gather all SDK libs
_skd_libs="$(
while IFS= read -r -d '' _lib; do
_lib="${_lib#"${_sdk}/usr/lib/"}"
_lib="${_lib%.*}"
printf '%s.dylib\n' "$_lib"
done < <(find "${_sdk}/usr/lib" \( -name '*.tbd' -o -name '*.dylib' \) -print0) \
| sort -u
)"
# Change cwd to the script directory (which should be ffmpeg source root)
CDPATH='' cd -- "$(dirname -- "$0")"
# Save FFmpeg version
FFMPEG_VERSION="$(xargs printf '%s' <VERSION)"
# Create a tmp TARGET_DIR
TARGET_DIR="$(mktemp -d -t ffmpeg-macos-XXXXXXXXXX)"
trap 'rm -rf "$TARGET_DIR"' EXIT
# Configure FFMpeg.
# NOTICE: This isn't autotools
# TODO: Metal suport is disabled because no open source toolchain is available for it
# TODO: Maybe try macOS own metal compiler under darling? https://github.com/darlinghq/darling/issues/326
./configure \
--nm="${TRIPLE}-nm" \
--ar="${TRIPLE}-ar" \
--as="$CC" \
--ld="$CC" \
--cc="$CC" \
--cxx="${TRIPLE}-clang++" \
--arch="${ARCH}" \
--objcc="$CC" \
--strip="${TRIPLE}-strip" \
--dep-cc="$CC" \
--sysroot="$_sdk" \
--cross-prefix="${TRIPLE}-" \
--ranlib="${TRIPLE}-ranlib" \
--prefix="${TARGET_DIR}" \
--arch="${TARGET_ARCH}" \
--cpu="${TARGET_CPU}" \
--target-os=darwin \
--pkg-config="${TRIPLE}-pkg-config" \
--pkg-config-flags="--static" \
--extra-ldflags="-Bstatic -headerpad_max_install_names" \
--extra-ldexeflags="-Bstatic" \
--extra-cxxflags="-xc++-header" \
--disable-alsa \
--disable-cuda \
--disable-cuvid \
--disable-debug \
--disable-doc \
--disable-htmlpages \
--disable-indevs \
--disable-libjack \
--disable-libopencore-amrnb \
--disable-libopencore-amrwb \
--disable-libpulse \
--disable-libxcb \
--disable-libxcb-shape \
--disable-libxcb-shm \
--disable-libxcb-xfixes \
--disable-manpages \
--disable-metal \
--disable-neon-clobber-test \
--disable-network \
--disable-nvdec \
--disable-nvenc \
--disable-openssl \
--disable-outdevs \
--disable-podpages \
--disable-postproc \
--disable-programs \
--disable-schannel \
--disable-sdl2 \
--disable-securetransport \
--disable-sndio \
--disable-static \
--disable-txtpages \
--disable-v4l2-m2m \
--disable-vaapi \
--disable-vdpau \
--disable-vulkan \
--disable-xlib \
--disable-xmm-clobber-test \
--enable-appkit \
--enable-audiotoolbox \
--enable-avcodec \
--enable-avfilter \
--enable-avformat \
--enable-avfoundation \
--enable-bzlib \
--enable-coreimage \
--enable-cross-compile \
--enable-fontconfig \
--enable-gpl \
--enable-iconv \
--enable-inline-asm \
--enable-libaom \
--enable-libfreetype \
--enable-libfribidi \
--enable-libgsm \
--enable-libmp3lame \
--enable-libopenjpeg \
--enable-libopus \
--enable-librav1e \
--enable-libsoxr \
--enable-libsvtav1 \
--enable-libtheora \
--enable-libtwolame \
--enable-libvorbis \
--enable-libvpx \
--enable-libwebp \
--enable-libx264 \
--enable-libx265 \
--enable-libxvid \
--enable-libzimg \
--enable-lto \
--enable-lzma \
--enable-opencl \
--enable-opengl \
--enable-optimizations \
--enable-pic \
--enable-postproc \
--enable-pthreads \
--enable-small \
--enable-shared \
--enable-swscale \
--enable-version3 \
--enable-videotoolbox \
--enable-zlib \
"$@"
make -j"$(nproc)" install
# Create FFMpeg.framework
# https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPFrameworks/Concepts/FrameworkAnatomy.html
# Create the framework basic directory structure
_framework="FFMpeg.framework"
mkdir -p "/${_framework}/Versions/A/"{Headers,Resources,Libraries}
# Copy licenses to Framework
_framework_docs="/${_framework}/Versions/A/Resources/English.lproj/Documentation"
mkdir -p "$_framework_docs"
# FFMpeg license
cp -avt "$_framework_docs" COPYING* LICENSE*
# Dependency licenses which are not covered by FFMpeg licenses
(cd "${_macports_root}/share/doc" \
&& cp -avt "$_framework_docs" --parents \
zimg/COPYING \
webp/COPYING \
libpng/LICENSE \
libvorbis/COPYING \
freetype/LICENSE.TXT \
fontconfig/COPYING)
# libvorbis, libogg and libtheora share the same license
ln -s libvorbis "${_framework_docs}/libogg"
ln -s libvorbis "${_framework_docs}/libtheora"
# Create required framework symlinks
ln -s A "/${_framework}/Versions/Current"
ln -s Versions/Current/Headers "/${_framework}/Headers"
ln -s Versions/Current/Resources "/${_framework}/Resources"
ln -s Versions/Current/Libraries "/${_framework}/Libraries"
# Framework Info.plist (based on macOS internal OpenGL.framework Info.plist)
cat <<EOF >"/${_framework}/Versions/Current/Resources/Info.plist"
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleDevelopmentRegion</key>
<string>English</string>
<key>CFBundleExecutable</key>
<string>FFMpeg</string>
<key>CFBundleGetInfoString</key>
<string>FFMpeg ${FFMPEG_VERSION}</string>
<key>CFBundleIdentifier</key>
<string>com.spacedrive.ffmpeg</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundleName</key>
<string>FFMpeg</string>
<key>CFBundlePackageType</key>
<string>FMWK</string>
<key>CFBundleShortVersionString</key>
<string>${FFMPEG_VERSION}</string>
<key>CFBundleSignature</key>
<string>????</string>
<key>CFBundleVersion</key>
<string>${FFMPEG_VERSION}</string>
</dict>
</plist>
EOF
# Process FFMpeg libraries to be compatible with the Framework structure
cd "$TARGET_DIR/lib"
# Move all symlinks of ffmpeg libraries to Framework
while IFS= read -r -d '' _lib; do
# Copy symlinks to the output directory
cp -Ppv "$_lib" "/${_framework}/Libraries/${_lib#./}"
rm "$_lib"
done < <(find . -type l -print0)
# Populate queue with ffmpeg libraries
set -- # Clear command line arguments
while IFS= read -r -d '' _lib; do
set -- "$@" "${_lib#./}"
done < <(find . -name '*.dylib' -print0)
# Copy all symlinks of libheif libraries to Framework
while IFS= read -r -d '' _lib; do
# Copy symlinks to the output directory
cp -Ppv "$_lib" "/${_framework}/Libraries/${_lib#"${_macports_root}/lib/"}"
done < <(find "${_macports_root}/lib" -type l \( -name 'libheif.*' -a -name '*.dylib' \) -print0)
# Copy libheif to cwd and add it to queue
while IFS= read -r -d '' _lib; do
_lib_rel="${_lib#"${_macports_root}/lib/"}"
cp -Lpv "$_lib" "./${_lib_rel}"
set -- "$@" "${_lib_rel}"
done < <(find "${_macports_root}/lib" -type f \( -name 'libheif.*' -a -name '*.dylib' \) -print0)
while [ $# -gt 0 ]; do
# Loop through each of the library's dependencies
for _dep in $("${TRIPLE}-otool" -L "$1" | tail -n+2 | awk '{print $1}'); do
case "$_dep" in
# FFMpeg inter dependency
"${TARGET_DIR}/lib/"*)
_linker_path="@loader_path/${_dep#"${TARGET_DIR}/lib/"}"
;;
# Macports dependency (/opt/local/lib means it was installed by Macports)
/opt/local/lib/*)
_dep_rel="${_dep#/opt/local/lib/}"
# Check if the macports dependency is already included in the macOS SDK
if [ -n "$(comm -12 <(printf "%s" "$_dep_rel") <(printf "%s" "$_skd_libs"))" ]; then
# Relink libs already included in macOS SDK
_linker_path="/usr/lib/${_dep_rel}"
else
_linker_path="@loader_path/${_dep_rel}"
if ! [ -e "${_macports_root}/lib/${_dep_rel}" ]; then
echo "Missing macports dependency: ${_dep_rel}"
exit 1
elif ! { [ -f "$_dep_rel" ] || [ -e "/${_framework}/Libraries/${_dep_rel}" ]; }; then
# Copy dependency to the current directory if this is the first time we see it
cp -Lpv "${_macports_root}/lib/${_dep_rel}" "./${_dep_rel}"
# Add it to the queue to have it's own dependencies processed
set -- "$@" "$_dep_rel"
fi
fi
;;
*) # Ignore system libraries
continue
;;
esac
# Change the dependency linker path to make it compatible with an .app bundle
"${TRIPLE}-install_name_tool" -change "$_dep" "$_linker_path" "$1"
done
# Update the library's own id
"${TRIPLE}-install_name_tool" -id "@executable_path/../Frameworks/${_framework}/Libraries/${1}" "$1"
# Copy the library to framework
cp -Lpv "$1" "/${_framework}/Libraries/${1}"
# Remove library from queue
shift
done
# Copy all libheif headers to framework
cp -av "${_macports_root}/include/libheif" "/${_framework}/Headers/"
# Copy all FFMPEG headers to framework
cp -av "${TARGET_DIR}/include/"* "/${_framework}/Headers/"