mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint Adds the plumbing for image-conditioned 3D asset generation (binary glTF / GLB output), modeled on the video generation path: - backend.proto: Generate3D RPC + Generate3DRequest (staged image src, glb dst, seed/step/cfg_scale/texture_steps, quality and background enums, params map for backend-specific extras) - pkg/grpc: thread Generate3D through client, server, embed, base and the backend interfaces; connection-evicting and distributed-node wrappers (in-flight tracking + file staging) included - core/config: FLAG_3D usecase (guessed only for the trellis2cpp backend), '3d' canonical usecase string mapped to the Generate3D method, and a '3d' output modality - REST: POST /v1/3d/generations (+ unversioned alias) returning OpenAIResponse with a /generated-3d URL or b64_json; conditioning image accepted as URL, base64, or data URI; quality/background validated at the edge; .glb served as model/gltf-binary - auth: '3d' route feature (default ON); /api/instructions entry Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(trellis2cpp): add the trellis2.cpp image-to-3D backend Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2, pbr-textures branch) as a Go+purego backend, following the stablediffusion-ggml pattern: - backend/go/trellis2cpp: purego bindings to the flat C ABI (v9, asserted at startup), eager pipeline load with model-set validation (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured exactly like the upstream demo), Generate3D via t2_generate + t2_bake_glb writing a binary glTF to dst. Weight-free unit tests cover resolution/validation/param mapping — CI never downloads the multi-GB GGUF set or runs inference. - CPU SIMD variants build into per-variant directories (the shared libggml sonames collide across variants, unlike sd-ggml's flat renamed-.so scheme); run.sh picks one via /proc/cpuinfo. - CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta + latest/master image entries, bump_deps tracking of the pbr-textures branch, changed-backends.js mapping, top-level Makefile targets. - Importer: auto-detects trellis GGUF repos/URIs (registered before llama-cpp so the .gguf match isn't stolen) and expands any trellis URI to the full 10-file component set spanning the three LocalAI-io HF repos. - Gallery: trellis2-4b (full PBR + 1024 cascade) and trellis2-4b-geometry (512 untextured) with verified sha256s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(ui): 3D generation page with native GLB viewer and IndexedDB history Adds a Studio tab + /app/3d page for the new image-to-3D endpoint: - GlbViewer ports the trellis2cpp demo's dependency-free WebGL2 renderer (quaternion trackball, metallic-roughness PBR, ACES, hidden-line wireframe with a bounded index budget) and pairs it with a minimal GLB parser for the two forms t2_bake_glb emits — dense vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as normalized integers) and the opt-in UV-atlas textured form. Parsing happens before any GL so stats and errors render without WebGL2. - use3DHistory stores past generations (params, input thumbnail, and the GLB blob itself) in IndexedDB with keep-newest-20 eviction — GLBs are multi-MB binaries localStorage can't hold — and the page offers a download button for the active GLB. - Wiring: CAP_3D capability constant (FLAG_3D — the exact string /api/models/capabilities serves), threeDApi, router entries, Studio tab, vite dev proxy, en locale keys. - e2e: render-smoke entry plus a focused spec that feeds a real one-triangle vertex-PBR GLB through the parser/viewer and exercises IndexedDB persistence, selection, deletion, and API errors. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(3d): address API correctness and UX issues Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it. Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add previewable print remeshing Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download. Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(trellis2cpp): centralize remesh dependency pins Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement Generate3D stub for new proto RPC The Generate3D RPC added to backend.proto for the trellis2cpp backend made tonic's generated Backend trait require generate3_d, breaking the kokoros-grpc build. Return unimplemented like the other unsupported modalities. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
133 lines
4.7 KiB
Makefile
133 lines
4.7 KiB
Makefile
CMAKE_ARGS?=
|
|
BUILD_TYPE?=
|
|
NATIVE?=false
|
|
|
|
CURRENT_DIR=$(abspath ./)
|
|
GOCMD?=go
|
|
GO_TAGS?=
|
|
JOBS?=$(shell nproc --ignore=1)
|
|
|
|
# trellis2.cpp — C++/ggml port of Microsoft TRELLIS.2 (image -> 3D GLB).
|
|
# The ggml submodule is pinned by trellis2cpp's .gitmodules and fetched via
|
|
# --recursive. The commit pin lives here so bump_deps.yaml can update it.
|
|
TRELLIS2CPP_REPO?=https://github.com/localai-org/trellis2cpp
|
|
TRELLIS2CPP_VERSION?=73dfbe5dfc2cbefd0950853718086556c6d9b043
|
|
|
|
# libtrellis2 + ggml as shared libraries; no example/test binaries.
|
|
CMAKE_ARGS+=-DCMAKE_BUILD_TYPE=Release
|
|
CMAKE_ARGS+=-DBUILD_SHARED_LIBS=ON
|
|
CMAKE_ARGS+=-DTRELLIS2_BUILD_EXAMPLES=OFF
|
|
CMAKE_ARGS+=-DTRELLIS2_BUILD_TESTS=OFF
|
|
# Print remeshing is part of trellis2cpp's ABI, so upstream owns the tested
|
|
# CGAL/Boost versions, checksums, fetch logic, and update automation. LocalAI
|
|
# only opts into that dependency set and pins the trellis2cpp commit above.
|
|
CMAKE_ARGS+=-DTRELLIS2_FETCH_PRINT_REMESH_DEPS=ON
|
|
CMAKE_ARGS+=-DTRELLIS2_PRINT_REMESH_DEPS_DIR=$(CURRENT_DIR)/sources/print-remesh-deps
|
|
|
|
ifeq ($(NATIVE),false)
|
|
CMAKE_ARGS+=-DGGML_NATIVE=OFF
|
|
endif
|
|
|
|
ifeq ($(BUILD_TYPE),cublas)
|
|
CMAKE_ARGS+=-DGGML_CUDA=ON
|
|
else ifeq ($(BUILD_TYPE),vulkan)
|
|
CMAKE_ARGS+=-DGGML_VULKAN=ON
|
|
else ifeq ($(BUILD_TYPE),hipblas)
|
|
ROCM_HOME ?= /opt/rocm
|
|
ROCM_PATH ?= /opt/rocm
|
|
export CXX=$(ROCM_HOME)/llvm/bin/clang++
|
|
export CC=$(ROCM_HOME)/llvm/bin/clang
|
|
AMDGPU_TARGETS?=gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1200,gfx1201
|
|
CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
|
|
else ifeq ($(OS),Darwin)
|
|
ifneq ($(BUILD_TYPE),metal)
|
|
CMAKE_ARGS+=-DTRELLIS2_METAL=OFF -DGGML_METAL=OFF
|
|
else
|
|
# trellis2cpp turns on GGML_METAL(+EMBED_LIBRARY) itself when
|
|
# TRELLIS2_METAL is enabled on Apple platforms.
|
|
CMAKE_ARGS+=-DTRELLIS2_METAL=ON
|
|
endif
|
|
# Dependent libggml*.dylib resolve next to libtrellis2.dylib even
|
|
# without DYLD_LIBRARY_PATH being exported.
|
|
CMAKE_ARGS+=-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON -DCMAKE_INSTALL_RPATH=@loader_path
|
|
endif
|
|
|
|
ifeq ($(BUILD_TYPE),sycl_f16)
|
|
CMAKE_ARGS+=-DGGML_SYCL=ON \
|
|
-DCMAKE_C_COMPILER=icx \
|
|
-DCMAKE_CXX_COMPILER=icpx \
|
|
-DGGML_SYCL_F16=ON
|
|
endif
|
|
|
|
ifeq ($(BUILD_TYPE),sycl_f32)
|
|
CMAKE_ARGS+=-DGGML_SYCL=ON \
|
|
-DCMAKE_C_COMPILER=icx \
|
|
-DCMAKE_CXX_COMPILER=icpx
|
|
endif
|
|
|
|
sources/trellis2cpp:
|
|
git clone --recursive $(TRELLIS2CPP_REPO) sources/trellis2cpp && \
|
|
cd sources/trellis2cpp && \
|
|
git checkout $(TRELLIS2CPP_VERSION) && \
|
|
git submodule update --init --recursive --depth 1 --single-branch
|
|
|
|
# Detect OS
|
|
UNAME_S := $(shell uname -s)
|
|
UNAME_M := $(shell uname -m)
|
|
|
|
# The AVX variants are x86-only. ARM64 images use the portable fallback while
|
|
# still enabling the selected GPU backend (Vulkan/CUDA) through CMAKE_ARGS.
|
|
ifeq ($(UNAME_S),Linux)
|
|
ifneq (,$(filter x86_64 amd64,$(UNAME_M)))
|
|
VARIANTS = avx avx2 avx512 fallback
|
|
else
|
|
VARIANTS = fallback
|
|
endif
|
|
else
|
|
# On non-Linux (e.g., Darwin), build only the fallback variant
|
|
VARIANTS = fallback
|
|
endif
|
|
VARIANT_TARGETS = $(foreach v,$(VARIANTS),variants/$(v)/.built)
|
|
|
|
VARIANT_FLAGS_avx = -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off
|
|
VARIANT_FLAGS_avx2 = -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on
|
|
VARIANT_FLAGS_avx512 = -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on
|
|
VARIANT_FLAGS_fallback = -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off
|
|
|
|
# libtrellis2 links libggml/libggml-base/libggml-cpu (+ the GPU backend) by
|
|
# soname, and those sonames collide across SIMD variants — so each variant
|
|
# lives in its own directory and run.sh selects one via LD_LIBRARY_PATH,
|
|
# unlike stablediffusion-ggml's flat renamed-.so scheme.
|
|
variants/%/.built: sources/trellis2cpp
|
|
rm -rf build-$* variants/$*
|
|
mkdir -p build-$* variants/$*
|
|
cd build-$* && cmake ../sources/trellis2cpp $(CMAKE_ARGS) $(VARIANT_FLAGS_$*) && \
|
|
cmake --build . --config Release -j$(JOBS)
|
|
@for f in build-$*/libtrellis2.so build-$*/libtrellis2.dylib; do \
|
|
if [ -e $$f ]; then cp -a $$f variants/$*/; fi; done
|
|
find build-$*/ggml \( -name 'libggml*.so*' -o -name 'libggml*.dylib' \) -exec cp -a {} variants/$*/ \;
|
|
rm -rf build-$*
|
|
touch $@
|
|
|
|
trellis2cpp: main.go trellis2.go $(VARIANT_TARGETS)
|
|
CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o trellis2cpp ./
|
|
|
|
package: trellis2cpp
|
|
bash package.sh
|
|
|
|
build: package
|
|
|
|
clean: purge
|
|
rm -rf variants trellis2cpp package sources
|
|
|
|
purge:
|
|
rm -rf build-*
|
|
|
|
# Weight-free by construction: pure-Go unit tests over model-path resolution,
|
|
# validation, and request-parameter mapping. The multi-GB GGUF weights are
|
|
# never downloaded in CI; end-to-end generation is exercised manually.
|
|
test:
|
|
$(GOCMD) test -v ./...
|
|
|
|
all: trellis2cpp package
|