Files
LocalAI/backend/go/trellis2cpp/Makefile
Richard Palethorpe 9058a2bb46 feat: Add 3d generation UI/API and trellis2cpp backend (#10979)
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint

Adds the plumbing for image-conditioned 3D asset generation (binary
glTF / GLB output), modeled on the video generation path:

- backend.proto: Generate3D RPC + Generate3DRequest (staged image src,
  glb dst, seed/step/cfg_scale/texture_steps, quality and background
  enums, params map for backend-specific extras)
- pkg/grpc: thread Generate3D through client, server, embed, base and
  the backend interfaces; connection-evicting and distributed-node
  wrappers (in-flight tracking + file staging) included
- core/config: FLAG_3D usecase (guessed only for the trellis2cpp
  backend), '3d' canonical usecase string mapped to the Generate3D
  method, and a '3d' output modality
- REST: POST /v1/3d/generations (+ unversioned alias) returning
  OpenAIResponse with a /generated-3d URL or b64_json; conditioning
  image accepted as URL, base64, or data URI; quality/background
  validated at the edge; .glb served as model/gltf-binary
- auth: '3d' route feature (default ON); /api/instructions entry

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(trellis2cpp): add the trellis2.cpp image-to-3D backend

Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2,
pbr-textures branch) as a Go+purego backend, following the
stablediffusion-ggml pattern:

- backend/go/trellis2cpp: purego bindings to the flat C ABI (v9,
  asserted at startup), eager pipeline load with model-set validation
  (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured
  exactly like the upstream demo), Generate3D via t2_generate +
  t2_bake_glb writing a binary glTF to dst. Weight-free unit tests
  cover resolution/validation/param mapping — CI never downloads the
  multi-GB GGUF set or runs inference.
- CPU SIMD variants build into per-variant directories (the shared
  libggml sonames collide across variants, unlike sd-ggml's flat
  renamed-.so scheme); run.sh picks one via /proc/cpuinfo.
- CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan
  amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta +
  latest/master image entries, bump_deps tracking of the pbr-textures
  branch, changed-backends.js mapping, top-level Makefile targets.
- Importer: auto-detects trellis GGUF repos/URIs (registered before
  llama-cpp so the .gguf match isn't stolen) and expands any trellis
  URI to the full 10-file component set spanning the three LocalAI-io
  HF repos.
- Gallery: trellis2-4b (full PBR + 1024 cascade) and
  trellis2-4b-geometry (512 untextured) with verified sha256s.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(ui): 3D generation page with native GLB viewer and IndexedDB history

Adds a Studio tab + /app/3d page for the new image-to-3D endpoint:

- GlbViewer ports the trellis2cpp demo's dependency-free WebGL2
  renderer (quaternion trackball, metallic-roughness PBR, ACES,
  hidden-line wireframe with a bounded index budget) and pairs it with
  a minimal GLB parser for the two forms t2_bake_glb emits — dense
  vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as
  normalized integers) and the opt-in UV-atlas textured form. Parsing
  happens before any GL so stats and errors render without WebGL2.
- use3DHistory stores past generations (params, input thumbnail, and
  the GLB blob itself) in IndexedDB with keep-newest-20 eviction —
  GLBs are multi-MB binaries localStorage can't hold — and the page
  offers a download button for the active GLB.
- Wiring: CAP_3D capability constant (FLAG_3D — the exact string
  /api/models/capabilities serves), threeDApi, router entries, Studio
  tab, vite dev proxy, en locale keys.
- e2e: render-smoke entry plus a focused spec that feeds a real
  one-triangle vertex-PBR GLB through the parser/viewer and exercises
  IndexedDB persistence, selection, deletion, and API errors.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(3d): address API correctness and UX issues

Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it.

Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(3d): add previewable print remeshing

Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download.

Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* build(trellis2cpp): centralize remesh dependency pins

Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(kokoros): implement Generate3D stub for new proto RPC

The Generate3D RPC added to backend.proto for the trellis2cpp backend
made tonic's generated Backend trait require generate3_d, breaking the
kokoros-grpc build. Return unimplemented like the other unsupported
modalities.

Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
2026-07-29 16:15:04 +02:00

133 lines
4.7 KiB
Makefile

CMAKE_ARGS?=
BUILD_TYPE?=
NATIVE?=false
CURRENT_DIR=$(abspath ./)
GOCMD?=go
GO_TAGS?=
JOBS?=$(shell nproc --ignore=1)
# trellis2.cpp — C++/ggml port of Microsoft TRELLIS.2 (image -> 3D GLB).
# The ggml submodule is pinned by trellis2cpp's .gitmodules and fetched via
# --recursive. The commit pin lives here so bump_deps.yaml can update it.
TRELLIS2CPP_REPO?=https://github.com/localai-org/trellis2cpp
TRELLIS2CPP_VERSION?=73dfbe5dfc2cbefd0950853718086556c6d9b043
# libtrellis2 + ggml as shared libraries; no example/test binaries.
CMAKE_ARGS+=-DCMAKE_BUILD_TYPE=Release
CMAKE_ARGS+=-DBUILD_SHARED_LIBS=ON
CMAKE_ARGS+=-DTRELLIS2_BUILD_EXAMPLES=OFF
CMAKE_ARGS+=-DTRELLIS2_BUILD_TESTS=OFF
# Print remeshing is part of trellis2cpp's ABI, so upstream owns the tested
# CGAL/Boost versions, checksums, fetch logic, and update automation. LocalAI
# only opts into that dependency set and pins the trellis2cpp commit above.
CMAKE_ARGS+=-DTRELLIS2_FETCH_PRINT_REMESH_DEPS=ON
CMAKE_ARGS+=-DTRELLIS2_PRINT_REMESH_DEPS_DIR=$(CURRENT_DIR)/sources/print-remesh-deps
ifeq ($(NATIVE),false)
CMAKE_ARGS+=-DGGML_NATIVE=OFF
endif
ifeq ($(BUILD_TYPE),cublas)
CMAKE_ARGS+=-DGGML_CUDA=ON
else ifeq ($(BUILD_TYPE),vulkan)
CMAKE_ARGS+=-DGGML_VULKAN=ON
else ifeq ($(BUILD_TYPE),hipblas)
ROCM_HOME ?= /opt/rocm
ROCM_PATH ?= /opt/rocm
export CXX=$(ROCM_HOME)/llvm/bin/clang++
export CC=$(ROCM_HOME)/llvm/bin/clang
AMDGPU_TARGETS?=gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1200,gfx1201
CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
else ifeq ($(OS),Darwin)
ifneq ($(BUILD_TYPE),metal)
CMAKE_ARGS+=-DTRELLIS2_METAL=OFF -DGGML_METAL=OFF
else
# trellis2cpp turns on GGML_METAL(+EMBED_LIBRARY) itself when
# TRELLIS2_METAL is enabled on Apple platforms.
CMAKE_ARGS+=-DTRELLIS2_METAL=ON
endif
# Dependent libggml*.dylib resolve next to libtrellis2.dylib even
# without DYLD_LIBRARY_PATH being exported.
CMAKE_ARGS+=-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON -DCMAKE_INSTALL_RPATH=@loader_path
endif
ifeq ($(BUILD_TYPE),sycl_f16)
CMAKE_ARGS+=-DGGML_SYCL=ON \
-DCMAKE_C_COMPILER=icx \
-DCMAKE_CXX_COMPILER=icpx \
-DGGML_SYCL_F16=ON
endif
ifeq ($(BUILD_TYPE),sycl_f32)
CMAKE_ARGS+=-DGGML_SYCL=ON \
-DCMAKE_C_COMPILER=icx \
-DCMAKE_CXX_COMPILER=icpx
endif
sources/trellis2cpp:
git clone --recursive $(TRELLIS2CPP_REPO) sources/trellis2cpp && \
cd sources/trellis2cpp && \
git checkout $(TRELLIS2CPP_VERSION) && \
git submodule update --init --recursive --depth 1 --single-branch
# Detect OS
UNAME_S := $(shell uname -s)
UNAME_M := $(shell uname -m)
# The AVX variants are x86-only. ARM64 images use the portable fallback while
# still enabling the selected GPU backend (Vulkan/CUDA) through CMAKE_ARGS.
ifeq ($(UNAME_S),Linux)
ifneq (,$(filter x86_64 amd64,$(UNAME_M)))
VARIANTS = avx avx2 avx512 fallback
else
VARIANTS = fallback
endif
else
# On non-Linux (e.g., Darwin), build only the fallback variant
VARIANTS = fallback
endif
VARIANT_TARGETS = $(foreach v,$(VARIANTS),variants/$(v)/.built)
VARIANT_FLAGS_avx = -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off
VARIANT_FLAGS_avx2 = -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on
VARIANT_FLAGS_avx512 = -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on
VARIANT_FLAGS_fallback = -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off
# libtrellis2 links libggml/libggml-base/libggml-cpu (+ the GPU backend) by
# soname, and those sonames collide across SIMD variants — so each variant
# lives in its own directory and run.sh selects one via LD_LIBRARY_PATH,
# unlike stablediffusion-ggml's flat renamed-.so scheme.
variants/%/.built: sources/trellis2cpp
rm -rf build-$* variants/$*
mkdir -p build-$* variants/$*
cd build-$* && cmake ../sources/trellis2cpp $(CMAKE_ARGS) $(VARIANT_FLAGS_$*) && \
cmake --build . --config Release -j$(JOBS)
@for f in build-$*/libtrellis2.so build-$*/libtrellis2.dylib; do \
if [ -e $$f ]; then cp -a $$f variants/$*/; fi; done
find build-$*/ggml \( -name 'libggml*.so*' -o -name 'libggml*.dylib' \) -exec cp -a {} variants/$*/ \;
rm -rf build-$*
touch $@
trellis2cpp: main.go trellis2.go $(VARIANT_TARGETS)
CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o trellis2cpp ./
package: trellis2cpp
bash package.sh
build: package
clean: purge
rm -rf variants trellis2cpp package sources
purge:
rm -rf build-*
# Weight-free by construction: pure-Go unit tests over model-path resolution,
# validation, and request-parameter mapping. The multi-GB GGUF weights are
# never downloaded in CI; end-to-end generation is exercised manually.
test:
$(GOCMD) test -v ./...
all: trellis2cpp package