mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-30 01:54:31 -04:00
e183ea11d1ba700dda42ca7a762508fa231d00a8
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Richard Palethorpe
|
2facfc0d88 |
feat: Add kimodo.cpp and 3D animation API/UI (#12095)
* fix(vulkan): preserve host ICD discovery for packaged backends Add bundled Mesa manifests through VK_ADD_DRIVER_FILES instead of replacing the system driver list. Merge inherited and model-specific additive paths while preserving explicit operator overrides, with regression coverage. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add Kimodo CPU and Vulkan animation backend Introduce a distinct animation capability and model-described 3D operations, with a typed /3d/animate API, RPC transport, distributed media staging, permissions, and tracing. Add a persistent kimodo.cpp adapter, skeleton GLB export, CPU/Vulkan packages, model and backend galleries, importer support, CI builds, and documentation. Adapt Studio inputs to each model and provide real-time skeleton playback, seeking, and history. Cover backend validation, packaging, API behavior, importer inventories, distributed staging, and Studio workflows. Validate real-model CPU/Vulkan generation and deploy the integration to the local QA instance. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(kimodocpp): adopt monolithic encoders and resident inference Update upstream for resident weights, packed execution paths, and cached motion graphs. Default to all 32 text layers while retaining configurable streaming and legacy bundle support. Use monolithic Q8_0 encoders by default and offer all six published quantizations through the gallery and importer. Refresh pinned hashes, tests, and documentation; remove the obsolete thread patch and ensure cached source checkouts follow the upstream pin. Validated CPU and Vulkan generation, lower-bit streaming, gallery/importer suites, packaging, lint, and cold/warm Studio generation on localai-dev. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Richard Palethorpeandlocalai-org-maint-bot
|
9058a2bb46 |
feat: Add 3d generation UI/API and trellis2cpp backend (#10979)
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint Adds the plumbing for image-conditioned 3D asset generation (binary glTF / GLB output), modeled on the video generation path: - backend.proto: Generate3D RPC + Generate3DRequest (staged image src, glb dst, seed/step/cfg_scale/texture_steps, quality and background enums, params map for backend-specific extras) - pkg/grpc: thread Generate3D through client, server, embed, base and the backend interfaces; connection-evicting and distributed-node wrappers (in-flight tracking + file staging) included - core/config: FLAG_3D usecase (guessed only for the trellis2cpp backend), '3d' canonical usecase string mapped to the Generate3D method, and a '3d' output modality - REST: POST /v1/3d/generations (+ unversioned alias) returning OpenAIResponse with a /generated-3d URL or b64_json; conditioning image accepted as URL, base64, or data URI; quality/background validated at the edge; .glb served as model/gltf-binary - auth: '3d' route feature (default ON); /api/instructions entry Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(trellis2cpp): add the trellis2.cpp image-to-3D backend Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2, pbr-textures branch) as a Go+purego backend, following the stablediffusion-ggml pattern: - backend/go/trellis2cpp: purego bindings to the flat C ABI (v9, asserted at startup), eager pipeline load with model-set validation (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured exactly like the upstream demo), Generate3D via t2_generate + t2_bake_glb writing a binary glTF to dst. Weight-free unit tests cover resolution/validation/param mapping — CI never downloads the multi-GB GGUF set or runs inference. - CPU SIMD variants build into per-variant directories (the shared libggml sonames collide across variants, unlike sd-ggml's flat renamed-.so scheme); run.sh picks one via /proc/cpuinfo. - CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta + latest/master image entries, bump_deps tracking of the pbr-textures branch, changed-backends.js mapping, top-level Makefile targets. - Importer: auto-detects trellis GGUF repos/URIs (registered before llama-cpp so the .gguf match isn't stolen) and expands any trellis URI to the full 10-file component set spanning the three LocalAI-io HF repos. - Gallery: trellis2-4b (full PBR + 1024 cascade) and trellis2-4b-geometry (512 untextured) with verified sha256s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(ui): 3D generation page with native GLB viewer and IndexedDB history Adds a Studio tab + /app/3d page for the new image-to-3D endpoint: - GlbViewer ports the trellis2cpp demo's dependency-free WebGL2 renderer (quaternion trackball, metallic-roughness PBR, ACES, hidden-line wireframe with a bounded index budget) and pairs it with a minimal GLB parser for the two forms t2_bake_glb emits — dense vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as normalized integers) and the opt-in UV-atlas textured form. Parsing happens before any GL so stats and errors render without WebGL2. - use3DHistory stores past generations (params, input thumbnail, and the GLB blob itself) in IndexedDB with keep-newest-20 eviction — GLBs are multi-MB binaries localStorage can't hold — and the page offers a download button for the active GLB. - Wiring: CAP_3D capability constant (FLAG_3D — the exact string /api/models/capabilities serves), threeDApi, router entries, Studio tab, vite dev proxy, en locale keys. - e2e: render-smoke entry plus a focused spec that feeds a real one-triangle vertex-PBR GLB through the parser/viewer and exercises IndexedDB persistence, selection, deletion, and API errors. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(3d): address API correctness and UX issues Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it. Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add previewable print remeshing Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download. Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(trellis2cpp): centralize remesh dependency pins Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement Generate3D stub for new proto RPC The Generate3D RPC added to backend.proto for the trellis2cpp backend made tonic's generated Backend trait require generate3_d, breaking the kokoros-grpc build. Return unimplemented like the other unsupported modalities. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
294170d3ed |
feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery (#10352)
* feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery Mirrors the locate-anything-cpp backend to register a new depth-anything backend that wraps the Depth Anything 3 ggml port (depth-anything.cpp) via purego (cgo-less, no Python at inference). - backend/go/depth-anything-cpp/: gRPC backend (Load + Predict + GenerateImage), purego binding to the da_capi_* C ABI, CMake/Makefile/run/package/test scripts building depth-anything.cpp's DA_SHARED static .so per CPU variant. - backend/index.yaml: depth-anything backend meta + all hardware-variant capability entries (cpu/cuda12/cuda13/intel-sycl-f32+f16/vulkan/nvidia-l4t). - gallery/index.yaml: 8 Depth Anything 3 GGUF models (base q4_k/q8_0/f16/f32, small, large, giant, mono-large). - .github/backend-matrix.yml: one build entry per hardware variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): typed Depth RPC + REST endpoint exposing full DA3 data Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to e0b6814 (ABI 3 dense C-API) The Depth RPC handler calls da_capi_depth_dense / da_capi_points (C-API ABI 3); pin the native build to the commit that exports them. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to v0.1.0 release (b515c31) Repoint the native version from the now-orphaned e0b6814 to the b515c31 release commit, kept alive by the upstream v0.1.0 tag. C-API is unchanged (da_capi_abi_version == 3). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): wire depth-anything-cpp into build, CI bump, and importer The backend dir, gallery index, and CI build-matrix were present but the backend was never wired into the integration points that adding-backends.md requires: - root Makefile: add to .NOTPARALLEL, the test-extra chain, a BACKEND_* definition, the docker-build target eval, and docker-build-backends (mirrors parakeet-cpp; the backend's own Makefile already documented that its `test` target is driven by test-extra). - bump_deps.yaml: register the DEPTHANYTHING_VERSION pin so the daily auto-bump bot tracks mudler/depth-anything.cpp master (it cannot see an unregistered Makefile pin). - import form: add a preference-only KnownBackend entry so depth-anything is selectable at /import-model (mirrors sam3-cpp; no reliable GGUF auto-detect signal, so pref-only per the doc's default). changed-backends.js needs no entry: the generic golang suffix branch already resolves backend/go/depth-anything-cpp/. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): auto-detect importer for depth-anything GGUFs Replace the preference-only entry with a real auto-detect importer (mirrors parakeet-cpp / locate-anything): - DepthAnythingImporter matches a .gguf whose name carries a depth-anything token (depth-anything-<size>-<quant>.gguf), so /import-model recognises mudler/depth-anything.cpp-gguf repos and direct GGUF URLs without an explicit backend preference. preferences.backend= "depth-anything" still forces it. - Registered before LlamaCPPImporter so its GGUF bundles aren't claimed by the generic .gguf importer; the narrow name match means it cannot claim arbitrary llama GGUFs or the upstream safetensors PyTorch repos. - Multi-quant repos pick the smallest quant by default (q4_k -> ... -> f32, depth stays >0.998 corr even at q4_k); quantizations preference overrides. - Drops the now-redundant knownPrefOnlyBackends entry (importer-backed backends are not listed there, matching parakeet-cpp). - Table-driven Ginkgo test covers detection, negative cases (llama GGUF, upstream safetensors), default/override/fallback quant pick, and direct URL import. 10/10 specs pass. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): check conn.Close error in grpc Depth client (errcheck) The new Depth() client method used a bare `defer conn.Close()`. golangci-lint runs with new-from-merge-base, so although the 39 sibling methods use the same bare form (grandfathered), the newly added line trips errcheck. Drop the result explicitly to satisfy the linter. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.1 (embeddable CMake) v0.1.0 (b515c31) used ${CMAKE_SOURCE_DIR} for its include dirs, which points at the parent project when built via add_subdirectory() as this backend does, so the container build failed with missing stb_image.h / da_gguf_keys.h. v0.1.1 (2d42897) switches to project-relative paths. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): resolve gosec findings in the backend wrapper The code-scanning gate flagged three new failure-level alerts in godepthanythingcpp.go (gosec runs with -no-fail; GitHub gates on new alerts): - G301: export dirs were created with 0o755. Tighten to 0o750 (no world access needed for backend-written export output). - G304: writeDepthPNG creates req.GetDst(). That path is chosen by the LocalAI core as the intended output destination (same pattern every image backend uses), not attacker input, so annotate with #nosec G304 and document why. The remaining G103 "audit unsafe" notes on the unsafe.Slice C-buffer copies are warning-level (the same purego interop whisper/parakeet use) and do not gate the check, per the supertonic exclusion precedent in secscan.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.2 (CUDA cross-build arch) v0.1.1 forced CMAKE_CUDA_ARCHITECTURES=native, which breaks the GPU-less l4t/cublas CI builds (nvcc "Unsupported gpu architecture 'compute_'" on CMake 3.22). v0.1.2 (442eea4) drops the override and lets ggml pick its default cross-build arch list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
6a80e23733 |
feat(middleware): Model routing, PII filtering, Cloud model proxies (#9802)
Add a routing middleware stack and a cloud-proxy backend. * cloud-proxy: a Go gRPC backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers, with an optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response) and full tool-calling support. * routing: admission control, content-aware model routing (embedding cache + classifier + rerank + Arch-Router score), PII detection/redaction (regex + NER) with streaming filter and OpenAI/Anthropic adapters, and a per-user/per-key billing recorder backed by GORM or in-memory storage. * middleware: UsageMiddleware records usage via the billing recorder, plus admission, route-model, usage-stamp and trace middlewares. * observability: BackendTrace ring buffer stores full request bodies (capped), MITM proxy emits structured trace events, and router classifier decisions surface at /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0). * UI: cloud-proxy model-editor fields, classifier system-prompt and score-normalization config, and a Traces page rendering request bodies. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Ettore Di Giacinto
|
87e6de1989 |
feat: wire transcription for llama.cpp, add streaming support (#9353)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
510d6759fe |
fix(nodes): better detection if nodes goes down or model is not available (#9274)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |