chore(moss-transcribe-cpp): bump pin to CUDA K-quant embed fix
Bumps the moss-transcribe.cpp pin to 190a569c, which merges the
host-side embed-lookup fallback for K-quant token_embd tensors
(localai-org/moss-transcribe.cpp#2).
Before this, running a q5_K/q4_K/q6_K moss-transcribe GGUF on CUDA
(or any non-CPU backend) aborted in getrows.cu with
"unsupported src0 type: q5_K" because ggml's GET_ROWS op has no
K-quant implementation on GPU, killing the backend process on the
first request. The engine now dequantizes the needed rows on the
host when the backend cannot run GET_ROWS for the tensor type,
producing bit-identical results.
Fixes#10862
Assisted-by: Claude:claude-opus-4-8 [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add moss-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the moss-tts.cpp ggml port of the OpenMOSS
MOSS-TTS-Local v1.5 text-to-speech model (GPT-J local transformer decoded through
MOSS-Audio-Tokenizer-v2), producing 48 kHz stereo audio with optional
reference-audio voice cloning. Mirrors the qwen3-tts-cpp backend: dlopen the
static-ggml shared library, bind the moss-tts.cpp C-API via purego, and serve
the gRPC TTS method. A thin C shim holds the pipeline handle and copies engine
PCM into a Go-freeable buffer.
Wires the CI registration: backend-matrix.yml (CPU, CUDA 12/13, Intel SYCL
f16/f32, Vulkan, ROCm, NVIDIA L4T, plus Darwin metal), backend/index.yaml metas
and image entries pointing at mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF, the
root Makefile build targets, and the changed-backends.js path mapping.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: list the moss-tts-cpp backend among the LocalAI-maintained engines
Add moss-tts.cpp to the README "Backends built by us" table, the
Text-to-Speech compatibility table, and the reference-audio voice-cloning
backend list, so the new backend is documented alongside its peers.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): pin moss-tts.cpp to the squashed single-commit release
moss-tts.cpp history was collapsed to a single commit; repoint MOSSTTS_CPP_VERSION
to ee722b8e9205ee9b1b1c398a4e87e4e393e9be41.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): add the moss-tts-cpp-development gallery meta
The gallery had the -development image entries but no matching -development
meta anchor (as locate-anything-cpp and depth-anything-cpp have), so the master
build was not installable as a gallery backend. Add moss-tts-cpp-development
mirroring the production meta with the -development capability image names.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): rewrite c2pa-audio submodule path for subproject builds
CrispASR a38cb89 adds a crispasr_c2pa_native static library whose sources
live in the new third_party/c2pa-audio git submodule, located via
CMAKE_SOURCE_DIR in src/CMakeLists.txt. That variable assumes CrispASR is
the top-level CMake project; LocalAI embeds it via add_subdirectory, so
the path resolved to backend/go/crispasr/third_party/c2pa-audio and every
build variant failed at CMake generate with 'Cannot find source file:
c2pa_native.cpp'.
Extend the existing talk-llama sed workaround to also rewrite the
c2pa-audio reference to PROJECT_SOURCE_DIR, which is correct both
standalone and as a subproject. The submodule itself is already checked
out by the recursive submodule init. Verified locally: the exact CI error
reproduces with CMAKE_SOURCE_DIR, and with the rewrite CMake configure,
crispasr_c2pa_native, and crispasr-lib all build cleanly on a CPU-only
fallback configuration.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The turboquant and bonsai backends copy backend/cpp/llama-cpp/ wholesale
into their build directories and reuse its Makefile/prepare.sh against
their own llama.cpp forks. When PR #10837 added
backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch, the
copied patches/ directory was mis-applied to the fork checkouts: the
fork trees diverge from upstream, hunks rejected, and because the
patch-apply loop in prepare.sh ran before set -e took effect the build
kept going and died much later with a confusing compile error
("'LLM_ARCH_MINIMAX_M3' was not declared in this scope"). This broke
tests-turboquant-grpc on that PR.
Two hardening changes:
- turboquant/bonsai Makefiles: delete the copied patches/ directory
right after the cp -rf of backend/cpp/llama-cpp/. Patches vendored
for upstream llama.cpp must never be applied to the forks; each fork
carries its own patch series under backend/cpp/<backend>/patches/,
applied by its apply-patches.sh.
- llama-cpp prepare.sh: run the patch-apply loop under set -e so a
rejecting patch fails fast and loudly at apply time instead of
surfacing as a downstream compile error. A missing or empty patches/
directory remains a no-op success, so all existing callers (the
llama-cpp Makefile targets and the turboquant/bonsai copies) are
unaffected when no patches ship.
Exposed by PR #10837.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The sglang Python backend inherits the default Status RPC from
backend_pb2_grpc.BackendServicer, which raises NotImplementedError.
LocalAI's backend-monitor polls /backend.Backend/Status periodically on
every registered backend; when the call fails, /backend/monitor returns
HTTP 500 and downstream inference requests to the sglang backend are
blocked even though the model is loaded and answering directly via the
gRPC endpoint.
Add a minimal Status shim that mirrors the existing Health method and
returns StatusResponse{state=READY} unconditionally. This unblocks the
monitor path; a state-aware follow-up (UNINITIALIZED during load, BUSY
under active inference) is left for a subsequent change.
Reproduced on DGX Spark (GB10, arm64-l4t-cuda-13 image) with the sglang
v0.5.15 backend and Qwen3-Coder-Next-NVFP4-GB10; verified locally that
patching the shim in place immediately restores /backend/monitor and
inference across the sglang slot.
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models
Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp
(github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0
(1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the
Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants.
Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp
against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models
are served with the same OpenAI-compatible API. No grpc-server allow-list patch
is needed (bonsai adds weight quants, transparent to the server, not KV-cache
types), and the reused server compiles cleanly against the fork with no skew
patches (validated locally via a CPU docker build; patches/ is present but empty
for any future re-pin skew).
Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh,
backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build
rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml
meta-backend + per-platform images, and a nightly bump_deps entry tracking the
`prism` branch.
Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64,
+pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B
models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not
wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add stable and development gallery variants for Linux and Darwin, and wire the backend build matrix so the referenced images are published.
Assisted-by: Codex:gpt-5 [yq]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
PyTorch 2.13 XPU pulls oneAPI 2026 libraries that conflict with the oneAPI 2025.3 backend image. Pin torch and torchaudio to the matching 2.11 XPU pair so the build resolves a coherent 2025.3 runtime.
Assisted-by: Codex:GPT-5 [uv]
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
SetDefaults injected the llama.cpp server options cache_reuse
(ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied
per selected node by the distributed router) onto every model config
regardless of backend. Every other backend ignores options it does not
understand, so this was harmless until longcat-video, which strictly
validates its options and fails LoadModel with
"unknown model option(s): cache_reuse, parallel".
Gate both injections behind a new UsesLlamaCppServingOptions allow-list
(llama-cpp plus the empty/auto-detect case that resolves to llama.cpp
from a GGUF file, mirroring how llamaCppDefaults is registered). This
follows the existing UsesLlamaSamplerDefaults precedent for llama-only
defaults. The typed NBatch field is deliberately left alone: it is a
proto field every backend simply ignores, which is why batch never
triggered the error.
Also harden the longcat-video backend to warn-and-ignore unknown model
options and request params through a testable select_known_options
helper, matching the other LocalAI Python backends, so a future
server-injected option cannot break loading again.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): add voice library workflow
Give administrators a production-ready flow to record or upload consented reference audio, manage reusable profiles, inspect API usage, discover compatible models, and hand a saved voice directly to text-to-speech.
Assisted-by: Codex:gpt-5
* feat(voice): add managed voice cloning profiles
Make reusable reference voices manageable through the admin API instead of requiring model-directory and YAML edits. Discover compatible installed and gallery models from server-side backend capabilities, retain explicit model configuration controls, and stage saved references for supported backends.
Expose profile management through REST and MCP, document backend-specific behavior, and cover the workflow from profile creation through real Qwen3-TTS synthesis. Harden the agent-job HTTP test against completion racing cancellation.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
dependabot[bot]