The "Bump Backend dependencies" workflow has failed every night for over
ten days. Four upstreams — ced.cpp, moss-transcribe.cpp, voice-detect.cpp
and rf-detr.cpp — moved from the mudler org to localai-org, so the GitHub
API answers 301 for the old slugs. ced.cpp additionally renamed its
default branch to main.
bump_deps.sh fetched without -L or -f and never checked the response, so
the redirect's JSON body was passed straight to sed, which died with
"unterminated `s' command". The loud failure was luck: an error body
without slashes would have been substituted into the Makefile as the new
pin, silently corrupting the version and shipping it in a bump PR.
Point the matrix at the new slugs and branch, and harden the script so a
bad response can never reach sed: follow redirects, fail on HTTP errors,
and require a bare 40-hex SHA before rewriting anything. Also refresh the
now-stale repository URLs in the backend Makefiles, test scripts,
backend/index.yaml and the docs.
Verified all 25 matrix entries resolve to a commit SHA and that the four
previously-failing jobs run end to end against the real API.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
#10970 gave the four PredictOptions RPCs a model-identity check so a
backend reached through a stale distributed route rejects the request
instead of answering from whatever model it holds (#10952). Every other
modality shares that exposure: the route is cached by host:port, a worker
can recycle a stopped backend's port for another model's backend, and a
liveness-only probe cannot tell a stale row from a valid one.
Extends the same mechanism to the 21 remaining request messages that reach
a backend through the router, using the pattern #10970 established rather
than a parallel one:
- proto: ModelIdentity on each modality request message.
- controller: populated from ModelConfig.Model at the call site that also
builds ModelOptions, so load-time and request-time values are equal by
construction.
- backends: one generic guard in pkg/grpc/server.go (27 Go backends), the
method set in backend/python/common (36 Python backends), llama-cpp
(AudioTranscription/Stream, Rerank, Score) and privacy-filter
(TokenClassify).
- reconcile already drops the stale row on IsModelMismatch; no change.
TTSRequest and SoundGenerationRequest get a SEPARATE ModelIdentity field
rather than reusing their existing `model`: FileStagingClient rewrites
`model` to a worker-local path, so comparing it would reject valid
requests in exactly the configuration this guards.
AudioEncode/AudioDecode are deliberately left unguarded: the opus codec
backend is loaded from a literal rather than a ModelConfig, so no value
carries the equality guarantee the comparison depends on. The four
bidirectional stream RPCs are out of scope; they bypass reconcile.
Empty means skip on both sides, so an old controller, an old backend, and
the bare request structs in tests/e2e-backends all keep working.
Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(distributed): reject wrong-model requests at the backend (#10952)
In distributed mode the controller caches a NodeModel row naming a backend's
host:port. A worker can recycle a stopped backend's gRPC port for a different
model's backend, and probeHealth verifies liveness rather than identity, so the
probe succeeds against whatever now occupies the port and the request is
dispatched to the wrong backend. The caller gets a silent wrong-model answer.
Nothing in the request could catch this: PredictOptions had no model field, so
model identity crossed the wire only in ModelOptions.Model at LoadModel time,
and the cached-hit path issues no LoadModel. Every backend's "model not loaded"
guard checks a nil handle, which a process holding a different model passes, so
the stale row was never dropped either.
Add PredictOptions.ModelIdentity and enforce it at the point of use:
- The controller populates it in gRPCPredictOpts from ModelConfig.Model, the
same expression ModelOptions feeds to model.WithModel and therefore the
same value the backend received as ModelOptions.Model. Both are read from
one config value in one function, so they are equal by construction and the
comparison cannot false-reject.
- Backends compare it against what they loaded and return NOT_FOUND with a
fixed sentinel. Enforced in pkg/grpc/server.go (27 Go backends), an
interceptor in backend/python/common (all 36 Python backends, no
per-backend change), and the llama-cpp / ik-llama-cpp / ds4 C++ servers.
That is every backend with real exposure: kokoros answers all four RPCs
with unimplemented and privacy-filter implements none of them.
- The router's reconcile drops the stale replica row on a mismatch, so the
next request reloads somewhere correct.
Empty means "skip the check" on both sides: a controller that predates the
field sends nothing, a backend loaded by such a controller has nothing to
compare, and the C++ server synthesizes PredictOptions internally for ASR. That
keeps upgrades working in both directions.
Scoped to the four PredictOptions RPCs. TTSRequest.model and
SoundGenerationRequest.model are deliberately NOT validated: FileStagingClient
already rewrites them to worker-local absolute paths, so in distributed mode
they already differ from the load-time value and comparing them would reject
valid requests.
IsModelMismatch requires both the NOT_FOUND code and the sentinel, unlike the
neighbouring helpers which accept either. insightface's Embedding returns
NOT_FOUND "no face detected" on a PredictOptions RPC, and a code-only check
would drop a healthy replica row on every faceless image.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* chore(deps): bump the pip group across 6 directories with 1 update
Bumps the pip group with 1 update in the /backend/python/ace-step directory: torch.
Bumps the pip group with 1 update in the /backend/python/llama-cpp-quantization directory: torch.
Bumps the pip group with 1 update in the /backend/python/longcat-video directory: torch.
Bumps the pip group with 1 update in the /backend/python/sglang directory: torch.
Bumps the pip group with 1 update in the /backend/python/trl directory: torch.
Bumps the pip group with 1 update in the /backend/python/vllm-omni directory: torch.
Updates `torch` from 2.10.0+rocm7.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.12.1 to 2.12.1+cu130
Updates `torch` from 2.9.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.7.0 to 2.12.1+cu130
---
updated-dependencies:
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(deps): preserve platform-specific torch requirements
Keep the 2.12.1 CPU bump only where uv resolves it cleanly, and restore ROCm, CUDA, MPS, and unrelated transformers constraints that Dependabot rewrote to incompatible wheel variants.
Assisted-by: Codex:gpt-5 [uv]
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Every vibevoice requirements file pulled diffusers straight from
git+https://github.com/huggingface/diffusers. That branch now reports
itself as 0.40.0.dev0 and requires huggingface-hub>=1.23.0,<2.0, while
transformers>=4.51.3,<5.0.0 (which upstream VibeVoice mandates) still
caps huggingface-hub at <1.0. Because a git URL offers the resolver
exactly one candidate version, uv has nothing to backtrack to and the
install fails outright:
Because only diffusers==0.40.0.dev0 is available and diffusers==0.40.0.dev0
depends on huggingface-hub>=1.23.0,<2.0 [...] we can conclude that your
requirements are unsatisfiable.
This broke the vibevoice build on every variant - cpu (amd64/arm64),
cuda 12/13, l4t 12/13, intel and rocm, plus the darwin metal job - and
has been failing the weekly full-matrix rebuild for three weeks. It is
not caught by master pushes because backend builds are path-filtered
there, so it only surfaces on the Sunday cron and on release tags.
Use the PyPI package instead. That is what upstream VibeVoice declares
in its own pyproject.toml, and what every other LocalAI backend already
does - vibevoice was the only one tracking the git branch. With a real
release series available the resolver settles on diffusers 0.39.0 with
huggingface-hub 0.36.2 and transformers 4.57.6, and it can keep
backtracking on its own if upstream shifts again.
Verified with uv pip compile against cpu, cublas12, cublas13, hipblas,
intel, mps and l4t13: all resolve to that same coherent set. l4t12 only
resolves on aarch64, since its Jetson index ships no x86_64 torch wheel.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Bash] [uv]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Adds inference parameter defaults for the minimax-m3 model family and
includes a vendored patch of upstream llama.cpp PR #24523 to recognize
the minimax-m3 architecture. Once the upstream PR merges, the patch can
be removed and LLAMA_VERSION bumped normally.
Changes:
- backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch:
vendored patch from ggml-org/llama.cpp#24523 (Preliminary MiniMax-M3
support). Applied by prepare.sh during the build; keeps the pinned
LLAMA_VERSION pointing at the latest upstream tag.
- core/config/inference_defaults.json: add minimax-m3 family entry
(temperature=1.0, top_p=0.95, top_k=40, min_p=0.01,
repeat_penalty=1.0, matching the existing minimax defaults) and
register it in the patterns list before the shorter minimax-m2.7
entry for correct longest-match-first ordering.
Upstream: depends on ggml-org/llama.cpp#24523
Closes: https://github.com/mudler/LocalAI/issues/10820
Signed-off-by: Nandana Dileep <110280757+nandanadileep@users.noreply.github.com>
* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): initialize only declared submodules
The latest upstream commit contains an undeclared CrispASR gitlink that makes a blanket recursive submodule update fail. Limit initialization to the two submodules used by the backend build.
Assisted-by: Codex:gpt-5 [Codex]
* fix(crispasr): resolve vendored WebRTC from project root
CrispASR now builds a vendored WebRTC VAD, but its include paths assume CrispASR is the top-level CMake project. Extend the existing embedded-project rewrite to the shared third_party root.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
request.language usually carries an ISO 639-1 code (e.g. "de"), which
OpenAI-compatible clients such as Home Assistant / wyoming_openai send,
but qwen_asr.validate_language() only accepts full English names
("German") and raises ValueError otherwise. Normalize the requested
language: accept full names case-insensitively, translate ISO codes
(with optional region suffix like "de-DE") to the expected name, and
pass anything unrecognised through so qwen_asr still reports it clearly.
Fixes#10958
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(model-artifacts): materialize longcat-video checkpoints on the controller
longcat-video loads a checkpoint directory: its backend.py takes
request.ModelFile when os.path.isdir(request.ModelFile) and otherwise
falls back to snapshot_download. That places it in the same class as
transformers/vllm/diffusers/sglang, but the allow-list added in #10910
did not enumerate it, so PrimaryArtifactSpec returned no managed
artifact for a bare HuggingFace repo id.
The consequence in distributed mode: nothing was acquired on the
controller, ModelFileName fell through to the raw repo id, and staging
skipped the resulting phantom /models/<owner>/<repo> path. The worker
received a blank ModelFile, fell back to request.Model, and downloaded
~83GB from HuggingFace inside the remote LoadModel deadline - so the
load could only ever fail with DeadlineExceeded while an abandoned
backend process kept downloading.
Note this materializes the full repository. The backend restricts its
own snapshot_download with allow_patterns, and the avatar repo ships
both base_model/ and base_model_int8/ where only one is ever loaded;
inferred specs have no way to carry patterns today. Tracked separately.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): warn when staging skips a non-existent model path
stageModelFiles logs "Staging model files for remote node" up front, then
silently drops any path field that does not exist on the controller. The
skip itself is legitimate and must stay: a backend outside
managedArtifactBackends that takes a bare HuggingFace repo id gets an
optimistically constructed path (ModelFileName falls through to the raw
model reference) that was never materialized, and sources its own weights
on the worker. Erroring would break those configs.
But at debug level the operator is left with a reassuring staging line and
no trace of the skip, so a genuine controller-side acquisition gap is
indistinguishable from a healthy pass-through - it surfaces much later as
a remote LoadModel timeout, on a worker that is quietly downloading tens
of gigabytes. Raise the skip to warn and name the field, path, node and
tracking key. Behavior is unchanged.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): allow a config to declare companion artifacts
A composed pipeline needs more than one HuggingFace snapshot.
LongCat-Video-Avatar-1.5 loads its own transformer but takes the
tokenizer, text encoder and VAE from the separate LongCat-Video base
repo, so a single-artifact config cannot express it and the backend is
left to fetch the second repo itself at load time.
Widen the artifact model to target: model plus any number of named
target: companion entries. Normalize accepts the new target and
constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that
name is the option key the backend later receives; a companion may not
claim primary_file, which only means anything for a load target.
ModelConfig.Validate requires exactly one primary and requires it first,
since Artifacts[0] is what ModelFileName, size estimation and staging all
resolve from.
Both acquisition paths now loop instead of touching index 0 alone:
preloadOne for an already-installed config, bindPrimaryArtifact for a
gallery install. Failure policy differs by provenance. An inferred
primary keeps its warn-and-fall-back, because the legacy download path
still exists for it. Companions are explicit by construction, so they are
all-or-nothing: a config naming one is asserting the backend needs it,
and failing at the acquisition boundary is far more legible than a
missing-weights error surfacing later inside the backend.
The cache key is deliberately unchanged. It hashes source identity only,
never name or target, so every already-installed managed model still hits
its existing snapshot instead of silently re-downloading. Two specs pin
that: one proving a companion and a primary with identical sources agree
on the key, and one pinning the digest of a known primary outright.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): hand resolved companion snapshots to the backend
A materialized companion is useless until the backend can find it, and
its location is a content-addressed cache key that does not exist until
the artifact resolves. A static gallery override cannot carry that, and
persisting it into the config YAML would rot the moment a re-resolve
produced a new key.
Synthesize it instead at load time: each resolved companion becomes
"<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the
key:value convention backends already parse for options like
attention_backend. The value stays relative to the models directory so a
remote worker can resolve it under its own ModelPath once staging has
rewritten the model root. An option the author set explicitly always
wins, so pinning a companion to a local checkout still beats the managed
snapshot.
longcat-video resolves base_model through ModelPath, the same convention
qwen-tts, voxcpm, outetts and ace-step already use for companion assets.
Its sibling-directory heuristic is deleted: it looked for a LongCat-Video
directory next to the model, which cannot exist under the content
addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead
code the moment the model became managed.
The gallery entry declares both repositories and restricts each with
allow_patterns. The avatar repo ships base_model/ and base_model_int8/
and only ever loads one, so fetching the whole repo would roughly double
the download. The patterns match the entry's own options (use_distill
true, use_int8 default false); enabling use_int8 here also requires
adding base_model_int8/**, which is called out in the entry.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage managed artifact trees from the models root
Staging anchored the worker's models directory on the primary snapshot
whenever a model was managed, so a companion snapshot could not reach the
worker at all.
frontendModelsDir was derived by stripping the Model relative path off
the end of ModelFile. For a managed artifact nothing matches: ModelFile
is .artifacts/huggingface/<key>/snapshot while Model stays a bare
HuggingFace repo id, so the strip was a no-op and the "models directory"
came out as the snapshot itself. Two consequences, both silent. Staging
keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two
snapshots of one model were indistinguishable on the worker. And a
companion, which lives in a sibling snapshot directory outside the
primary, fell outside that directory entirely: StagingKeyMapper.Key
collapsed its files to bare basenames and resolveOptionPath could not
resolve the relative option at all, so it was skipped without a word.
Derive the models root from the artifact tree instead when the path runs
through it, and compute the worker's ModelPath from the file's path
relative to that root rather than from the Model field. The legacy layout
is unaffected: where Model really is the relative path, the new
derivation reduces to the old one, which a regression spec pins.
This deliberately changes an invariant that router_dirstage_test.go
pinned: for a managed primary, ModelFile and ModelPath were both the
snapshot directory, and staging keys were relative to it. Now ModelFile
is the snapshot, ModelPath is the models root above it, and keys keep the
full relative path. That spec is updated rather than accommodated, with
the reasoning recorded inline, because the old invariant is exactly what
made a sibling companion unreachable.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(ci): build Bonsai backend images
Register the Bonsai C++ source path with the backend matrix filter so changes select its image jobs. Also make shared llama.cpp changes rebuild the Bonsai and Turboquant fork images in the actual matrix, not only their test flags.\n\nAssisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
cuDNN 9 is a dispatcher (libcudnn.so.9) plus seven sublibraries the dispatcher
dlopen()s by bare soname. Only the dispatcher is ever a DT_NEEDED, so ldd finds
it and never the seven. The allowlist force-copied three of them
(libcudnn.so*, libcudnn_ops.so*, libcudnn_cnn.so*) into every CUDA backend,
which is wrong in both directions at once: too few libraries for a backend that
uses cuDNN, and too many for one that does not.
On an L4T fleet, ten of the eleven backends carrying cuDNN were in a broken end
state; the one that was correct was correct by accident, being BUILD_TYPE=cpu
so package_cuda_libs never ran for it.
longcat-video bundled 4 of 8 at 9.24.0 over a complete pip set at 9.20.0.48
in its venv. libbackend.sh puts lib/ on LD_LIBRARY_PATH, searched before
DT_RUNPATH, so the bundle won and the rest still came from the venv:
CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH.
Nine others bundled 3 of 8 and had no venv cuDNN. None bundled
libcudnn_graph, which libcudnn_cnn has a hard DT_NEEDED on, so it resolved
out of the runtime image and the process ran bundled 9.22.0 against system
9.23.2.
Five of those nine - llama-cpp, whisper, rfdetr-cpp, sam3-cpp,
stablediffusion-ggml - do not reference cuDNN at all. ggml goes through cuBLAS.
They were carrying ~57 MB of cuDNN with no consumer, and completing the family
for them would have taken that to ~576 MB for nothing.
Sizes overall: backends with no cuDNN consumer shed ~57 MB each (seven
instances on the fleet measured, plus longcat's ~60 MB), while the ones that
genuinely use cuDNN grow from ~57 MB to ~576 MB, because the five missing
sublibraries are ~517 MB, dominated by libcudnn_engines_precompiled. Net on
that fleet is an increase of roughly 570 MB. That growth is the bug being paid
off, not a regression: those backends only work today by silently borrowing the
missing five from the runtime image. Whether the engines set can be trimmed is
an open question, not addressed here.
So bundle per backend, by what that backend actually needs:
- venv has a complete pip cuDNN -> bundle nothing; $ORIGIN resolves the pip
set, which is the one its torch was built against (longcat-video)
- venv has no pip cuDNN -> bundle the complete family. Stays
conservative rather than detecting consumers: for a Python backend they sit
inside the venv (torch, ctranslate2, onnxruntime) where the sweep does not
look (vllm)
- no venv, nothing references cuDNN -> bundle nothing (llama-cpp, whisper,
rfdetr-cpp, sam3-cpp, stablediffusion-ggml)
- no venv, something references it -> bundle the complete family
(face-detect, voice-detect)
The no-venv case needs no new machinery. Go backends stage their own shared
object into package/lib, which IS the target dir, so sweep_transitive_deps
already pulls the dispatcher when it is a genuine dependency - that is exactly
how libcudnn_graph reached longcat. cuDNN simply comes off the force-copy list,
and complete_cudnn_family fills in the seven dlopen'd sublibraries around
whatever the sweep found. Detection is a string scan rather than ldd, so a
consumer that only dlopen()s cuDNN is seen too; over-matching costs an unused
library, under-matching costs a backend that cannot load.
Keeping bundled and pip versions in agreement instead is not viable: nothing
here pins nvidia-cudnn (zero occurrences), torch is unpinned for l4t13 except
longcat-video, and the fleet already runs five concurrent cuDNN versions -
9.19.0.56, 9.20.0.48, 9.22.0, 9.23.2, 9.24.0.
verify_cudnn_bundle asserts the end state: exactly one complete cuDNN visible to
whoever needs one - never both, never partial, and never zero for a backend that
references it. Zero is correct and common otherwise. It deliberately does not
accept the build image's system cuDNN as completing a partial bundle, which is
the shape that had been shipping silently; the build image is not the runtime
image. A version check alone would have missed longcat too, whose four bundled
libs were all 9.24.0 and mutually consistent.
Match per family for the other components for the same dlopen reason: TensorRT
(libnvinfer_plugin, libnvinfer_builder_resource), cuBLAS, cuFFT, cuSPARSE,
cuSOLVER, nvRTC. Exclusions bind inside copy_lib so they cover the sweep.
The packaging scripts' shell tests ran nowhere in CI. Add make
test-build-scripts and a lint workflow job so they gate every PR.
Fixes#10905
Assisted-by: Claude:claude-opus-4-8 golangci-lint shellcheck
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Pin vLLM protogen to grpcio-tools 1.78.0 so its generated code remains importable by protobuf 6.33.x, and remove stale generated artifacts before regeneration.
Closes#10940
Assisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
chore(moss-transcribe-cpp): bump pin to CUDA K-quant embed fix
Bumps the moss-transcribe.cpp pin to 190a569c, which merges the
host-side embed-lookup fallback for K-quant token_embd tensors
(localai-org/moss-transcribe.cpp#2).
Before this, running a q5_K/q4_K/q6_K moss-transcribe GGUF on CUDA
(or any non-CPU backend) aborted in getrows.cu with
"unsupported src0 type: q5_K" because ggml's GET_ROWS op has no
K-quant implementation on GPU, killing the backend process on the
first request. The engine now dequantizes the needed rows on the
host when the backend cannot run GET_ROWS for the tensor type,
producing bit-identical results.
Fixes#10862
Assisted-by: Claude:claude-opus-4-8 [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add moss-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the moss-tts.cpp ggml port of the OpenMOSS
MOSS-TTS-Local v1.5 text-to-speech model (GPT-J local transformer decoded through
MOSS-Audio-Tokenizer-v2), producing 48 kHz stereo audio with optional
reference-audio voice cloning. Mirrors the qwen3-tts-cpp backend: dlopen the
static-ggml shared library, bind the moss-tts.cpp C-API via purego, and serve
the gRPC TTS method. A thin C shim holds the pipeline handle and copies engine
PCM into a Go-freeable buffer.
Wires the CI registration: backend-matrix.yml (CPU, CUDA 12/13, Intel SYCL
f16/f32, Vulkan, ROCm, NVIDIA L4T, plus Darwin metal), backend/index.yaml metas
and image entries pointing at mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF, the
root Makefile build targets, and the changed-backends.js path mapping.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: list the moss-tts-cpp backend among the LocalAI-maintained engines
Add moss-tts.cpp to the README "Backends built by us" table, the
Text-to-Speech compatibility table, and the reference-audio voice-cloning
backend list, so the new backend is documented alongside its peers.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): pin moss-tts.cpp to the squashed single-commit release
moss-tts.cpp history was collapsed to a single commit; repoint MOSSTTS_CPP_VERSION
to ee722b8e9205ee9b1b1c398a4e87e4e393e9be41.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): add the moss-tts-cpp-development gallery meta
The gallery had the -development image entries but no matching -development
meta anchor (as locate-anything-cpp and depth-anything-cpp have), so the master
build was not installable as a gallery backend. Add moss-tts-cpp-development
mirroring the production meta with the -development capability image names.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): rewrite c2pa-audio submodule path for subproject builds
CrispASR a38cb89 adds a crispasr_c2pa_native static library whose sources
live in the new third_party/c2pa-audio git submodule, located via
CMAKE_SOURCE_DIR in src/CMakeLists.txt. That variable assumes CrispASR is
the top-level CMake project; LocalAI embeds it via add_subdirectory, so
the path resolved to backend/go/crispasr/third_party/c2pa-audio and every
build variant failed at CMake generate with 'Cannot find source file:
c2pa_native.cpp'.
Extend the existing talk-llama sed workaround to also rewrite the
c2pa-audio reference to PROJECT_SOURCE_DIR, which is correct both
standalone and as a subproject. The submodule itself is already checked
out by the recursive submodule init. Verified locally: the exact CI error
reproduces with CMAKE_SOURCE_DIR, and with the rewrite CMake configure,
crispasr_c2pa_native, and crispasr-lib all build cleanly on a CPU-only
fallback configuration.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The turboquant and bonsai backends copy backend/cpp/llama-cpp/ wholesale
into their build directories and reuse its Makefile/prepare.sh against
their own llama.cpp forks. When PR #10837 added
backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch, the
copied patches/ directory was mis-applied to the fork checkouts: the
fork trees diverge from upstream, hunks rejected, and because the
patch-apply loop in prepare.sh ran before set -e took effect the build
kept going and died much later with a confusing compile error
("'LLM_ARCH_MINIMAX_M3' was not declared in this scope"). This broke
tests-turboquant-grpc on that PR.
Two hardening changes:
- turboquant/bonsai Makefiles: delete the copied patches/ directory
right after the cp -rf of backend/cpp/llama-cpp/. Patches vendored
for upstream llama.cpp must never be applied to the forks; each fork
carries its own patch series under backend/cpp/<backend>/patches/,
applied by its apply-patches.sh.
- llama-cpp prepare.sh: run the patch-apply loop under set -e so a
rejecting patch fails fast and loudly at apply time instead of
surfacing as a downstream compile error. A missing or empty patches/
directory remains a no-op success, so all existing callers (the
llama-cpp Makefile targets and the turboquant/bonsai copies) are
unaffected when no patches ship.
Exposed by PR #10837.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The sglang Python backend inherits the default Status RPC from
backend_pb2_grpc.BackendServicer, which raises NotImplementedError.
LocalAI's backend-monitor polls /backend.Backend/Status periodically on
every registered backend; when the call fails, /backend/monitor returns
HTTP 500 and downstream inference requests to the sglang backend are
blocked even though the model is loaded and answering directly via the
gRPC endpoint.
Add a minimal Status shim that mirrors the existing Health method and
returns StatusResponse{state=READY} unconditionally. This unblocks the
monitor path; a state-aware follow-up (UNINITIALIZED during load, BUSY
under active inference) is left for a subsequent change.
Reproduced on DGX Spark (GB10, arm64-l4t-cuda-13 image) with the sglang
v0.5.15 backend and Qwen3-Coder-Next-NVFP4-GB10; verified locally that
patching the shim in place immediately restores /backend/monitor and
inference across the sglang slot.
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models
Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp
(github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0
(1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the
Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants.
Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp
against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models
are served with the same OpenAI-compatible API. No grpc-server allow-list patch
is needed (bonsai adds weight quants, transparent to the server, not KV-cache
types), and the reused server compiles cleanly against the fork with no skew
patches (validated locally via a CPU docker build; patches/ is present but empty
for any future re-pin skew).
Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh,
backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build
rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml
meta-backend + per-platform images, and a nightly bump_deps entry tracking the
`prism` branch.
Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64,
+pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B
models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not
wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
