Keep the GNU constant-folding diagnostics visible on Darwin without allowing vllm.cpp's global -Werror to fail the Metal backend build.
Assisted-by: Codex:gpt-5 [systematic-debugging]
The pin sat at f384edcd while vllm.cpp main moved a long way. The ABI is
unchanged at v10 and both POD structs are field-identical to the pinned commit
(verified by diffing vllm_model_params and vllm_sampling_params across the
range), so the Go mirror needs no edit and this is a clean bump.
What it picks up matters for this backend:
- MTP speculative decoding from a GGUF target, gated end to end on GPU.
- DFlash speculative decoding with a GGUF draft AND a GGUF target.
- NVFP4 GGUF: dequant, plus a native fp4 compute path for dense and
full-attention projections. On the 27B that closed a cross-container
divergence entirely (the GGUF and safetensors builds of the same
quantization run now emit identical tokens) and halved peak RSS.
- A real engine fix: the GDN speculative state gather/scatter was mis-striding
the widened conv row, so speculation silently corrupted the target's own
recurrent state on CPU.
Docs corrected accordingly. The section previously told users that mtp and
dflash are rejected on a .gguf target and called it a gap in the engine's GGUF
loader. That is no longer true, and leaving it would send people to safetensors
for no reason. A head-less GGUF is still refused, and the text now says so with
the actual cause.
The real-library ABI handshake was re-run against a libvllm.so built at the
exact pinned commit rather than a stale one: 43 specs pass, reported ABI 10.
make lint clean.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
vllm.cpp landed ABI v10 on main while this branch was open. The backend's
runtime handshake refuses any library whose reported ABI differs from the
mirrors', so the pin and the Go PODs move together or not at all.
v10 appends one int32, `enable_jump_forward`, AFTER the v9 fields. Nothing else
in the config surface changed: the SGLang reconciliation that carried it
explicitly dropped its own duplicate scheduler_policy int in favour of the v9
`scheduling_policy` string this branch already wires, and a diff of EngineParams
and the server flags across the window turns up jump forward and nothing else.
So the exposure is one new knob, `engine_args.enable_jump_forward` - SGLang's
grammar-speed subset, which emits grammar-forced tokens without spending a model
step and therefore only affects constrained requests.
It is the SECOND tri-state on this struct, and it repeats the trap the first one
had: 0 is not "off", it is "defer" (to the environment here, to the model
capability for prefix caching), so an explicit `false` has to reach the engine as
2. The bool->tri-state helper and the log renderer are now shared rather than
duplicated per field, and named for the encoding instead of for prefix caching,
since the next tri-state will want them too. The docs say this outright, because
"omitting the key" and "setting it false" being different is not guessable.
The Go mirror grows the field plus an EXPLICIT trailing pad: the struct is
8-aligned and now ends on a lone int32, so it is 88 bytes rather than 84. The
offset assertions cover it, and the real-library handshake spec (VLLM_CPP_LIBRARY
against a CPU libvllm.so built at the new pin) confirms the version agrees:
43 specs pass, ABI reported 10.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The engine resolves speculative_config.model against a directory containing
config.json, or against ~/.cache/huggingface/hub/models--<org>--<repo>/
snapshots/*, and it never downloads. LocalAI keeps models in its own directory,
so the repo-id spelling the vLLM docs teach - "z-lab/Qwen3.6-27B-DFlash" - misses
the HF cache and dies deep inside the load with "draft checkpoint not found",
which reads like a broken checkpoint rather than a model nobody fetched.
Resolve it before the load call: the reference as given, then its last path
segment under LocalAI's models dir (what LocalAI's own downloader produces),
then the whole reference under the models dir. When none resolve, fail there
naming both what was asked for and every location tried, so the message says
what to do about it.
mtp and ngram pass through untouched - neither has a separate draft checkpoint.
A speculative_config that does not parse also passes through, because the engine
owns config validation and produces the better error.
Docs also gain the two limits that were missing and are easy to lose an
afternoon to: speculation is Qwen3.5/3.6-only at this engine pin regardless of
format, and mtp/dflash need a safetensors target. The latter is a gap in the
engine's GGUF loader rather than a property of GGUF - the format carries MTP
weights fine, llama.cpp reads them as nextn.* tensors plus a
<arch>.nextn_predict_layers key - so the docs say that rather than implying GGUF
cannot express it.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
974d9d72 was the pre-merge commit on the feature branch. eec09bed is the merged
main tip carrying the same ABI v9 surface, and is the tree the capi suite
(31/31) and the Go ABI handshake were actually verified against.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The Go PODs in govllmcpp.go are hand-written against one VLLM_ABI_VERSION and
the Makefile pins the vllm.cpp commit that produces it. Nothing checked those
two agree short of the e2e suite, which needs a model to run at all, so a pin
bump could land with a stale mirror and only fail at a user's first load.
VLLM_CPP_LIBRARY now drives a handshake spec that dlopens a built libvllm,
binds every symbol, and compares the library's reported ABI against the
mirrors'. No weights required.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The vllm-cpp backend could configure four of the engine's knobs - block size,
KV block count, max sequence length and max concurrent sequences - out of a
config surface that is considerably larger. Speculative decoding, prefix
caching, the chunked-prefill token budget, the scheduling policy and the
external KV connector were all reachable from vllm.cpp's own HTTP server and
from nothing LocalAI could write in a model config.
Part of that gap was the C ABI itself, which carried strictly less than
EngineParams does; that is fixed upstream in vllm.cpp ABI v9 (this bumps the pin
to it). The rest was here: the backend parsed a flat `options:` list with five
recognised keys and had no way to express a nested JSON document at all.
Configuration now goes through `engine_args:`, the same map the vLLM and SGLang
backends already take, with keys spelled as vLLM's own CLI flags - so a
`speculative_config` or `kv_transfer_config` block written for vLLM works
verbatim:
engine_args:
max_num_batched_tokens: 8192
enable_prefix_caching: true
scheduling_policy: lpm
speculative_config:
method: dflash
model: z-lab/Qwen3.6-27B-DFlash
num_speculative_tokens: 4
kv_transfer_config:
kv_connector: LMCacheConnector
kv_role: kv_both
kv_connector_extra_config: {host: 127.0.0.1, port: 65432}
The `options:` list keeps working, and now reads every key too, so no existing
config breaks; engine_args wins where both set the same key.
Two details worth calling out. `enable_prefix_caching: false` maps to the ABI's
force-OFF state (2), not the 0 that means "let the model capability decide" -
collapsing them would silently turn the cache ON for the dense architectures
that default it on. And cSamplingParams grows the ABI v8 logits-processor tail:
LocalAI installs no processor, but the C side reads those fields off the pointer
we hand it, so a Go struct that stopped short would have had the engine read 16
bytes past our allocation and call whatever sat there.
The importer gets the safetensors counterpart of the llama-cpp MTP hook: a
`vllm-cpp` import of a HuggingFace repo probes config.json and, on a checkpoint
that declares an MTP head, writes speculative_config {method: mtp} into the
generated engine_args. DFlash draft repositories are detected and refused with a
warning rather than configured as standalone models, since a drafter cannot
serve alone. The llama-cpp importer stops applying its own `spec_type:draft-mtp`
options when the chosen backend is vllm-cpp: those are llama.cpp option keys
vllm-cpp does not read, and vllm.cpp rejects MTP over a GGUF source anyway
because the `mtp.*` draft tensors only exist in the safetensors checkpoint.
docs/content/features/text-generation.md gains a vllm.cpp section - the backend
had no documentation page at all - covering the engine_args table, all three
speculative methods, LMCache, and the legacy options list.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The Anthropic translate provider builds the upstream request from scratch and
never emitted cache_control, so prompt caching was impossible for OpenAI-format
clients routed through cloud-proxy — even though the entire system prompt + tools
prefix is re-sent on every agentic turn.
Add an opt-in cache_prompt flag (ProxyOptions.cache_prompt; model YAML
proxy.cache_prompt: true). On a translate+anthropic model, buildAnthropicRequest
injects cache_control:{type:ephemeral} on the stable prefix — the system block,
the last tool, and the last message block (at most 3 of Anthropic's 4 allowed
breakpoints). Anthropic then serves the repeated prefix at the cache-read rate
(0.1x input) on subsequent calls, cutting cost on multi-turn/agentic workloads.
No effect in passthrough mode, for non-Anthropic providers, or when unset.
System is widened to any so it can carry the block form required to attach
cache_control, while still marshalling as a bare string when caching is off.
Adds a unit test asserting exactly three breakpoints when on and none when off,
and documents the option in docs/content/operations/cloud-proxy.md.
Assisted-by: Claude:opus-4.8
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
* feat(backend): add vllm-cpp text-generation backend (vllm.cpp)
Wrap https://github.com/mudler/vllm.cpp - the LocalAI-team from-scratch C++20
port of vLLM (paged KV cache, continuous batching, prefix caching, safetensors
+ GGUF loading, no Python at inference) - as a Go gRPC backend over its stable
C ABI (ABI v2) via purego.
Backend (backend/go/vllm-cpp):
- Load -> vllm_engine_load: accepts a .gguf file or a config.json model dir
(anything else is refused, satisfying the greedy-probe rule); context_size
maps to max_model_len, options block_size/num_blocks/max_num_seqs size the
KV cache and scheduler admission.
- Predict -> vllm_complete (blocking); PredictStream -> vllm_complete_stream
with the per-delta C callback bridged into the gRPC stream. The backend
embeds base.Base (not SingleThread): concurrent requests batch continuously
in the engine's shared AsyncLLM scheduler.
- PredictOptions.Grammar -> the ABI's structured_grammar (GBNF), giving
grammar-constrained tool calling at parity with llama-cpp; the ABI also
exposes JSON-schema/regex/choice constraints.
- Hand-mirrored POD structs with layout locked by unit tests
(unsafe.Offsetof vs the C offsets) and a runtime vllm_abi_version gate.
- One portable library per platform (vllm.cpp uses per-file SIMD tiers with
runtime dispatch), so no avx/avx2/avx512 variant builds.
Wiring:
- backend-matrix: CPU amd64+arm64 (per-arch + manifest merge), CUDA 12/13
amd64 (120a;121a Blackwell fat binary), L4T arm64 (121a, GB10/DGX Spark -
the runtime-proven GPU target), Vulkan amd64, and Darwin arm64 Metal.
- backend/index.yaml meta + 12 image entries (latest/development x cpu,
cuda12, cuda13, l4t, vulkan, metal); bump_deps registration for the
VLLM_CPP_VERSION pin; root Makefile registration; test-extra runs the unit
specs (pure Go, no engine build).
- Importers: preference-only swaps - llama-cpp (GGUF) and vllm (safetensors)
advertise vllm-cpp via AdditionalBackends and emit backend: vllm-cpp
without tokenizer templating (the C ABI takes the FINAL prompt; templating
and tool parsing stay LocalAI-side). No auto-detect importer.
- Docs: backends list, top-level README maintained-engines table,
compatibility table.
Verified: 20/20 Ginkgo specs against the real pinned engine and
Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU - blocking + streaming parity, greedy
determinism, stop words, GBNF-constrained generation, and 4 concurrent
streams; plus a dlopen/ABI-gate smoke of the built gRPC server binary.
Upstream ABI v2 + production structured-output wiring landed as
mudler/vllm.cpp@86013f3.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): ride the autoparser code path - engine-side chat templating and tool engagement (ABI v3)
The backend now implements AIModelRich (PredictRich / PredictStreamRich) over
vllm.cpp's ABI v3 chat entry points, so chat and tool calling ride the SAME
code path as the llama.cpp autoparser: the ENGINE renders the model's chat
template, decides when a tool call engages, and parses it - LocalAI receives
pre-parsed ChatDelta / ToolCallDelta protos exactly as it does from llama-cpp.
- With use_tokenizer_template + structured Messages, PredictOptions lowers to
ONE OpenAI chat request JSON (messages, tools, tool_choice, sampling,
stream_options.include_usage) for vllm_chat / vllm_chat_stream. tool_choice
auto lowers engine-side to a LAZY structural-tag decode constraint - free
text until the model emits the tool trigger, then the call is
grammar-constrained; required/named force a call. Tool output is parsed by
the engine's streaming Hermes-style parser; each chat.completion.chunk maps
onto ChatDeltas (content / reasoning_content / tool_calls) which the host
already prefers over Go-side tag extraction. Without structured messages the
plain path (LocalAI templating + optional GBNF grammar) applies unchanged.
- The engine resolves the chat template from the GGUF tokenizer.chat_template
metadata (or tokenizer_config.json); templates beyond its minja subset -
e.g. the full Qwen3.5 namespace()/macro template - degrade engine-side to a
Hermes-aware fallback prompt (tools schemas + <tool_call> instruction) with
a stderr witness, so structural-tag engagement keeps working.
- Importers now emit the same config shape as llama-cpp for vllm-cpp
(use_tokenizer_template: true, no-grammar autoparser flow); only the
llama-cpp-specific use_jinja option and the vllm-python parser options are
dropped.
- Pin bumped to mudler/vllm.cpp@aaed7ec (ABI v3 + chat-prompt resolution).
Verified against the real engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU: full
suite green - blocking chat, streaming deltas concatenating byte-equal to the
blocking answer, a REQUIRED tool call returning schema-valid arguments JSON,
and an AUTO run where the engine itself engages get_weather and streams parsed
tool deltas; plus unit specs for the request lowering, chunk->ChatDelta
mapping, and the C struct mirrors (ABI gate now v3).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): ABI v5 - engine-side parser selection for 30 tool dialects + reasoning
Bump the vllm.cpp pin to the autoparser-parity engine: 30 tool-call dialects
(every pure-text parser in the pinned vLLM registry, each ported 1:1 with its
upstream tests), 7 reasoning parsers, google/minja as the template renderer
(the full Qwen3.5 template now renders engine-side), per-family structural
tags (tool_choice required/named compiles the model's NATIVE syntax where
expressible), and template auto-detection for both parser axes.
Backend changes:
- cModelParams mirrors ABI v5 (tool_parser + reasoning_parser fields,
layout-locked by the offset tests; ABI gate now v5).
- New model options tool_parser:<name> / reasoning_parser:<name> pass through
to the engine; unset means template auto-detection (18-row tool marker
table; [THINK]->mistral, <think>->think_auto for reasoning); "none"
disables the reasoning split; unknown names fail the first chat call.
- Chat chunks parse the `reasoning` field (the pin renamed
reasoning_content), flowing into ChatDelta.ReasoningContent which the host
already prefers.
Live e2e against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU, full suite green: the
real chat template renders (no more fallback), reasoning auto-detection picks
think_auto so markerless answers stay pure content (the live run caught the
deepseek_r1 content-swallow upstream and drove the think_auto fix), required
tool_choice returns schema-valid arguments, auto tool_choice engages
engine-side and streams parsed deltas, and blocking/streaming stay
byte-identical. Turn latency also dropped (proper template EOS behavior).
Upstream program landed as mudler/vllm.cpp 86013f3..5fffe7e (ABI v2-v5,
minja, parser waves B1/B2/B4, reasoning seam, structural-tag registry,
think_auto).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(vllm-cpp): bump the engine pin to the ENG-wave close-out
mudler/vllm.cpp@df8909b: the six engine-backed vLLM tool-parser families
(qwen3-coder/xml/mimo, kimi_k2, glm45/47, minimax_m2, gemma4, seed_oss)
text-reimplemented from their wire formats and held to the upstream test
suites - 39 registered dialects; the pinned vLLM registry is now covered
except the three Rust/Harmony-backed families, descoped by decision. kimi_k2
also gains a full native structural-tag builder; four new template
auto-detection rows land with test-pinned ordering.
Full backend e2e re-run green against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): add the vllm-cpp-development gallery meta
The gallery grew the twelve latest/development image entries but was missing
the separate vllm-cpp-development meta (own capabilities map targeting the
-development image names), which every backend ships so the development
gallery resolves per-platform. Validated: all capability targets in both
metas resolve to existing entries, and every image URI's tag suffix matches
a backend-matrix build.
Also full-stack verified in this change's context (single-node local-ai from
this branch, locally-built backend under --backends-path, Qwen3.5-2B GGUF):
/v1/chat/completions non-stream (clean content + usage), streaming (SSE
deltas), tool_choice auto engaging get_weather engine-side with schema-valid
arguments and finish_reason=tool_calls, and streamed tool-call deltas in the
standard name-first cadence.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): repair the CI backend builds - gcc-14 -Werror + fat-arch Triton
Two distinct failures took down all five vllm-cpp backend builds on the PR:
1. gcc-14 (ubuntu:24.04 CI images; the local toolchain is gcc-13) fails the
engine build with -Werror=maybe-uninitialized in InputBatch::condense - a
false positive through a staging std::optional's raw storage. Fixed
upstream (mudler/vllm.cpp@61f3e85) by moving slot-to-slot directly;
verified BOTH ways under dockerized g++-14.2 (unfixed reproduces CI's two
diagnostics exactly, fixed compiles clean) with the engine's behavior
suites green. Pin bumped to that sha.
2. The amd64 CUDA builds died at CMake configure: the vendored Triton-AOT
cubin trees are per-arch and the engine refuses -DVLLM_CPP_TRITON=ON on a
multi-arch (120a;121a) fat build unless pinned to one tree, which would be
unsound for the other arch. Triton is now enabled only on the single-arch
arm64/GB10 build (where the cubins matter); the fat amd64 binary uses the
engine's non-AOT GDN path.
Backend e2e re-run green at the new pin (Qwen3.5-2B on CPU, full suite).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): cuda-12 images cannot compile compute_121a - target 120a only
The second CI round surfaced a CUDA-version constraint: the cuda-12 (12.8)
image's nvcc rejects 'compute_121a' (GB10 arch support landed with CUDA 13),
killing the amd64 cuda-12 build at nvcc. Gate the architecture list on
CUDA_MAJOR_VERSION (exported by Dockerfile.golang): cuda-12 builds consumer
Blackwell 120a only, cuda-13 keeps the 120a;121a fat binary, arm64/l4t
(cuda-13) keeps single-arch 121a with the Triton cubins. GB10 is arm64, so
the amd64 cuda-12 image never served it - no capability change.
Verified by Makefile dry-run variable dumps for all three combinations
(cuda12 -> 120a; cuda13 -> 120a;121a; cpu -> CUDA off).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): drop the cuda-12 variant - the engine needs the CUDA 13 toolchain
Third CI round, third layer: with the arch list already narrowed to 120a,
the cuda-12 (12.8) build still dies in ptxas compiling the sm_120a NVFP4 MMA
kernels ("Vector type too large, exceeds 128 bit limit") - the Blackwell fp4
path genuinely requires the CUDA 13 toolchain, and vllm.cpp supports
Blackwell-family GPUs only. Shipping a cuda-12 image without the fp4 kernels
would be a crippled build of an engine whose whole GPU story is fp4, so the
variant is dropped instead:
- backend-matrix: cuda-12 vllm-cpp entry removed (cuda-13 amd64, l4t arm64,
cpu, vulkan, metal remain).
- gallery: cuda12 image entries removed; the nvidia capability now resolves
to the cuda13 image in both metas; the nvidia-cuda-12 key is dropped so
older-driver hosts fall back to the CPU image instead of an unrunnable one.
- backend Makefile: BUILD_TYPE=cublas under CUDA_MAJOR_VERSION=12 now fails
fast with a clear message; cuda-13 keeps the 120a;121a fat binary and
arm64/l4t keeps 121a with the Triton cubins.
Verified: Makefile branch dumps for all four combinations (cuda12 loud
error, cuda13 fat, arm64 121a+Triton, cpu off), YAML parses, matrix filter
tests green, gallery capability targets all resolve.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): forward multi-turn tool identity and reasoning to the engine
chatRequestJSON dropped Message.ToolCallId and Message.Name on role="tool"
replies and Message.ReasoningContent on assistant history, so a second
turn after tool execution reached the engine's chat template without the
fields that bind a tool result to the call it answers. Forward all three
(present-only, matching the OpenAI wire shape) and pin vllm.cpp to
6a0bd3e7, where ChatMessage parses/round-trips tool_calls, tool_call_id,
name and reasoning and the minja adapter exposes them to the template
context.
Adds the round-trip request-lowering spec (user -> assistant tool_call ->
tool reply -> lowered request) and re-ran the gated e2e suite against the
new engine pin with a real Qwen3.5 GGUF: chat, reasoning split, streaming
parity, required-tool and auto-tool cases all green.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): bump vllm.cpp for the darwin arm64 i8mm build fix
The darwin-metal CI job was the first build to compile the engine's arm
CPU-quant files on macOS and hit their Linux-only <asm/hwcap.h> /
<sys/auxv.h> includes. vllm.cpp 9e1c9025 detects i8mm per-OS (auxv on
Linux, sysctl on Apple Silicon) with kernels untouched. Gated e2e suite
re-run green against the new pin with a real Qwen3.5 GGUF.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): darwin build - bound cmake parallelism when nproc is absent
The macOS runners have no nproc, so JOBS evaluated empty and
`cmake --build -j$(JOBS)` became bare `-j`: unlimited clang jobs on a
3-core/7GB Mac, which swap-thrashed until the 6h GHA timeout (the log
shows "nproc: Command not found" and 7+ concurrent clang processes being
reaped at the cutoff). Use the same portable fallback chain as the other
darwin backends: nproc, then sysctl hw.ncpu, then 4.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add magpie-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the magpie-tts.cpp ggml port of NVIDIA's
Magpie TTS Multilingual 357M (encoder + autoregressive decoder over NanoCodec
tokens), producing 22.05 kHz mono audio in 5 baked voices (Aria, Jason, John,
Leo, Sofia; case-insensitive names or indices 0-4) across 9+ languages from a
single self-contained GGUF. Mirrors qwen3-tts-cpp / moss-tts-cpp: dlopen the
static-ggml shared library, bind the flat magpie_tts_capi_* C-API via purego
(no local C shim needed, the upstream .so exports it directly), and serve the
gRPC TTS + TTSStream methods behind base.SingleThread (the C context is not
reentrant across synthesize calls).
The backend CMakeLists translates the Makefile's -DGGML_{CUDA,METAL,VULKAN,HIP}
flags into upstream's MAGPIE_GGML_* toggles (upstream FORCE-overwrites the ggml
cache entries from those), pinned to magpie-tts.cpp v0.1.1
(e3f3dd1ebe22b64e7405f93b519f2d1930712568), which statically links ggml into
libmagpie-tts.so (ldd shows only system libs).
Wires the full registration: backend-matrix.yml (CPU amd64/arm64, CUDA 12/13,
Intel SYCL f16/f32, Vulkan amd64/arm64, ROCm, NVIDIA L4T + L4T CUDA 13, and
Darwin metal), backend/index.yaml metas and image entries, the root Makefile
build targets, the changed-backends backend-filter path mapping, the bump_deps
auto-bump matrix, a test-extra per-backend smoke job, the /backends/known
pref-only importer entry, the backend capabilities map (TTS + TTSStream, no
voice cloning), and the README / compatibility-table docs rows.
Verified locally: unit + e2e Ginkgo suites pass against the real q8_0 GGUF
(22.05 kHz mono WAV, RMS > 0.01), a live gRPC LoadModel + TTS round-trip
returns valid non-silent audio, and the pre-commit gates (make lint,
make test-coverage-check) pass, run manually with LOCALAI_TEST_HTTP_PORT
overriding the locally-occupied 9090.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* gallery: add magpie-tts-cpp model entries (q8_0 + f16)
Add the Magpie TTS Multilingual 357M GGUFs from mudler/magpie-tts.cpp-gguf to
the model gallery: q8_0 (~624 MB, near-lossless, fastest decode, recommended)
with an f16 (~784 MB) variant, both served by the magpie-tts-cpp backend.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* magpie-tts-cpp: bump pin to rewritten upstream v0.1.1 SHA
Upstream history was rewritten to purge accidentally committed build
artifacts; v0.1.1 now resolves to 6f7696cf.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The "Bump Backend dependencies" workflow has failed every night for over
ten days. Four upstreams — ced.cpp, moss-transcribe.cpp, voice-detect.cpp
and rf-detr.cpp — moved from the mudler org to localai-org, so the GitHub
API answers 301 for the old slugs. ced.cpp additionally renamed its
default branch to main.
bump_deps.sh fetched without -L or -f and never checked the response, so
the redirect's JSON body was passed straight to sed, which died with
"unterminated `s' command". The loud failure was luck: an error body
without slashes would have been substituted into the Makefile as the new
pin, silently corrupting the version and shipping it in a bump PR.
Point the matrix at the new slugs and branch, and harden the script so a
bad response can never reach sed: follow redirects, fail on HTTP errors,
and require a bare 40-hex SHA before rewriting anything. Also refresh the
now-stale repository URLs in the backend Makefiles, test scripts,
backend/index.yaml and the docs.
Verified all 25 matrix entries resolve to a commit SHA and that the four
previously-failing jobs run end to end against the real API.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): initialize only declared submodules
The latest upstream commit contains an undeclared CrispASR gitlink that makes a blanket recursive submodule update fail. Limit initialization to the two submodules used by the backend build.
Assisted-by: Codex:gpt-5 [Codex]
* fix(crispasr): resolve vendored WebRTC from project root
CrispASR now builds a vendored WebRTC VAD, but its include paths assume CrispASR is the top-level CMake project. Extend the existing embedded-project rewrite to the shared third_party root.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
chore(moss-transcribe-cpp): bump pin to CUDA K-quant embed fix
Bumps the moss-transcribe.cpp pin to 190a569c, which merges the
host-side embed-lookup fallback for K-quant token_embd tensors
(localai-org/moss-transcribe.cpp#2).
Before this, running a q5_K/q4_K/q6_K moss-transcribe GGUF on CUDA
(or any non-CPU backend) aborted in getrows.cu with
"unsupported src0 type: q5_K" because ggml's GET_ROWS op has no
K-quant implementation on GPU, killing the backend process on the
first request. The engine now dequantizes the needed rows on the
host when the backend cannot run GET_ROWS for the tensor type,
producing bit-identical results.
Fixes#10862
Assisted-by: Claude:claude-opus-4-8 [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add moss-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the moss-tts.cpp ggml port of the OpenMOSS
MOSS-TTS-Local v1.5 text-to-speech model (GPT-J local transformer decoded through
MOSS-Audio-Tokenizer-v2), producing 48 kHz stereo audio with optional
reference-audio voice cloning. Mirrors the qwen3-tts-cpp backend: dlopen the
static-ggml shared library, bind the moss-tts.cpp C-API via purego, and serve
the gRPC TTS method. A thin C shim holds the pipeline handle and copies engine
PCM into a Go-freeable buffer.
Wires the CI registration: backend-matrix.yml (CPU, CUDA 12/13, Intel SYCL
f16/f32, Vulkan, ROCm, NVIDIA L4T, plus Darwin metal), backend/index.yaml metas
and image entries pointing at mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF, the
root Makefile build targets, and the changed-backends.js path mapping.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: list the moss-tts-cpp backend among the LocalAI-maintained engines
Add moss-tts.cpp to the README "Backends built by us" table, the
Text-to-Speech compatibility table, and the reference-audio voice-cloning
backend list, so the new backend is documented alongside its peers.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): pin moss-tts.cpp to the squashed single-commit release
moss-tts.cpp history was collapsed to a single commit; repoint MOSSTTS_CPP_VERSION
to ee722b8e9205ee9b1b1c398a4e87e4e393e9be41.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): add the moss-tts-cpp-development gallery meta
The gallery had the -development image entries but no matching -development
meta anchor (as locate-anything-cpp and depth-anything-cpp have), so the master
build was not installable as a gallery backend. Add moss-tts-cpp-development
mirroring the production meta with the -development capability image names.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): rewrite c2pa-audio submodule path for subproject builds
CrispASR a38cb89 adds a crispasr_c2pa_native static library whose sources
live in the new third_party/c2pa-audio git submodule, located via
CMAKE_SOURCE_DIR in src/CMakeLists.txt. That variable assumes CrispASR is
the top-level CMake project; LocalAI embeds it via add_subdirectory, so
the path resolved to backend/go/crispasr/third_party/c2pa-audio and every
build variant failed at CMake generate with 'Cannot find source file:
c2pa_native.cpp'.
Extend the existing talk-llama sed workaround to also rewrite the
c2pa-audio reference to PROJECT_SOURCE_DIR, which is correct both
standalone and as a subproject. The submodule itself is already checked
out by the recursive submodule init. Verified locally: the exact CI error
reproduces with CMAKE_SOURCE_DIR, and with the rewrite CMake configure,
crispasr_c2pa_native, and crispasr-lib all build cleanly on a CPU-only
fallback configuration.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
