vllm.cpp's CUDA kernels need the CUDA 13 toolchain: 12.x nvcc lacks
compute_121a and its ptxas rejects the sm_120a NVFP4 MMA kernels, so
backend/go/vllm-cpp/Makefile ships no CUDA 12 variant and the arm64 CUDA
build targets sm_121a (GB10 / DGX Spark) only.
backend/index.yaml declared nvidia-l4t and nvidia-l4t-cuda-13 but no
nvidia-l4t-cuda-12, so a Jetson AGX Orin (sm_87, JetPack 6) fell through
SystemState.Capability's "default" catch-all and silently installed
cpu-vllm-cpp. That is the right build for that host, but it was
indistinguishable from an oversight both to a reader of the index and to
a user wondering why their GPU box pulled a CPU backend.
Map nvidia-cuda-12 and nvidia-l4t-cuda-12 explicitly at the CPU build,
state the Blackwell-only constraint in the gallery description and the
backend README, and add specs that pin the routing. No behaviour change:
these hosts already resolved to the CPU build via the catch-all.
The README also records the image-tag trap behind the same symptom on a
supported host: /run/localai/capability is baked in at image build time,
so a DGX Spark on the CUDA 12 -nvidia-l4t-arm64 image reports
nvidia-l4t-cuda-12 and gets the CPU build; -nvidia-l4t-arm64-cuda-13 (or
LOCALAI_FORCE_META_BACKEND_CAPABILITY) gets the GPU one.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
sherpa-onnx links onnxruntime's CUDA execution provider, and
libonnxruntime_providers_cuda.so carries cuDNN as a hard DT_NEEDED. The
onnxruntime GPU tarball ships no cuDNN of its own, and Dockerfile.golang
only installs libcudnn9 on the arm64 + CUDA 13 branch, so the amd64 CUDA
builders have none at all.
Since #10946 added the packaging guard, that combination is fatal rather
than silent: package-gpu-libs.sh reports 'cuDNN: venv=absent system=absent
-> bundle=detect', correctly detects the reference, finds nothing to copy
and refuses to emit the package. Both -gpu-nvidia-cuda-12-sherpa-onnx and
-gpu-nvidia-cuda-13-sherpa-onnx have failed to build since 2026-07-19, so
neither image has been published. Before the guard existed they shipped
without cuDNN and failed at load time instead.
Install the runtime package for this backend only. The auto-detection
bundles solely what a package references, so no other backend would grow,
but every Go CUDA builder would pay ~1.1 GB of layer and registry cache
for a library ggml never calls.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Bump LLAMA_VERSION to 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3 (73 commits
touching common/, src/ and tools/server/). Two upstream changes break the
backend:
* ggml-org/llama.cpp#20834 folded common_params::use_mmap / use_mlock /
use_direct_io into a single `load_mode` enum. LocalAI still exposes the three
as independent settings (`mmap`, `mmlock`, and the `direct_io` option), so
params_parse folds them once all three have been read, keeping the precedence
the separate booleans had: direct I/O bypasses the page cache, mlock implies
mmap, everything off is a plain buffered read. turboquant and bonsai compile
this same grpc-server.cpp against forks that predate the refactor, so
prepare.sh probes the checkout for LLAMA_LOAD_MODE_MMAP and generates
llama_compat.h with LOCALAI_LEGACY_LOAD_MODE set accordingly. Probing beats a
per-fork build flag here because the fork flavor targets disagree on whether
they forward CMAKE_ARGS or EXTRA_CMAKE_ARGS, and it heals itself once a fork
rebases past the refactor.
* The MiniMax M3 patch no longer applies. Upstream merged the model half of
llama.cpp#24523 (LLM_ARCH_MINIMAX_M3, src/models/minimax-m3.cpp, the gguf-py
constants and conversion/minimax.py) but not the chat half, so the patch is
re-cut to carry only the common/chat.cpp template detection and PEG parser,
rebased onto the new pin and onto the thinking_end_tag -> thinking_end_tags
rename. Dropping it wholesale (as #11008 did, reverted in #11136) would have
silently regressed MiniMax M3 tool calling and thinking.
Verified with a CPU docker build of the backend plus LoadModel and Predict
against a real GGUF over gRPC in all four load modes.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add vllm-cpp text-generation backend (vllm.cpp)
Wrap https://github.com/mudler/vllm.cpp - the LocalAI-team from-scratch C++20
port of vLLM (paged KV cache, continuous batching, prefix caching, safetensors
+ GGUF loading, no Python at inference) - as a Go gRPC backend over its stable
C ABI (ABI v2) via purego.
Backend (backend/go/vllm-cpp):
- Load -> vllm_engine_load: accepts a .gguf file or a config.json model dir
(anything else is refused, satisfying the greedy-probe rule); context_size
maps to max_model_len, options block_size/num_blocks/max_num_seqs size the
KV cache and scheduler admission.
- Predict -> vllm_complete (blocking); PredictStream -> vllm_complete_stream
with the per-delta C callback bridged into the gRPC stream. The backend
embeds base.Base (not SingleThread): concurrent requests batch continuously
in the engine's shared AsyncLLM scheduler.
- PredictOptions.Grammar -> the ABI's structured_grammar (GBNF), giving
grammar-constrained tool calling at parity with llama-cpp; the ABI also
exposes JSON-schema/regex/choice constraints.
- Hand-mirrored POD structs with layout locked by unit tests
(unsafe.Offsetof vs the C offsets) and a runtime vllm_abi_version gate.
- One portable library per platform (vllm.cpp uses per-file SIMD tiers with
runtime dispatch), so no avx/avx2/avx512 variant builds.
Wiring:
- backend-matrix: CPU amd64+arm64 (per-arch + manifest merge), CUDA 12/13
amd64 (120a;121a Blackwell fat binary), L4T arm64 (121a, GB10/DGX Spark -
the runtime-proven GPU target), Vulkan amd64, and Darwin arm64 Metal.
- backend/index.yaml meta + 12 image entries (latest/development x cpu,
cuda12, cuda13, l4t, vulkan, metal); bump_deps registration for the
VLLM_CPP_VERSION pin; root Makefile registration; test-extra runs the unit
specs (pure Go, no engine build).
- Importers: preference-only swaps - llama-cpp (GGUF) and vllm (safetensors)
advertise vllm-cpp via AdditionalBackends and emit backend: vllm-cpp
without tokenizer templating (the C ABI takes the FINAL prompt; templating
and tool parsing stay LocalAI-side). No auto-detect importer.
- Docs: backends list, top-level README maintained-engines table,
compatibility table.
Verified: 20/20 Ginkgo specs against the real pinned engine and
Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU - blocking + streaming parity, greedy
determinism, stop words, GBNF-constrained generation, and 4 concurrent
streams; plus a dlopen/ABI-gate smoke of the built gRPC server binary.
Upstream ABI v2 + production structured-output wiring landed as
mudler/vllm.cpp@86013f3.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): ride the autoparser code path - engine-side chat templating and tool engagement (ABI v3)
The backend now implements AIModelRich (PredictRich / PredictStreamRich) over
vllm.cpp's ABI v3 chat entry points, so chat and tool calling ride the SAME
code path as the llama.cpp autoparser: the ENGINE renders the model's chat
template, decides when a tool call engages, and parses it - LocalAI receives
pre-parsed ChatDelta / ToolCallDelta protos exactly as it does from llama-cpp.
- With use_tokenizer_template + structured Messages, PredictOptions lowers to
ONE OpenAI chat request JSON (messages, tools, tool_choice, sampling,
stream_options.include_usage) for vllm_chat / vllm_chat_stream. tool_choice
auto lowers engine-side to a LAZY structural-tag decode constraint - free
text until the model emits the tool trigger, then the call is
grammar-constrained; required/named force a call. Tool output is parsed by
the engine's streaming Hermes-style parser; each chat.completion.chunk maps
onto ChatDeltas (content / reasoning_content / tool_calls) which the host
already prefers over Go-side tag extraction. Without structured messages the
plain path (LocalAI templating + optional GBNF grammar) applies unchanged.
- The engine resolves the chat template from the GGUF tokenizer.chat_template
metadata (or tokenizer_config.json); templates beyond its minja subset -
e.g. the full Qwen3.5 namespace()/macro template - degrade engine-side to a
Hermes-aware fallback prompt (tools schemas + <tool_call> instruction) with
a stderr witness, so structural-tag engagement keeps working.
- Importers now emit the same config shape as llama-cpp for vllm-cpp
(use_tokenizer_template: true, no-grammar autoparser flow); only the
llama-cpp-specific use_jinja option and the vllm-python parser options are
dropped.
- Pin bumped to mudler/vllm.cpp@aaed7ec (ABI v3 + chat-prompt resolution).
Verified against the real engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU: full
suite green - blocking chat, streaming deltas concatenating byte-equal to the
blocking answer, a REQUIRED tool call returning schema-valid arguments JSON,
and an AUTO run where the engine itself engages get_weather and streams parsed
tool deltas; plus unit specs for the request lowering, chunk->ChatDelta
mapping, and the C struct mirrors (ABI gate now v3).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): ABI v5 - engine-side parser selection for 30 tool dialects + reasoning
Bump the vllm.cpp pin to the autoparser-parity engine: 30 tool-call dialects
(every pure-text parser in the pinned vLLM registry, each ported 1:1 with its
upstream tests), 7 reasoning parsers, google/minja as the template renderer
(the full Qwen3.5 template now renders engine-side), per-family structural
tags (tool_choice required/named compiles the model's NATIVE syntax where
expressible), and template auto-detection for both parser axes.
Backend changes:
- cModelParams mirrors ABI v5 (tool_parser + reasoning_parser fields,
layout-locked by the offset tests; ABI gate now v5).
- New model options tool_parser:<name> / reasoning_parser:<name> pass through
to the engine; unset means template auto-detection (18-row tool marker
table; [THINK]->mistral, <think>->think_auto for reasoning); "none"
disables the reasoning split; unknown names fail the first chat call.
- Chat chunks parse the `reasoning` field (the pin renamed
reasoning_content), flowing into ChatDelta.ReasoningContent which the host
already prefers.
Live e2e against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU, full suite green: the
real chat template renders (no more fallback), reasoning auto-detection picks
think_auto so markerless answers stay pure content (the live run caught the
deepseek_r1 content-swallow upstream and drove the think_auto fix), required
tool_choice returns schema-valid arguments, auto tool_choice engages
engine-side and streams parsed deltas, and blocking/streaming stay
byte-identical. Turn latency also dropped (proper template EOS behavior).
Upstream program landed as mudler/vllm.cpp 86013f3..5fffe7e (ABI v2-v5,
minja, parser waves B1/B2/B4, reasoning seam, structural-tag registry,
think_auto).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(vllm-cpp): bump the engine pin to the ENG-wave close-out
mudler/vllm.cpp@df8909b: the six engine-backed vLLM tool-parser families
(qwen3-coder/xml/mimo, kimi_k2, glm45/47, minimax_m2, gemma4, seed_oss)
text-reimplemented from their wire formats and held to the upstream test
suites - 39 registered dialects; the pinned vLLM registry is now covered
except the three Rust/Harmony-backed families, descoped by decision. kimi_k2
also gains a full native structural-tag builder; four new template
auto-detection rows land with test-pinned ordering.
Full backend e2e re-run green against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): add the vllm-cpp-development gallery meta
The gallery grew the twelve latest/development image entries but was missing
the separate vllm-cpp-development meta (own capabilities map targeting the
-development image names), which every backend ships so the development
gallery resolves per-platform. Validated: all capability targets in both
metas resolve to existing entries, and every image URI's tag suffix matches
a backend-matrix build.
Also full-stack verified in this change's context (single-node local-ai from
this branch, locally-built backend under --backends-path, Qwen3.5-2B GGUF):
/v1/chat/completions non-stream (clean content + usage), streaming (SSE
deltas), tool_choice auto engaging get_weather engine-side with schema-valid
arguments and finish_reason=tool_calls, and streamed tool-call deltas in the
standard name-first cadence.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): repair the CI backend builds - gcc-14 -Werror + fat-arch Triton
Two distinct failures took down all five vllm-cpp backend builds on the PR:
1. gcc-14 (ubuntu:24.04 CI images; the local toolchain is gcc-13) fails the
engine build with -Werror=maybe-uninitialized in InputBatch::condense - a
false positive through a staging std::optional's raw storage. Fixed
upstream (mudler/vllm.cpp@61f3e85) by moving slot-to-slot directly;
verified BOTH ways under dockerized g++-14.2 (unfixed reproduces CI's two
diagnostics exactly, fixed compiles clean) with the engine's behavior
suites green. Pin bumped to that sha.
2. The amd64 CUDA builds died at CMake configure: the vendored Triton-AOT
cubin trees are per-arch and the engine refuses -DVLLM_CPP_TRITON=ON on a
multi-arch (120a;121a) fat build unless pinned to one tree, which would be
unsound for the other arch. Triton is now enabled only on the single-arch
arm64/GB10 build (where the cubins matter); the fat amd64 binary uses the
engine's non-AOT GDN path.
Backend e2e re-run green at the new pin (Qwen3.5-2B on CPU, full suite).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): cuda-12 images cannot compile compute_121a - target 120a only
The second CI round surfaced a CUDA-version constraint: the cuda-12 (12.8)
image's nvcc rejects 'compute_121a' (GB10 arch support landed with CUDA 13),
killing the amd64 cuda-12 build at nvcc. Gate the architecture list on
CUDA_MAJOR_VERSION (exported by Dockerfile.golang): cuda-12 builds consumer
Blackwell 120a only, cuda-13 keeps the 120a;121a fat binary, arm64/l4t
(cuda-13) keeps single-arch 121a with the Triton cubins. GB10 is arm64, so
the amd64 cuda-12 image never served it - no capability change.
Verified by Makefile dry-run variable dumps for all three combinations
(cuda12 -> 120a; cuda13 -> 120a;121a; cpu -> CUDA off).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): drop the cuda-12 variant - the engine needs the CUDA 13 toolchain
Third CI round, third layer: with the arch list already narrowed to 120a,
the cuda-12 (12.8) build still dies in ptxas compiling the sm_120a NVFP4 MMA
kernels ("Vector type too large, exceeds 128 bit limit") - the Blackwell fp4
path genuinely requires the CUDA 13 toolchain, and vllm.cpp supports
Blackwell-family GPUs only. Shipping a cuda-12 image without the fp4 kernels
would be a crippled build of an engine whose whole GPU story is fp4, so the
variant is dropped instead:
- backend-matrix: cuda-12 vllm-cpp entry removed (cuda-13 amd64, l4t arm64,
cpu, vulkan, metal remain).
- gallery: cuda12 image entries removed; the nvidia capability now resolves
to the cuda13 image in both metas; the nvidia-cuda-12 key is dropped so
older-driver hosts fall back to the CPU image instead of an unrunnable one.
- backend Makefile: BUILD_TYPE=cublas under CUDA_MAJOR_VERSION=12 now fails
fast with a clear message; cuda-13 keeps the 120a;121a fat binary and
arm64/l4t keeps 121a with the Triton cubins.
Verified: Makefile branch dumps for all four combinations (cuda12 loud
error, cuda13 fat, arm64 121a+Triton, cpu off), YAML parses, matrix filter
tests green, gallery capability targets all resolve.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): forward multi-turn tool identity and reasoning to the engine
chatRequestJSON dropped Message.ToolCallId and Message.Name on role="tool"
replies and Message.ReasoningContent on assistant history, so a second
turn after tool execution reached the engine's chat template without the
fields that bind a tool result to the call it answers. Forward all three
(present-only, matching the OpenAI wire shape) and pin vllm.cpp to
6a0bd3e7, where ChatMessage parses/round-trips tool_calls, tool_call_id,
name and reasoning and the minja adapter exposes them to the template
context.
Adds the round-trip request-lowering spec (user -> assistant tool_call ->
tool reply -> lowered request) and re-ran the gated e2e suite against the
new engine pin with a real Qwen3.5 GGUF: chat, reasoning split, streaming
parity, required-tool and auto-tool cases all green.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): bump vllm.cpp for the darwin arm64 i8mm build fix
The darwin-metal CI job was the first build to compile the engine's arm
CPU-quant files on macOS and hit their Linux-only <asm/hwcap.h> /
<sys/auxv.h> includes. vllm.cpp 9e1c9025 detects i8mm per-OS (auxv on
Linux, sysctl on Apple Silicon) with kernels untouched. Gated e2e suite
re-run green against the new pin with a real Qwen3.5 GGUF.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): darwin build - bound cmake parallelism when nproc is absent
The macOS runners have no nproc, so JOBS evaluated empty and
`cmake --build -j$(JOBS)` became bare `-j`: unlimited clang jobs on a
3-core/7GB Mac, which swap-thrashed until the 6h GHA timeout (the log
shows "nproc: Command not found" and 7+ concurrent clang processes being
reaped at the cutoff). Use the same portable fallback chain as the other
darwin backends: nproc, then sysctl hw.ncpu, then 4.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add magpie-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the magpie-tts.cpp ggml port of NVIDIA's
Magpie TTS Multilingual 357M (encoder + autoregressive decoder over NanoCodec
tokens), producing 22.05 kHz mono audio in 5 baked voices (Aria, Jason, John,
Leo, Sofia; case-insensitive names or indices 0-4) across 9+ languages from a
single self-contained GGUF. Mirrors qwen3-tts-cpp / moss-tts-cpp: dlopen the
static-ggml shared library, bind the flat magpie_tts_capi_* C-API via purego
(no local C shim needed, the upstream .so exports it directly), and serve the
gRPC TTS + TTSStream methods behind base.SingleThread (the C context is not
reentrant across synthesize calls).
The backend CMakeLists translates the Makefile's -DGGML_{CUDA,METAL,VULKAN,HIP}
flags into upstream's MAGPIE_GGML_* toggles (upstream FORCE-overwrites the ggml
cache entries from those), pinned to magpie-tts.cpp v0.1.1
(e3f3dd1ebe22b64e7405f93b519f2d1930712568), which statically links ggml into
libmagpie-tts.so (ldd shows only system libs).
Wires the full registration: backend-matrix.yml (CPU amd64/arm64, CUDA 12/13,
Intel SYCL f16/f32, Vulkan amd64/arm64, ROCm, NVIDIA L4T + L4T CUDA 13, and
Darwin metal), backend/index.yaml metas and image entries, the root Makefile
build targets, the changed-backends backend-filter path mapping, the bump_deps
auto-bump matrix, a test-extra per-backend smoke job, the /backends/known
pref-only importer entry, the backend capabilities map (TTS + TTSStream, no
voice cloning), and the README / compatibility-table docs rows.
Verified locally: unit + e2e Ginkgo suites pass against the real q8_0 GGUF
(22.05 kHz mono WAV, RMS > 0.01), a live gRPC LoadModel + TTS round-trip
returns valid non-silent audio, and the pre-commit gates (make lint,
make test-coverage-check) pass, run manually with LOCALAI_TEST_HTTP_PORT
overriding the locally-occupied 9090.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* gallery: add magpie-tts-cpp model entries (q8_0 + f16)
Add the Magpie TTS Multilingual 357M GGUFs from mudler/magpie-tts.cpp-gguf to
the model gallery: q8_0 (~624 MB, near-lossless, fastest decode, recommended)
with an f16 (~784 MB) variant, both served by the magpie-tts-cpp backend.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* magpie-tts-cpp: bump pin to rewritten upstream v0.1.1 SHA
Upstream history was rewritten to purge accidentally committed build
artifacts; v0.1.1 now resolves to 6f7696cf.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix(trl): disable inline GRPO reward code by default (RCE)
POST /api/fine-tuning/jobs accepts reward_functions[].code, an inline Python
body, and compile_inline_reward() execs it against a restricted-builtins
allowlist (_SAFE_BUILTINS). That allowlist is not a security boundary:
().__class__.__bases__[0].__subclasses__() reaches os._wrap_close and thus
os.system, giving arbitrary code execution. The fine-tuning endpoint is
unauthenticated by default, so any caller could run code on the host.
Hardening the allowlist is a losing game against CPython introspection, so
inline reward code is now refused unless the operator explicitly opts in with
LOCALAI_TRL_ALLOW_INLINE_REWARD=true on the backend. Builtin reward functions
are unaffected. The gate lives in build_reward_functions(), the single point
all inline specs flow through. Docs updated to stop describing the allowlist as
a sandbox and to document the opt-in.
Fixes#11015
Signed-off-by: Isabel Wu <231155141+wuisabel-gif@users.noreply.github.com>
Co-authored-by: Isabel Wu <231155141+wuisabel-gif@users.noreply.github.com>
The Python gRPC bindings expose message fields as plain attributes
(request.language, request.caption), not Go/Java-style Get*() accessors.
Because request.language is an empty string when unset, the
request.language or request.GetLanguage() or "en"
expression falls through to request.GetLanguage(), which does not exist
on the generated Python message and raises AttributeError, surfaced to
clients as:
rpc error: code = Unknown desc = Exception calling application: GetLanguage
Every /v1/sound-generation request without an explicit language field
failed. Drop the bogus accessor calls (TTS already uses the plain-field
form a few lines below).
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(backends): choose the protoc generator from the protobuf runtime, and regenerate stubs after late installs
The vLLM backends still crash on startup with
VersionError: Detected incompatible Protobuf Gencode/Runtime versions when
loading backend.proto: gencode 7.35.0 runtime 6.33.6
despite #10735 and #10944. Three separate defects kept it alive.
1. runProtogen picked the generator from the installed *grpcio* version.
grpcio-tools' version tracks grpcio, but the gencode its bundled protoc
emits tracks *protobuf*, and the two move independently: grpcio-tools
1.82.1 (the version #10735 pins to, matching grpcio 1.82.1) requires
protobuf>=7.35.1 and stamps gencode 7.35.0. Pinning to grpcio could
therefore never constrain the gencode. Constrain the install to the
protobuf already in the venv instead and let the resolver pick the newest
compatible grpcio-tools. That both selects a generator the runtime accepts
and stops protogen from moving the runtime under the backend's other deps.
This is self-correcting, so the hardcoded GRPCIO_TOOLS_VERSION=1.78.0
escape hatch from #10944 is no longer needed and is removed.
2. The stubs were generated too early. Most branches of vllm/install.sh (and
vllm-omni) install vllm *after* installRequirements, and vllm re-resolves
the protobuf runtime as it lands. Stubs generated against the pre-vllm
runtime can end up newer than the runtime that finally ships, which is the
ROCm failure exactly. Regenerate once the dependency set is final.
3. rm -f of the .py sources left __pycache__ behind. CPython validates a .pyc
against source mtime and size, both of which can be unchanged across a
regeneration (the gencode triple is the same width whether it reads 7.35.0
or 6.33.5), so a stale backend_pb2.pyc could shadow the stub just written.
Also fail the build when the generated stub cannot be imported, so a
gencode/runtime mismatch surfaces at image build time instead of reaching
users as an opaque "grpc service not ready".
Verified by driving the real runProtogen through the ROCm install sequence in
a venv harness: before, gencode 7.35.0 against runtime 6.33.6 (reproducing the
reported error verbatim); after, gencode 6.33.5 against runtime 6.33.6 and the
stub imports cleanly.
Closes#10940Closes#10718
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): regenerate protobuf stubs in the other backends that install after installRequirements
Same defect as the vllm change: installRequirements generates the stubs at the
end of its own run, so any backend that installs further packages afterwards can
have the protobuf runtime moved out from under stubs that were already written.
The gencode stamped into backend_pb2.py then exceeds the runtime that ships and
the backend dies at model load with "grpc service not ready".
fish-speech already had this bug and worked around the symptom: it forces
protobuf>=5.29.0 after installRequirements precisely because "transitive deps
(wandb, tensorboard) may downgrade protobuf to 3.x but our generated
backend_pb2.py requires protobuf 5+". Regenerating after the pin addresses the
cause rather than propping up the runtime to match stale stubs.
Applied to the backends whose post-installRequirements step resolves a
dependency graph and can therefore move protobuf:
fish-speech -e . plus an explicit protobuf install
vibevoice pip install . (with deps)
llama-cpp-quantization gguf / GGUF_PIP_SPEC
trl gguf / GGUF_PIP_SPEC
Deliberately not applied to ace-step and chatterbox (both --no-deps, so the
dependency graph cannot change) or voxcpm (pins setuptools only). gguf does not
depend on protobuf today, but it resolves dependencies, and "this package does
not touch protobuf right now" is exactly the assumption that made the earlier
fix ineffective.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): resolve the protoc generator in a throwaway env so it cannot edit the backend's pinned deps
Installing grpcio-tools into the backend's own venv to generate the stubs also
drags its dependencies in: grpcio-tools 1.82.1 requires grpcio>=1.82.1, so a
backend that pinned grpcio==1.78.1 silently shipped 1.82.1 instead. Caught by
building the llama-cpp-quantization image and reading the versions back out of
the artifact:
before grpcio 1.82.1 (requirements.txt pins grpcio==1.78.1)
after grpcio 1.78.1 grpcio-tools absent from the venv entirely
Resolve the generator in a throwaway environment instead, still constrained to
the protobuf the backend ships so the gencode stays compatible. The backend's
dependency set is then exactly what its requirements files declared. protoc's
output is plain Python and carries no dependency on the interpreter that
produced it, so generating from a different env is safe; the import check still
runs under the backend's python, since that is the interpreter that has to load
the stubs at model load.
Verified on the rebuilt image: gencode 7.35.0, runtime protobuf 7.35.1, grpcio
back at its pinned 1.78.1, and the shipped stub imports cleanly against 7.35.1.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): bound the protoc generator by BOTH the installed grpcio and protobuf
The generated stubs impose two independent constraints, and every fix so far,
including the previous commit on this branch, satisfied one while violating the
other:
backend_pb2.py needs protobuf runtime >= gencode
backend_pb2_grpc.py needs installed grpcio >= grpcio-tools
Resolving the generator against protobuf alone picked grpcio-tools 1.82.1 for a
backend holding grpcio at 1.78.1, so the gencode was fine but the gRPC stub was
not:
RuntimeError: The grpc package installed is at version 1.78.1, but the
generated code in backend_pb2_grpc.py depends on grpcio>=1.82.1.
That is also why installing grpcio-tools into the backend venv appeared to work
earlier: it dragged grpcio up to match, which was load-bearing rather than the
regression it looked like. Isolating the generator removed the accidental fix
and exposed the missing constraint.
Bound grpcio-tools from both sides instead and let the resolver find the newest
version satisfying both. The protobuf ceiling makes it back off to an older
generator when the runtime trails, bounding the gencode; the grpcio ceiling
keeps the _grpc stub loadable. Resolved against the four real runtime pairs
observed in built images:
grpcio 1.78.1 / protobuf 7.35.1 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.78.0 / protobuf 6.33.6 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.82.1 / protobuf 6.33.6 -> grpcio-tools 1.81.1, gencode 6.33.5 OK
grpcio 1.82.1 / protobuf 7.35.1 -> grpcio-tools 1.82.1, gencode 7.35.0 OK
Also restore the import check to cover backend_pb2_grpc as well as backend_pb2.
Narrowing it to backend_pb2 is why the image build passed while CI failed: the
guard could not see the constraint that was actually broken.
Verified by running the CI sequence locally for llama-cpp-quantization, the
backend whose test failed:
make -C backend/python/llama-cpp-quantization -> exit 0
make -C backend/python/llama-cpp-quantization test -> exit 0, OK
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update TheTom/llama-cpp-turboquant
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(turboquant): refresh HIP compatibility patch
The updated fork now carries its own HIP-safe peer-copy path, so the old hunk no longer applies. Keep only the event-creation compatibility change that the fork still needs.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(backend/python): don't await sync servicer behaviors in AsyncModelIdentityInterceptor
The model-identity interceptor (added for #10952) is installed on every Python
backend's gRPC server. Its grpc.aio variant invokes the wrapped servicer
behavior itself and awaits the result unconditionally:
result = await original(request, context) # LoadModel
return await original_unary(request, context) # guarded RPCs
async for response in original_stream(request, context): # streaming
But a backend's servicer methods may be plain sync functions. The transformers
backend, for one, defines `def LoadModel` and `def Embedding` (not `async def`).
grpc.aio's own dispatch adapts both shapes, but this interceptor calls the
behavior directly and bypasses that. For a sync method `original(...)` returns a
message object, not a coroutine, so the `await` raises:
TypeError: object Result can't be used in 'await' expression
The model loads, then the LoadModel RPC dies on return; the guarded sync
Embedding fails the same way. It happens on every platform, not just one backend
build. CI never caught it because AsyncModelIdentityInterceptor had no
behavioral test -- only an "is it installed" assertion.
Fix: await only when the behavior actually returned an awaitable
(inspect.isawaitable), mirroring grpc.aio's own sync/async adaptation. The
streaming guard iterates a sync generator with `for` and an async one with
`async for`.
Adds async-path coverage to model_identity_test.py exercising both sync and
async LoadModel / guarded-unary / streaming behaviors. The sync cases fail on
the current code with the TypeError above and pass with this fix.
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
* fix(backend/python): dispatch sync servicer behaviors off the event loop
Addresses review feedback: awaiting only awaitable results removed the
TypeError, but still ran a sync LoadModel/Embedding -- and stepped a sync stream
via next() -- on the asyncio event-loop thread, so a slow load/inference/stream
could freeze all aio RPC handling.
Route sync behavior through run_in_executor (a worker thread) while awaiting
native async behavior directly. A callable wrapper that returns an awaitable is
run in the thread and its awaitable awaited back on the loop. Sync streaming
pulls each item via the executor with a done sentinel, so StopIteration cannot
escape through a Future.
Adds regression tests that record the handler thread id and assert it differs
from the event-loop thread, for LoadModel, a guarded unary RPC and a sync stream.
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
---------
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link new tensor parallel objects
The updated ds4 revision split tensor-parallel transport and layer placement into separate translation units. Build and link those objects on CPU, CUDA, and Metal builds.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The "Bump Backend dependencies" workflow has failed every night for over
ten days. Four upstreams — ced.cpp, moss-transcribe.cpp, voice-detect.cpp
and rf-detr.cpp — moved from the mudler org to localai-org, so the GitHub
API answers 301 for the old slugs. ced.cpp additionally renamed its
default branch to main.
bump_deps.sh fetched without -L or -f and never checked the response, so
the redirect's JSON body was passed straight to sed, which died with
"unterminated `s' command". The loud failure was luck: an error body
without slashes would have been substituted into the Makefile as the new
pin, silently corrupting the version and shipping it in a bump PR.
Point the matrix at the new slugs and branch, and harden the script so a
bad response can never reach sed: follow redirects, fail on HTTP errors,
and require a bare 40-hex SHA before rewriting anything. Also refresh the
now-stale repository URLs in the backend Makefiles, test scripts,
backend/index.yaml and the docs.
Verified all 25 matrix entries resolve to a commit SHA and that the four
previously-failing jobs run end to end against the real API.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
#10970 gave the four PredictOptions RPCs a model-identity check so a
backend reached through a stale distributed route rejects the request
instead of answering from whatever model it holds (#10952). Every other
modality shares that exposure: the route is cached by host:port, a worker
can recycle a stopped backend's port for another model's backend, and a
liveness-only probe cannot tell a stale row from a valid one.
Extends the same mechanism to the 21 remaining request messages that reach
a backend through the router, using the pattern #10970 established rather
than a parallel one:
- proto: ModelIdentity on each modality request message.
- controller: populated from ModelConfig.Model at the call site that also
builds ModelOptions, so load-time and request-time values are equal by
construction.
- backends: one generic guard in pkg/grpc/server.go (27 Go backends), the
method set in backend/python/common (36 Python backends), llama-cpp
(AudioTranscription/Stream, Rerank, Score) and privacy-filter
(TokenClassify).
- reconcile already drops the stale row on IsModelMismatch; no change.
TTSRequest and SoundGenerationRequest get a SEPARATE ModelIdentity field
rather than reusing their existing `model`: FileStagingClient rewrites
`model` to a worker-local path, so comparing it would reject valid
requests in exactly the configuration this guards.
AudioEncode/AudioDecode are deliberately left unguarded: the opus codec
backend is loaded from a literal rather than a ModelConfig, so no value
carries the equality guarantee the comparison depends on. The four
bidirectional stream RPCs are out of scope; they bypass reconcile.
Empty means skip on both sides, so an old controller, an old backend, and
the bare request structs in tests/e2e-backends all keep working.
Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
