fix(vllm): apply Options[] engine flags before engine init (#11130)
CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`,
`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the
backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and
did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could
still reach the engine.
Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine
is constructed. Names are normalized the way vLLM's CLI spells them
(`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to
the target field's type (bare flag -> True for booleans), and unknown or
uncoercible flags warn and are skipped instead of failing the load, since
Options[] is a bag shared with backend-level settings. Field types come from
the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not
mistaken for a float.
Precedence is typed proto fields -> `options:` -> `engine_args:`. The
production engine_args defaults seeded in hooks_vllm.go therefore skip any key
the user already set as an option, otherwise the later engine_args pass would
silently override it. Parser lookups now accept both spellings, so
`--reasoning-parser:qwen3` selects LocalAI's parser as well.
The helper's tests are stdlib-only and run in the lint workflow's
dependency-light job via `make test-python-helpers`.
Assisted-by: Claude:claude-opus-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads
The cuda12-chatterbox gallery backend fails to load on a fresh install
because several deps in requirements-cublas12.txt are unpinned:
- torch/torchaudio: unlike requirements-cublas13.txt and
requirements-cpu.txt, this file has no --extra-index-url, so pip pulls
a wheel whose CUDA runtime (cu130) is newer than the host driver
supports ("NVIDIA driver on your system is too old"). Add the cu124
index and pin torch/torchaudio 2.6.0+cu124.
- transformers: resolves to 5.x, which dropped LlamaConfig.rope_theta
that chatterbox-tts 0.3.1's T3 config still reads. Cap to <5.
- setuptools: 81+ dropped pkg_resources, which perth imports under a
bare try/except and silently sets PerthImplicitWatermarker=None,
making ChatterboxTTS.__init__ raise 'NoneType' object is not callable.
Cap to <81 in requirements.txt.
Fixes#11070
Signed-off-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* fix(sglang): keep nvidia-modelopt on a stable release
Every cublas sglang image currently fails to build:
Failed to build `nvidia-modelopt==0.46.0rc0`
Call to `wheel_stub.buildapi.build_wheel` failed
ModuleNotFoundError: No module named 'wheel_stub'
sglang[all] pulls nvidia-modelopt in through its `diffusion` extra with no
version bound of its own, and install.sh adds a GLOBAL --prerelease=allow so
that flash-attn-4, which only ships 4.0.0b* wheels, can resolve. Unbounded plus
prereleases-allowed picks 0.46.0rc0, whose build backend imports wheel_stub
without declaring it in build-system.requires. EXTRA_PIP_INSTALL_FLAGS also
starts with --no-build-isolation, so nothing installs wheel_stub and the build
dies. Latest stable is 0.45.0 and resolves cleanly.
Bounding this one package rather than dropping the global flag, because the
flag is load-bearing for flash-attn-4 and this is the narrower change with the
smaller blast radius. Raise the bound when 0.46.0 final ships.
This is invisible on master because the backend build is path-filtered: sglang
is only rebuilt when sglang changes. It surfaces on any PR touching a shared
build input such as backend/backend.proto, which rebuilds the whole matrix.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(nemo): build the darwin venv on Python 3.12
The darwin nemo image fails to build:
ModuleNotFoundError: No module named 'maturin'
nemo_toolkit pulls in text2num, a Rust extension built with maturin, whose
macOS arm64 wheels start at cp311: 3.0.2 publishes cp311, cp312, cp313 and
cp314 and no cp310. libbackend.sh defaults PYTHON_VERSION to 3.10, so pip finds
no wheel, falls back to the sdist, and dies in the PEP 517 hook because
EXTRA_PIP_INSTALL_FLAGS carries --no-build-isolation and nothing installs the
build backend. Taking the prebuilt wheel avoids the source build entirely, so
the runner needs no Rust toolchain.
Darwin only, deliberately: the Linux profiles resolve a cp310 manylinux wheel
for the same package and have no reason to move. The override is set after
libbackend.sh is sourced and before installRequirements, the same shape
sglang's install.sh already uses for its l4t13 profile.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(nemo): pin the darwin portable-Python patch level too
The 3.12 bump alone traded one failure for another:
curl: (56) The requested URL returned error: 404
make[1]: *** [nemo-asr] Error 56
libbackend builds the portable-Python URL from
cpython-${PYTHON_VERSION}.${PYTHON_PATCH}+${PY_STANDALONE_TAG}, and
PYTHON_PATCH defaults to 18 because the default interpreter is 3.10.18. Setting
only PYTHON_VERSION asked for a 3.12.18 that was never released.
Patch 11, not the 12 that sglang/install.sh pairs with 3.12 for l4t13: at the
20250818 tag python-build-standalone published 3.12.12 for linux aarch64 but
not for aarch64-apple-darwin, where 3.12.11 is the newest. Both URLs were
checked against the release assets rather than assumed to match.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
Some Transformers processors used by MLX-VLM, including Qwen vision models, import both PyTorch and Torchvision. Include them in the Metal backend environment so model loading does not fail with missing-library errors.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
fix(trl): disable inline GRPO reward code by default (RCE)
POST /api/fine-tuning/jobs accepts reward_functions[].code, an inline Python
body, and compile_inline_reward() execs it against a restricted-builtins
allowlist (_SAFE_BUILTINS). That allowlist is not a security boundary:
().__class__.__bases__[0].__subclasses__() reaches os._wrap_close and thus
os.system, giving arbitrary code execution. The fine-tuning endpoint is
unauthenticated by default, so any caller could run code on the host.
Hardening the allowlist is a losing game against CPython introspection, so
inline reward code is now refused unless the operator explicitly opts in with
LOCALAI_TRL_ALLOW_INLINE_REWARD=true on the backend. Builtin reward functions
are unaffected. The gate lives in build_reward_functions(), the single point
all inline specs flow through. Docs updated to stop describing the allowlist as
a sandbox and to document the opt-in.
Fixes#11015
Signed-off-by: Isabel Wu <231155141+wuisabel-gif@users.noreply.github.com>
Co-authored-by: Isabel Wu <231155141+wuisabel-gif@users.noreply.github.com>
The Python gRPC bindings expose message fields as plain attributes
(request.language, request.caption), not Go/Java-style Get*() accessors.
Because request.language is an empty string when unset, the
request.language or request.GetLanguage() or "en"
expression falls through to request.GetLanguage(), which does not exist
on the generated Python message and raises AttributeError, surfaced to
clients as:
rpc error: code = Unknown desc = Exception calling application: GetLanguage
Every /v1/sound-generation request without an explicit language field
failed. Drop the bogus accessor calls (TTS already uses the plain-field
form a few lines below).
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(backends): choose the protoc generator from the protobuf runtime, and regenerate stubs after late installs
The vLLM backends still crash on startup with
VersionError: Detected incompatible Protobuf Gencode/Runtime versions when
loading backend.proto: gencode 7.35.0 runtime 6.33.6
despite #10735 and #10944. Three separate defects kept it alive.
1. runProtogen picked the generator from the installed *grpcio* version.
grpcio-tools' version tracks grpcio, but the gencode its bundled protoc
emits tracks *protobuf*, and the two move independently: grpcio-tools
1.82.1 (the version #10735 pins to, matching grpcio 1.82.1) requires
protobuf>=7.35.1 and stamps gencode 7.35.0. Pinning to grpcio could
therefore never constrain the gencode. Constrain the install to the
protobuf already in the venv instead and let the resolver pick the newest
compatible grpcio-tools. That both selects a generator the runtime accepts
and stops protogen from moving the runtime under the backend's other deps.
This is self-correcting, so the hardcoded GRPCIO_TOOLS_VERSION=1.78.0
escape hatch from #10944 is no longer needed and is removed.
2. The stubs were generated too early. Most branches of vllm/install.sh (and
vllm-omni) install vllm *after* installRequirements, and vllm re-resolves
the protobuf runtime as it lands. Stubs generated against the pre-vllm
runtime can end up newer than the runtime that finally ships, which is the
ROCm failure exactly. Regenerate once the dependency set is final.
3. rm -f of the .py sources left __pycache__ behind. CPython validates a .pyc
against source mtime and size, both of which can be unchanged across a
regeneration (the gencode triple is the same width whether it reads 7.35.0
or 6.33.5), so a stale backend_pb2.pyc could shadow the stub just written.
Also fail the build when the generated stub cannot be imported, so a
gencode/runtime mismatch surfaces at image build time instead of reaching
users as an opaque "grpc service not ready".
Verified by driving the real runProtogen through the ROCm install sequence in
a venv harness: before, gencode 7.35.0 against runtime 6.33.6 (reproducing the
reported error verbatim); after, gencode 6.33.5 against runtime 6.33.6 and the
stub imports cleanly.
Closes#10940Closes#10718
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): regenerate protobuf stubs in the other backends that install after installRequirements
Same defect as the vllm change: installRequirements generates the stubs at the
end of its own run, so any backend that installs further packages afterwards can
have the protobuf runtime moved out from under stubs that were already written.
The gencode stamped into backend_pb2.py then exceeds the runtime that ships and
the backend dies at model load with "grpc service not ready".
fish-speech already had this bug and worked around the symptom: it forces
protobuf>=5.29.0 after installRequirements precisely because "transitive deps
(wandb, tensorboard) may downgrade protobuf to 3.x but our generated
backend_pb2.py requires protobuf 5+". Regenerating after the pin addresses the
cause rather than propping up the runtime to match stale stubs.
Applied to the backends whose post-installRequirements step resolves a
dependency graph and can therefore move protobuf:
fish-speech -e . plus an explicit protobuf install
vibevoice pip install . (with deps)
llama-cpp-quantization gguf / GGUF_PIP_SPEC
trl gguf / GGUF_PIP_SPEC
Deliberately not applied to ace-step and chatterbox (both --no-deps, so the
dependency graph cannot change) or voxcpm (pins setuptools only). gguf does not
depend on protobuf today, but it resolves dependencies, and "this package does
not touch protobuf right now" is exactly the assumption that made the earlier
fix ineffective.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): resolve the protoc generator in a throwaway env so it cannot edit the backend's pinned deps
Installing grpcio-tools into the backend's own venv to generate the stubs also
drags its dependencies in: grpcio-tools 1.82.1 requires grpcio>=1.82.1, so a
backend that pinned grpcio==1.78.1 silently shipped 1.82.1 instead. Caught by
building the llama-cpp-quantization image and reading the versions back out of
the artifact:
before grpcio 1.82.1 (requirements.txt pins grpcio==1.78.1)
after grpcio 1.78.1 grpcio-tools absent from the venv entirely
Resolve the generator in a throwaway environment instead, still constrained to
the protobuf the backend ships so the gencode stays compatible. The backend's
dependency set is then exactly what its requirements files declared. protoc's
output is plain Python and carries no dependency on the interpreter that
produced it, so generating from a different env is safe; the import check still
runs under the backend's python, since that is the interpreter that has to load
the stubs at model load.
Verified on the rebuilt image: gencode 7.35.0, runtime protobuf 7.35.1, grpcio
back at its pinned 1.78.1, and the shipped stub imports cleanly against 7.35.1.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): bound the protoc generator by BOTH the installed grpcio and protobuf
The generated stubs impose two independent constraints, and every fix so far,
including the previous commit on this branch, satisfied one while violating the
other:
backend_pb2.py needs protobuf runtime >= gencode
backend_pb2_grpc.py needs installed grpcio >= grpcio-tools
Resolving the generator against protobuf alone picked grpcio-tools 1.82.1 for a
backend holding grpcio at 1.78.1, so the gencode was fine but the gRPC stub was
not:
RuntimeError: The grpc package installed is at version 1.78.1, but the
generated code in backend_pb2_grpc.py depends on grpcio>=1.82.1.
That is also why installing grpcio-tools into the backend venv appeared to work
earlier: it dragged grpcio up to match, which was load-bearing rather than the
regression it looked like. Isolating the generator removed the accidental fix
and exposed the missing constraint.
Bound grpcio-tools from both sides instead and let the resolver find the newest
version satisfying both. The protobuf ceiling makes it back off to an older
generator when the runtime trails, bounding the gencode; the grpcio ceiling
keeps the _grpc stub loadable. Resolved against the four real runtime pairs
observed in built images:
grpcio 1.78.1 / protobuf 7.35.1 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.78.0 / protobuf 6.33.6 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.82.1 / protobuf 6.33.6 -> grpcio-tools 1.81.1, gencode 6.33.5 OK
grpcio 1.82.1 / protobuf 7.35.1 -> grpcio-tools 1.82.1, gencode 7.35.0 OK
Also restore the import check to cover backend_pb2_grpc as well as backend_pb2.
Narrowing it to backend_pb2 is why the image build passed while CI failed: the
guard could not see the constraint that was actually broken.
Verified by running the CI sequence locally for llama-cpp-quantization, the
backend whose test failed:
make -C backend/python/llama-cpp-quantization -> exit 0
make -C backend/python/llama-cpp-quantization test -> exit 0, OK
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(backend/python): don't await sync servicer behaviors in AsyncModelIdentityInterceptor
The model-identity interceptor (added for #10952) is installed on every Python
backend's gRPC server. Its grpc.aio variant invokes the wrapped servicer
behavior itself and awaits the result unconditionally:
result = await original(request, context) # LoadModel
return await original_unary(request, context) # guarded RPCs
async for response in original_stream(request, context): # streaming
But a backend's servicer methods may be plain sync functions. The transformers
backend, for one, defines `def LoadModel` and `def Embedding` (not `async def`).
grpc.aio's own dispatch adapts both shapes, but this interceptor calls the
behavior directly and bypasses that. For a sync method `original(...)` returns a
message object, not a coroutine, so the `await` raises:
TypeError: object Result can't be used in 'await' expression
The model loads, then the LoadModel RPC dies on return; the guarded sync
Embedding fails the same way. It happens on every platform, not just one backend
build. CI never caught it because AsyncModelIdentityInterceptor had no
behavioral test -- only an "is it installed" assertion.
Fix: await only when the behavior actually returned an awaitable
(inspect.isawaitable), mirroring grpc.aio's own sync/async adaptation. The
streaming guard iterates a sync generator with `for` and an async one with
`async for`.
Adds async-path coverage to model_identity_test.py exercising both sync and
async LoadModel / guarded-unary / streaming behaviors. The sync cases fail on
the current code with the TypeError above and pass with this fix.
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
* fix(backend/python): dispatch sync servicer behaviors off the event loop
Addresses review feedback: awaiting only awaitable results removed the
TypeError, but still ran a sync LoadModel/Embedding -- and stepped a sync stream
via next() -- on the asyncio event-loop thread, so a slow load/inference/stream
could freeze all aio RPC handling.
Route sync behavior through run_in_executor (a worker thread) while awaiting
native async behavior directly. A callable wrapper that returns an awaitable is
run in the thread and its awaitable awaited back on the loop. Sync streaming
pulls each item via the executor with a done sentinel, so StopIteration cannot
escape through a Future.
Adds regression tests that record the handler thread id and assert it differs
from the event-loop thread, for LoadModel, a guarded unary RPC and a sync stream.
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
---------
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
#10970 gave the four PredictOptions RPCs a model-identity check so a
backend reached through a stale distributed route rejects the request
instead of answering from whatever model it holds (#10952). Every other
modality shares that exposure: the route is cached by host:port, a worker
can recycle a stopped backend's port for another model's backend, and a
liveness-only probe cannot tell a stale row from a valid one.
Extends the same mechanism to the 21 remaining request messages that reach
a backend through the router, using the pattern #10970 established rather
than a parallel one:
- proto: ModelIdentity on each modality request message.
- controller: populated from ModelConfig.Model at the call site that also
builds ModelOptions, so load-time and request-time values are equal by
construction.
- backends: one generic guard in pkg/grpc/server.go (27 Go backends), the
method set in backend/python/common (36 Python backends), llama-cpp
(AudioTranscription/Stream, Rerank, Score) and privacy-filter
(TokenClassify).
- reconcile already drops the stale row on IsModelMismatch; no change.
TTSRequest and SoundGenerationRequest get a SEPARATE ModelIdentity field
rather than reusing their existing `model`: FileStagingClient rewrites
`model` to a worker-local path, so comparing it would reject valid
requests in exactly the configuration this guards.
AudioEncode/AudioDecode are deliberately left unguarded: the opus codec
backend is loaded from a literal rather than a ModelConfig, so no value
carries the equality guarantee the comparison depends on. The four
bidirectional stream RPCs are out of scope; they bypass reconcile.
Empty means skip on both sides, so an old controller, an old backend, and
the bare request structs in tests/e2e-backends all keep working.
Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(distributed): reject wrong-model requests at the backend (#10952)
In distributed mode the controller caches a NodeModel row naming a backend's
host:port. A worker can recycle a stopped backend's gRPC port for a different
model's backend, and probeHealth verifies liveness rather than identity, so the
probe succeeds against whatever now occupies the port and the request is
dispatched to the wrong backend. The caller gets a silent wrong-model answer.
Nothing in the request could catch this: PredictOptions had no model field, so
model identity crossed the wire only in ModelOptions.Model at LoadModel time,
and the cached-hit path issues no LoadModel. Every backend's "model not loaded"
guard checks a nil handle, which a process holding a different model passes, so
the stale row was never dropped either.
Add PredictOptions.ModelIdentity and enforce it at the point of use:
- The controller populates it in gRPCPredictOpts from ModelConfig.Model, the
same expression ModelOptions feeds to model.WithModel and therefore the
same value the backend received as ModelOptions.Model. Both are read from
one config value in one function, so they are equal by construction and the
comparison cannot false-reject.
- Backends compare it against what they loaded and return NOT_FOUND with a
fixed sentinel. Enforced in pkg/grpc/server.go (27 Go backends), an
interceptor in backend/python/common (all 36 Python backends, no
per-backend change), and the llama-cpp / ik-llama-cpp / ds4 C++ servers.
That is every backend with real exposure: kokoros answers all four RPCs
with unimplemented and privacy-filter implements none of them.
- The router's reconcile drops the stale replica row on a mismatch, so the
next request reloads somewhere correct.
Empty means "skip the check" on both sides: a controller that predates the
field sends nothing, a backend loaded by such a controller has nothing to
compare, and the C++ server synthesizes PredictOptions internally for ASR. That
keeps upgrades working in both directions.
Scoped to the four PredictOptions RPCs. TTSRequest.model and
SoundGenerationRequest.model are deliberately NOT validated: FileStagingClient
already rewrites them to worker-local absolute paths, so in distributed mode
they already differ from the load-time value and comparing them would reject
valid requests.
IsModelMismatch requires both the NOT_FOUND code and the sentinel, unlike the
neighbouring helpers which accept either. insightface's Embedding returns
NOT_FOUND "no face detected" on a PredictOptions RPC, and a code-only check
would drop a healthy replica row on every faceless image.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* chore(deps): bump the pip group across 6 directories with 1 update
Bumps the pip group with 1 update in the /backend/python/ace-step directory: torch.
Bumps the pip group with 1 update in the /backend/python/llama-cpp-quantization directory: torch.
Bumps the pip group with 1 update in the /backend/python/longcat-video directory: torch.
Bumps the pip group with 1 update in the /backend/python/sglang directory: torch.
Bumps the pip group with 1 update in the /backend/python/trl directory: torch.
Bumps the pip group with 1 update in the /backend/python/vllm-omni directory: torch.
Updates `torch` from 2.10.0+rocm7.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.12.1 to 2.12.1+cu130
Updates `torch` from 2.9.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.7.0 to 2.12.1+cu130
---
updated-dependencies:
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(deps): preserve platform-specific torch requirements
Keep the 2.12.1 CPU bump only where uv resolves it cleanly, and restore ROCm, CUDA, MPS, and unrelated transformers constraints that Dependabot rewrote to incompatible wheel variants.
Assisted-by: Codex:gpt-5 [uv]
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Every vibevoice requirements file pulled diffusers straight from
git+https://github.com/huggingface/diffusers. That branch now reports
itself as 0.40.0.dev0 and requires huggingface-hub>=1.23.0,<2.0, while
transformers>=4.51.3,<5.0.0 (which upstream VibeVoice mandates) still
caps huggingface-hub at <1.0. Because a git URL offers the resolver
exactly one candidate version, uv has nothing to backtrack to and the
install fails outright:
Because only diffusers==0.40.0.dev0 is available and diffusers==0.40.0.dev0
depends on huggingface-hub>=1.23.0,<2.0 [...] we can conclude that your
requirements are unsatisfiable.
This broke the vibevoice build on every variant - cpu (amd64/arm64),
cuda 12/13, l4t 12/13, intel and rocm, plus the darwin metal job - and
has been failing the weekly full-matrix rebuild for three weeks. It is
not caught by master pushes because backend builds are path-filtered
there, so it only surfaces on the Sunday cron and on release tags.
Use the PyPI package instead. That is what upstream VibeVoice declares
in its own pyproject.toml, and what every other LocalAI backend already
does - vibevoice was the only one tracking the git branch. With a real
release series available the resolver settles on diffusers 0.39.0 with
huggingface-hub 0.36.2 and transformers 4.57.6, and it can keep
backtracking on its own if upstream shifts again.
Verified with uv pip compile against cpu, cublas12, cublas13, hipblas,
intel, mps and l4t13: all resolve to that same coherent set. l4t12 only
resolves on aarch64, since its Jetson index ships no x86_64 torch wheel.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Bash] [uv]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
request.language usually carries an ISO 639-1 code (e.g. "de"), which
OpenAI-compatible clients such as Home Assistant / wyoming_openai send,
but qwen_asr.validate_language() only accepts full English names
("German") and raises ValueError otherwise. Normalize the requested
language: accept full names case-insensitively, translate ISO codes
(with optional region suffix like "de-DE") to the expected name, and
pass anything unrecognised through so qwen_asr still reports it clearly.
Fixes#10958
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(model-artifacts): materialize longcat-video checkpoints on the controller
longcat-video loads a checkpoint directory: its backend.py takes
request.ModelFile when os.path.isdir(request.ModelFile) and otherwise
falls back to snapshot_download. That places it in the same class as
transformers/vllm/diffusers/sglang, but the allow-list added in #10910
did not enumerate it, so PrimaryArtifactSpec returned no managed
artifact for a bare HuggingFace repo id.
The consequence in distributed mode: nothing was acquired on the
controller, ModelFileName fell through to the raw repo id, and staging
skipped the resulting phantom /models/<owner>/<repo> path. The worker
received a blank ModelFile, fell back to request.Model, and downloaded
~83GB from HuggingFace inside the remote LoadModel deadline - so the
load could only ever fail with DeadlineExceeded while an abandoned
backend process kept downloading.
Note this materializes the full repository. The backend restricts its
own snapshot_download with allow_patterns, and the avatar repo ships
both base_model/ and base_model_int8/ where only one is ever loaded;
inferred specs have no way to carry patterns today. Tracked separately.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): warn when staging skips a non-existent model path
stageModelFiles logs "Staging model files for remote node" up front, then
silently drops any path field that does not exist on the controller. The
skip itself is legitimate and must stay: a backend outside
managedArtifactBackends that takes a bare HuggingFace repo id gets an
optimistically constructed path (ModelFileName falls through to the raw
model reference) that was never materialized, and sources its own weights
on the worker. Erroring would break those configs.
But at debug level the operator is left with a reassuring staging line and
no trace of the skip, so a genuine controller-side acquisition gap is
indistinguishable from a healthy pass-through - it surfaces much later as
a remote LoadModel timeout, on a worker that is quietly downloading tens
of gigabytes. Raise the skip to warn and name the field, path, node and
tracking key. Behavior is unchanged.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): allow a config to declare companion artifacts
A composed pipeline needs more than one HuggingFace snapshot.
LongCat-Video-Avatar-1.5 loads its own transformer but takes the
tokenizer, text encoder and VAE from the separate LongCat-Video base
repo, so a single-artifact config cannot express it and the backend is
left to fetch the second repo itself at load time.
Widen the artifact model to target: model plus any number of named
target: companion entries. Normalize accepts the new target and
constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that
name is the option key the backend later receives; a companion may not
claim primary_file, which only means anything for a load target.
ModelConfig.Validate requires exactly one primary and requires it first,
since Artifacts[0] is what ModelFileName, size estimation and staging all
resolve from.
Both acquisition paths now loop instead of touching index 0 alone:
preloadOne for an already-installed config, bindPrimaryArtifact for a
gallery install. Failure policy differs by provenance. An inferred
primary keeps its warn-and-fall-back, because the legacy download path
still exists for it. Companions are explicit by construction, so they are
all-or-nothing: a config naming one is asserting the backend needs it,
and failing at the acquisition boundary is far more legible than a
missing-weights error surfacing later inside the backend.
The cache key is deliberately unchanged. It hashes source identity only,
never name or target, so every already-installed managed model still hits
its existing snapshot instead of silently re-downloading. Two specs pin
that: one proving a companion and a primary with identical sources agree
on the key, and one pinning the digest of a known primary outright.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): hand resolved companion snapshots to the backend
A materialized companion is useless until the backend can find it, and
its location is a content-addressed cache key that does not exist until
the artifact resolves. A static gallery override cannot carry that, and
persisting it into the config YAML would rot the moment a re-resolve
produced a new key.
Synthesize it instead at load time: each resolved companion becomes
"<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the
key:value convention backends already parse for options like
attention_backend. The value stays relative to the models directory so a
remote worker can resolve it under its own ModelPath once staging has
rewritten the model root. An option the author set explicitly always
wins, so pinning a companion to a local checkout still beats the managed
snapshot.
longcat-video resolves base_model through ModelPath, the same convention
qwen-tts, voxcpm, outetts and ace-step already use for companion assets.
Its sibling-directory heuristic is deleted: it looked for a LongCat-Video
directory next to the model, which cannot exist under the content
addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead
code the moment the model became managed.
The gallery entry declares both repositories and restricts each with
allow_patterns. The avatar repo ships base_model/ and base_model_int8/
and only ever loads one, so fetching the whole repo would roughly double
the download. The patterns match the entry's own options (use_distill
true, use_int8 default false); enabling use_int8 here also requires
adding base_model_int8/**, which is called out in the entry.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage managed artifact trees from the models root
Staging anchored the worker's models directory on the primary snapshot
whenever a model was managed, so a companion snapshot could not reach the
worker at all.
frontendModelsDir was derived by stripping the Model relative path off
the end of ModelFile. For a managed artifact nothing matches: ModelFile
is .artifacts/huggingface/<key>/snapshot while Model stays a bare
HuggingFace repo id, so the strip was a no-op and the "models directory"
came out as the snapshot itself. Two consequences, both silent. Staging
keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two
snapshots of one model were indistinguishable on the worker. And a
companion, which lives in a sibling snapshot directory outside the
primary, fell outside that directory entirely: StagingKeyMapper.Key
collapsed its files to bare basenames and resolveOptionPath could not
resolve the relative option at all, so it was skipped without a word.
Derive the models root from the artifact tree instead when the path runs
through it, and compute the worker's ModelPath from the file's path
relative to that root rather than from the Model field. The legacy layout
is unaffected: where Model really is the relative path, the new
derivation reduces to the old one, which a regression spec pins.
This deliberately changes an invariant that router_dirstage_test.go
pinned: for a managed primary, ModelFile and ModelPath were both the
snapshot directory, and staging keys were relative to it. Now ModelFile
is the snapshot, ModelPath is the models root above it, and keys keep the
full relative path. That spec is updated rather than accommodated, with
the reasoning recorded inline, because the old invariant is exactly what
made a sibling companion unreachable.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Pin vLLM protogen to grpcio-tools 1.78.0 so its generated code remains importable by protobuf 6.33.x, and remove stale generated artifacts before regeneration.
Closes#10940
Assisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The sglang Python backend inherits the default Status RPC from
backend_pb2_grpc.BackendServicer, which raises NotImplementedError.
LocalAI's backend-monitor polls /backend.Backend/Status periodically on
every registered backend; when the call fails, /backend/monitor returns
HTTP 500 and downstream inference requests to the sglang backend are
blocked even though the model is loaded and answering directly via the
gRPC endpoint.
Add a minimal Status shim that mirrors the existing Health method and
returns StatusResponse{state=READY} unconditionally. This unblocks the
monitor path; a state-aware follow-up (UNINITIALIZED during load, BUSY
under active inference) is left for a subsequent change.
Reproduced on DGX Spark (GB10, arm64-l4t-cuda-13 image) with the sglang
v0.5.15 backend and Qwen3-Coder-Next-NVFP4-GB10; verified locally that
patching the shim in place immediately restores /backend/monitor and
inference across the sglang slot.
PyTorch 2.13 XPU pulls oneAPI 2026 libraries that conflict with the oneAPI 2025.3 backend image. Pin torch and torchaudio to the matching 2.11 XPU pair so the build resolves a coherent 2025.3 runtime.
Assisted-by: Codex:GPT-5 [uv]
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
SetDefaults injected the llama.cpp server options cache_reuse
(ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied
per selected node by the distributed router) onto every model config
regardless of backend. Every other backend ignores options it does not
understand, so this was harmless until longcat-video, which strictly
validates its options and fails LoadModel with
"unknown model option(s): cache_reuse, parallel".
Gate both injections behind a new UsesLlamaCppServingOptions allow-list
(llama-cpp plus the empty/auto-detect case that resolves to llama.cpp
from a GGUF file, mirroring how llamaCppDefaults is registered). This
follows the existing UsesLlamaSamplerDefaults precedent for llama-only
defaults. The typed NBatch field is deliberately left alone: it is a
proto field every backend simply ignores, which is why batch never
triggered the error.
Also harden the longcat-video backend to warn-and-ignore unknown model
options and request params through a testable select_known_options
helper, matching the other LocalAI Python backends, so a future
server-injected option cannot break loading again.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): add voice library workflow
Give administrators a production-ready flow to record or upload consented reference audio, manage reusable profiles, inspect API usage, discover compatible models, and hand a saved voice directly to text-to-speech.
Assisted-by: Codex:gpt-5
* feat(voice): add managed voice cloning profiles
Make reusable reference voices manageable through the admin API instead of requiring model-directory and YAML edits. Discover compatible installed and gallery models from server-side backend capabilities, retain explicit model configuration controls, and stage saved references for supported backends.
Expose profile management through REST and MCP, document backend-specific behavior, and cover the workflow from profile creation through real Qwen3-TTS synthesis. Harden the agent-job HTTP test against completion racing cancellation.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backends): add LongCat video and avatar generation
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web]
* refactor(config): declare model I/O modalities
Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook.
Assisted-by: Codex:GPT-5 [web]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
