The Python gRPC bindings expose message fields as plain attributes
(request.language, request.caption), not Go/Java-style Get*() accessors.
Because request.language is an empty string when unset, the
request.language or request.GetLanguage() or "en"
expression falls through to request.GetLanguage(), which does not exist
on the generated Python message and raises AttributeError, surfaced to
clients as:
rpc error: code = Unknown desc = Exception calling application: GetLanguage
Every /v1/sound-generation request without an explicit language field
failed. Drop the bogus accessor calls (TTS already uses the plain-field
form a few lines below).
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(backends): choose the protoc generator from the protobuf runtime, and regenerate stubs after late installs
The vLLM backends still crash on startup with
VersionError: Detected incompatible Protobuf Gencode/Runtime versions when
loading backend.proto: gencode 7.35.0 runtime 6.33.6
despite #10735 and #10944. Three separate defects kept it alive.
1. runProtogen picked the generator from the installed *grpcio* version.
grpcio-tools' version tracks grpcio, but the gencode its bundled protoc
emits tracks *protobuf*, and the two move independently: grpcio-tools
1.82.1 (the version #10735 pins to, matching grpcio 1.82.1) requires
protobuf>=7.35.1 and stamps gencode 7.35.0. Pinning to grpcio could
therefore never constrain the gencode. Constrain the install to the
protobuf already in the venv instead and let the resolver pick the newest
compatible grpcio-tools. That both selects a generator the runtime accepts
and stops protogen from moving the runtime under the backend's other deps.
This is self-correcting, so the hardcoded GRPCIO_TOOLS_VERSION=1.78.0
escape hatch from #10944 is no longer needed and is removed.
2. The stubs were generated too early. Most branches of vllm/install.sh (and
vllm-omni) install vllm *after* installRequirements, and vllm re-resolves
the protobuf runtime as it lands. Stubs generated against the pre-vllm
runtime can end up newer than the runtime that finally ships, which is the
ROCm failure exactly. Regenerate once the dependency set is final.
3. rm -f of the .py sources left __pycache__ behind. CPython validates a .pyc
against source mtime and size, both of which can be unchanged across a
regeneration (the gencode triple is the same width whether it reads 7.35.0
or 6.33.5), so a stale backend_pb2.pyc could shadow the stub just written.
Also fail the build when the generated stub cannot be imported, so a
gencode/runtime mismatch surfaces at image build time instead of reaching
users as an opaque "grpc service not ready".
Verified by driving the real runProtogen through the ROCm install sequence in
a venv harness: before, gencode 7.35.0 against runtime 6.33.6 (reproducing the
reported error verbatim); after, gencode 6.33.5 against runtime 6.33.6 and the
stub imports cleanly.
Closes#10940Closes#10718
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): regenerate protobuf stubs in the other backends that install after installRequirements
Same defect as the vllm change: installRequirements generates the stubs at the
end of its own run, so any backend that installs further packages afterwards can
have the protobuf runtime moved out from under stubs that were already written.
The gencode stamped into backend_pb2.py then exceeds the runtime that ships and
the backend dies at model load with "grpc service not ready".
fish-speech already had this bug and worked around the symptom: it forces
protobuf>=5.29.0 after installRequirements precisely because "transitive deps
(wandb, tensorboard) may downgrade protobuf to 3.x but our generated
backend_pb2.py requires protobuf 5+". Regenerating after the pin addresses the
cause rather than propping up the runtime to match stale stubs.
Applied to the backends whose post-installRequirements step resolves a
dependency graph and can therefore move protobuf:
fish-speech -e . plus an explicit protobuf install
vibevoice pip install . (with deps)
llama-cpp-quantization gguf / GGUF_PIP_SPEC
trl gguf / GGUF_PIP_SPEC
Deliberately not applied to ace-step and chatterbox (both --no-deps, so the
dependency graph cannot change) or voxcpm (pins setuptools only). gguf does not
depend on protobuf today, but it resolves dependencies, and "this package does
not touch protobuf right now" is exactly the assumption that made the earlier
fix ineffective.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): resolve the protoc generator in a throwaway env so it cannot edit the backend's pinned deps
Installing grpcio-tools into the backend's own venv to generate the stubs also
drags its dependencies in: grpcio-tools 1.82.1 requires grpcio>=1.82.1, so a
backend that pinned grpcio==1.78.1 silently shipped 1.82.1 instead. Caught by
building the llama-cpp-quantization image and reading the versions back out of
the artifact:
before grpcio 1.82.1 (requirements.txt pins grpcio==1.78.1)
after grpcio 1.78.1 grpcio-tools absent from the venv entirely
Resolve the generator in a throwaway environment instead, still constrained to
the protobuf the backend ships so the gencode stays compatible. The backend's
dependency set is then exactly what its requirements files declared. protoc's
output is plain Python and carries no dependency on the interpreter that
produced it, so generating from a different env is safe; the import check still
runs under the backend's python, since that is the interpreter that has to load
the stubs at model load.
Verified on the rebuilt image: gencode 7.35.0, runtime protobuf 7.35.1, grpcio
back at its pinned 1.78.1, and the shipped stub imports cleanly against 7.35.1.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): bound the protoc generator by BOTH the installed grpcio and protobuf
The generated stubs impose two independent constraints, and every fix so far,
including the previous commit on this branch, satisfied one while violating the
other:
backend_pb2.py needs protobuf runtime >= gencode
backend_pb2_grpc.py needs installed grpcio >= grpcio-tools
Resolving the generator against protobuf alone picked grpcio-tools 1.82.1 for a
backend holding grpcio at 1.78.1, so the gencode was fine but the gRPC stub was
not:
RuntimeError: The grpc package installed is at version 1.78.1, but the
generated code in backend_pb2_grpc.py depends on grpcio>=1.82.1.
That is also why installing grpcio-tools into the backend venv appeared to work
earlier: it dragged grpcio up to match, which was load-bearing rather than the
regression it looked like. Isolating the generator removed the accidental fix
and exposed the missing constraint.
Bound grpcio-tools from both sides instead and let the resolver find the newest
version satisfying both. The protobuf ceiling makes it back off to an older
generator when the runtime trails, bounding the gencode; the grpcio ceiling
keeps the _grpc stub loadable. Resolved against the four real runtime pairs
observed in built images:
grpcio 1.78.1 / protobuf 7.35.1 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.78.0 / protobuf 6.33.6 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.82.1 / protobuf 6.33.6 -> grpcio-tools 1.81.1, gencode 6.33.5 OK
grpcio 1.82.1 / protobuf 7.35.1 -> grpcio-tools 1.82.1, gencode 7.35.0 OK
Also restore the import check to cover backend_pb2_grpc as well as backend_pb2.
Narrowing it to backend_pb2 is why the image build passed while CI failed: the
guard could not see the constraint that was actually broken.
Verified by running the CI sequence locally for llama-cpp-quantization, the
backend whose test failed:
make -C backend/python/llama-cpp-quantization -> exit 0
make -C backend/python/llama-cpp-quantization test -> exit 0, OK
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(backend/python): don't await sync servicer behaviors in AsyncModelIdentityInterceptor
The model-identity interceptor (added for #10952) is installed on every Python
backend's gRPC server. Its grpc.aio variant invokes the wrapped servicer
behavior itself and awaits the result unconditionally:
result = await original(request, context) # LoadModel
return await original_unary(request, context) # guarded RPCs
async for response in original_stream(request, context): # streaming
But a backend's servicer methods may be plain sync functions. The transformers
backend, for one, defines `def LoadModel` and `def Embedding` (not `async def`).
grpc.aio's own dispatch adapts both shapes, but this interceptor calls the
behavior directly and bypasses that. For a sync method `original(...)` returns a
message object, not a coroutine, so the `await` raises:
TypeError: object Result can't be used in 'await' expression
The model loads, then the LoadModel RPC dies on return; the guarded sync
Embedding fails the same way. It happens on every platform, not just one backend
build. CI never caught it because AsyncModelIdentityInterceptor had no
behavioral test -- only an "is it installed" assertion.
Fix: await only when the behavior actually returned an awaitable
(inspect.isawaitable), mirroring grpc.aio's own sync/async adaptation. The
streaming guard iterates a sync generator with `for` and an async one with
`async for`.
Adds async-path coverage to model_identity_test.py exercising both sync and
async LoadModel / guarded-unary / streaming behaviors. The sync cases fail on
the current code with the TypeError above and pass with this fix.
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
* fix(backend/python): dispatch sync servicer behaviors off the event loop
Addresses review feedback: awaiting only awaitable results removed the
TypeError, but still ran a sync LoadModel/Embedding -- and stepped a sync stream
via next() -- on the asyncio event-loop thread, so a slow load/inference/stream
could freeze all aio RPC handling.
Route sync behavior through run_in_executor (a worker thread) while awaiting
native async behavior directly. A callable wrapper that returns an awaitable is
run in the thread and its awaitable awaited back on the loop. Sync streaming
pulls each item via the executor with a done sentinel, so StopIteration cannot
escape through a Future.
Adds regression tests that record the handler thread id and assert it differs
from the event-loop thread, for LoadModel, a guarded unary RPC and a sync stream.
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
---------
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
#10970 gave the four PredictOptions RPCs a model-identity check so a
backend reached through a stale distributed route rejects the request
instead of answering from whatever model it holds (#10952). Every other
modality shares that exposure: the route is cached by host:port, a worker
can recycle a stopped backend's port for another model's backend, and a
liveness-only probe cannot tell a stale row from a valid one.
Extends the same mechanism to the 21 remaining request messages that reach
a backend through the router, using the pattern #10970 established rather
than a parallel one:
- proto: ModelIdentity on each modality request message.
- controller: populated from ModelConfig.Model at the call site that also
builds ModelOptions, so load-time and request-time values are equal by
construction.
- backends: one generic guard in pkg/grpc/server.go (27 Go backends), the
method set in backend/python/common (36 Python backends), llama-cpp
(AudioTranscription/Stream, Rerank, Score) and privacy-filter
(TokenClassify).
- reconcile already drops the stale row on IsModelMismatch; no change.
TTSRequest and SoundGenerationRequest get a SEPARATE ModelIdentity field
rather than reusing their existing `model`: FileStagingClient rewrites
`model` to a worker-local path, so comparing it would reject valid
requests in exactly the configuration this guards.
AudioEncode/AudioDecode are deliberately left unguarded: the opus codec
backend is loaded from a literal rather than a ModelConfig, so no value
carries the equality guarantee the comparison depends on. The four
bidirectional stream RPCs are out of scope; they bypass reconcile.
Empty means skip on both sides, so an old controller, an old backend, and
the bare request structs in tests/e2e-backends all keep working.
Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(distributed): reject wrong-model requests at the backend (#10952)
In distributed mode the controller caches a NodeModel row naming a backend's
host:port. A worker can recycle a stopped backend's gRPC port for a different
model's backend, and probeHealth verifies liveness rather than identity, so the
probe succeeds against whatever now occupies the port and the request is
dispatched to the wrong backend. The caller gets a silent wrong-model answer.
Nothing in the request could catch this: PredictOptions had no model field, so
model identity crossed the wire only in ModelOptions.Model at LoadModel time,
and the cached-hit path issues no LoadModel. Every backend's "model not loaded"
guard checks a nil handle, which a process holding a different model passes, so
the stale row was never dropped either.
Add PredictOptions.ModelIdentity and enforce it at the point of use:
- The controller populates it in gRPCPredictOpts from ModelConfig.Model, the
same expression ModelOptions feeds to model.WithModel and therefore the
same value the backend received as ModelOptions.Model. Both are read from
one config value in one function, so they are equal by construction and the
comparison cannot false-reject.
- Backends compare it against what they loaded and return NOT_FOUND with a
fixed sentinel. Enforced in pkg/grpc/server.go (27 Go backends), an
interceptor in backend/python/common (all 36 Python backends, no
per-backend change), and the llama-cpp / ik-llama-cpp / ds4 C++ servers.
That is every backend with real exposure: kokoros answers all four RPCs
with unimplemented and privacy-filter implements none of them.
- The router's reconcile drops the stale replica row on a mismatch, so the
next request reloads somewhere correct.
Empty means "skip the check" on both sides: a controller that predates the
field sends nothing, a backend loaded by such a controller has nothing to
compare, and the C++ server synthesizes PredictOptions internally for ASR. That
keeps upgrades working in both directions.
Scoped to the four PredictOptions RPCs. TTSRequest.model and
SoundGenerationRequest.model are deliberately NOT validated: FileStagingClient
already rewrites them to worker-local absolute paths, so in distributed mode
they already differ from the load-time value and comparing them would reject
valid requests.
IsModelMismatch requires both the NOT_FOUND code and the sentinel, unlike the
neighbouring helpers which accept either. insightface's Embedding returns
NOT_FOUND "no face detected" on a PredictOptions RPC, and a code-only check
would drop a healthy replica row on every faceless image.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* chore(deps): bump the pip group across 6 directories with 1 update
Bumps the pip group with 1 update in the /backend/python/ace-step directory: torch.
Bumps the pip group with 1 update in the /backend/python/llama-cpp-quantization directory: torch.
Bumps the pip group with 1 update in the /backend/python/longcat-video directory: torch.
Bumps the pip group with 1 update in the /backend/python/sglang directory: torch.
Bumps the pip group with 1 update in the /backend/python/trl directory: torch.
Bumps the pip group with 1 update in the /backend/python/vllm-omni directory: torch.
Updates `torch` from 2.10.0+rocm7.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.12.1 to 2.12.1+cu130
Updates `torch` from 2.9.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.7.0 to 2.12.1+cu130
---
updated-dependencies:
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(deps): preserve platform-specific torch requirements
Keep the 2.12.1 CPU bump only where uv resolves it cleanly, and restore ROCm, CUDA, MPS, and unrelated transformers constraints that Dependabot rewrote to incompatible wheel variants.
Assisted-by: Codex:gpt-5 [uv]
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Every vibevoice requirements file pulled diffusers straight from
git+https://github.com/huggingface/diffusers. That branch now reports
itself as 0.40.0.dev0 and requires huggingface-hub>=1.23.0,<2.0, while
transformers>=4.51.3,<5.0.0 (which upstream VibeVoice mandates) still
caps huggingface-hub at <1.0. Because a git URL offers the resolver
exactly one candidate version, uv has nothing to backtrack to and the
install fails outright:
Because only diffusers==0.40.0.dev0 is available and diffusers==0.40.0.dev0
depends on huggingface-hub>=1.23.0,<2.0 [...] we can conclude that your
requirements are unsatisfiable.
This broke the vibevoice build on every variant - cpu (amd64/arm64),
cuda 12/13, l4t 12/13, intel and rocm, plus the darwin metal job - and
has been failing the weekly full-matrix rebuild for three weeks. It is
not caught by master pushes because backend builds are path-filtered
there, so it only surfaces on the Sunday cron and on release tags.
Use the PyPI package instead. That is what upstream VibeVoice declares
in its own pyproject.toml, and what every other LocalAI backend already
does - vibevoice was the only one tracking the git branch. With a real
release series available the resolver settles on diffusers 0.39.0 with
huggingface-hub 0.36.2 and transformers 4.57.6, and it can keep
backtracking on its own if upstream shifts again.
Verified with uv pip compile against cpu, cublas12, cublas13, hipblas,
intel, mps and l4t13: all resolve to that same coherent set. l4t12 only
resolves on aarch64, since its Jetson index ships no x86_64 torch wheel.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Bash] [uv]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
request.language usually carries an ISO 639-1 code (e.g. "de"), which
OpenAI-compatible clients such as Home Assistant / wyoming_openai send,
but qwen_asr.validate_language() only accepts full English names
("German") and raises ValueError otherwise. Normalize the requested
language: accept full names case-insensitively, translate ISO codes
(with optional region suffix like "de-DE") to the expected name, and
pass anything unrecognised through so qwen_asr still reports it clearly.
Fixes#10958
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(model-artifacts): materialize longcat-video checkpoints on the controller
longcat-video loads a checkpoint directory: its backend.py takes
request.ModelFile when os.path.isdir(request.ModelFile) and otherwise
falls back to snapshot_download. That places it in the same class as
transformers/vllm/diffusers/sglang, but the allow-list added in #10910
did not enumerate it, so PrimaryArtifactSpec returned no managed
artifact for a bare HuggingFace repo id.
The consequence in distributed mode: nothing was acquired on the
controller, ModelFileName fell through to the raw repo id, and staging
skipped the resulting phantom /models/<owner>/<repo> path. The worker
received a blank ModelFile, fell back to request.Model, and downloaded
~83GB from HuggingFace inside the remote LoadModel deadline - so the
load could only ever fail with DeadlineExceeded while an abandoned
backend process kept downloading.
Note this materializes the full repository. The backend restricts its
own snapshot_download with allow_patterns, and the avatar repo ships
both base_model/ and base_model_int8/ where only one is ever loaded;
inferred specs have no way to carry patterns today. Tracked separately.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): warn when staging skips a non-existent model path
stageModelFiles logs "Staging model files for remote node" up front, then
silently drops any path field that does not exist on the controller. The
skip itself is legitimate and must stay: a backend outside
managedArtifactBackends that takes a bare HuggingFace repo id gets an
optimistically constructed path (ModelFileName falls through to the raw
model reference) that was never materialized, and sources its own weights
on the worker. Erroring would break those configs.
But at debug level the operator is left with a reassuring staging line and
no trace of the skip, so a genuine controller-side acquisition gap is
indistinguishable from a healthy pass-through - it surfaces much later as
a remote LoadModel timeout, on a worker that is quietly downloading tens
of gigabytes. Raise the skip to warn and name the field, path, node and
tracking key. Behavior is unchanged.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): allow a config to declare companion artifacts
A composed pipeline needs more than one HuggingFace snapshot.
LongCat-Video-Avatar-1.5 loads its own transformer but takes the
tokenizer, text encoder and VAE from the separate LongCat-Video base
repo, so a single-artifact config cannot express it and the backend is
left to fetch the second repo itself at load time.
Widen the artifact model to target: model plus any number of named
target: companion entries. Normalize accepts the new target and
constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that
name is the option key the backend later receives; a companion may not
claim primary_file, which only means anything for a load target.
ModelConfig.Validate requires exactly one primary and requires it first,
since Artifacts[0] is what ModelFileName, size estimation and staging all
resolve from.
Both acquisition paths now loop instead of touching index 0 alone:
preloadOne for an already-installed config, bindPrimaryArtifact for a
gallery install. Failure policy differs by provenance. An inferred
primary keeps its warn-and-fall-back, because the legacy download path
still exists for it. Companions are explicit by construction, so they are
all-or-nothing: a config naming one is asserting the backend needs it,
and failing at the acquisition boundary is far more legible than a
missing-weights error surfacing later inside the backend.
The cache key is deliberately unchanged. It hashes source identity only,
never name or target, so every already-installed managed model still hits
its existing snapshot instead of silently re-downloading. Two specs pin
that: one proving a companion and a primary with identical sources agree
on the key, and one pinning the digest of a known primary outright.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): hand resolved companion snapshots to the backend
A materialized companion is useless until the backend can find it, and
its location is a content-addressed cache key that does not exist until
the artifact resolves. A static gallery override cannot carry that, and
persisting it into the config YAML would rot the moment a re-resolve
produced a new key.
Synthesize it instead at load time: each resolved companion becomes
"<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the
key:value convention backends already parse for options like
attention_backend. The value stays relative to the models directory so a
remote worker can resolve it under its own ModelPath once staging has
rewritten the model root. An option the author set explicitly always
wins, so pinning a companion to a local checkout still beats the managed
snapshot.
longcat-video resolves base_model through ModelPath, the same convention
qwen-tts, voxcpm, outetts and ace-step already use for companion assets.
Its sibling-directory heuristic is deleted: it looked for a LongCat-Video
directory next to the model, which cannot exist under the content
addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead
code the moment the model became managed.
The gallery entry declares both repositories and restricts each with
allow_patterns. The avatar repo ships base_model/ and base_model_int8/
and only ever loads one, so fetching the whole repo would roughly double
the download. The patterns match the entry's own options (use_distill
true, use_int8 default false); enabling use_int8 here also requires
adding base_model_int8/**, which is called out in the entry.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage managed artifact trees from the models root
Staging anchored the worker's models directory on the primary snapshot
whenever a model was managed, so a companion snapshot could not reach the
worker at all.
frontendModelsDir was derived by stripping the Model relative path off
the end of ModelFile. For a managed artifact nothing matches: ModelFile
is .artifacts/huggingface/<key>/snapshot while Model stays a bare
HuggingFace repo id, so the strip was a no-op and the "models directory"
came out as the snapshot itself. Two consequences, both silent. Staging
keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two
snapshots of one model were indistinguishable on the worker. And a
companion, which lives in a sibling snapshot directory outside the
primary, fell outside that directory entirely: StagingKeyMapper.Key
collapsed its files to bare basenames and resolveOptionPath could not
resolve the relative option at all, so it was skipped without a word.
Derive the models root from the artifact tree instead when the path runs
through it, and compute the worker's ModelPath from the file's path
relative to that root rather than from the Model field. The legacy layout
is unaffected: where Model really is the relative path, the new
derivation reduces to the old one, which a regression spec pins.
This deliberately changes an invariant that router_dirstage_test.go
pinned: for a managed primary, ModelFile and ModelPath were both the
snapshot directory, and staging keys were relative to it. Now ModelFile
is the snapshot, ModelPath is the models root above it, and keys keep the
full relative path. That spec is updated rather than accommodated, with
the reasoning recorded inline, because the old invariant is exactly what
made a sibling companion unreachable.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Pin vLLM protogen to grpcio-tools 1.78.0 so its generated code remains importable by protobuf 6.33.x, and remove stale generated artifacts before regeneration.
Closes#10940
Assisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The sglang Python backend inherits the default Status RPC from
backend_pb2_grpc.BackendServicer, which raises NotImplementedError.
LocalAI's backend-monitor polls /backend.Backend/Status periodically on
every registered backend; when the call fails, /backend/monitor returns
HTTP 500 and downstream inference requests to the sglang backend are
blocked even though the model is loaded and answering directly via the
gRPC endpoint.
Add a minimal Status shim that mirrors the existing Health method and
returns StatusResponse{state=READY} unconditionally. This unblocks the
monitor path; a state-aware follow-up (UNINITIALIZED during load, BUSY
under active inference) is left for a subsequent change.
Reproduced on DGX Spark (GB10, arm64-l4t-cuda-13 image) with the sglang
v0.5.15 backend and Qwen3-Coder-Next-NVFP4-GB10; verified locally that
patching the shim in place immediately restores /backend/monitor and
inference across the sglang slot.
PyTorch 2.13 XPU pulls oneAPI 2026 libraries that conflict with the oneAPI 2025.3 backend image. Pin torch and torchaudio to the matching 2.11 XPU pair so the build resolves a coherent 2025.3 runtime.
Assisted-by: Codex:GPT-5 [uv]
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
SetDefaults injected the llama.cpp server options cache_reuse
(ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied
per selected node by the distributed router) onto every model config
regardless of backend. Every other backend ignores options it does not
understand, so this was harmless until longcat-video, which strictly
validates its options and fails LoadModel with
"unknown model option(s): cache_reuse, parallel".
Gate both injections behind a new UsesLlamaCppServingOptions allow-list
(llama-cpp plus the empty/auto-detect case that resolves to llama.cpp
from a GGUF file, mirroring how llamaCppDefaults is registered). This
follows the existing UsesLlamaSamplerDefaults precedent for llama-only
defaults. The typed NBatch field is deliberately left alone: it is a
proto field every backend simply ignores, which is why batch never
triggered the error.
Also harden the longcat-video backend to warn-and-ignore unknown model
options and request params through a testable select_known_options
helper, matching the other LocalAI Python backends, so a future
server-injected option cannot break loading again.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): add voice library workflow
Give administrators a production-ready flow to record or upload consented reference audio, manage reusable profiles, inspect API usage, discover compatible models, and hand a saved voice directly to text-to-speech.
Assisted-by: Codex:gpt-5
* feat(voice): add managed voice cloning profiles
Make reusable reference voices manageable through the admin API instead of requiring model-directory and YAML edits. Discover compatible installed and gallery models from server-side backend capabilities, retain explicit model configuration controls, and stage saved references for supported backends.
Expose profile management through REST and MCP, document backend-specific behavior, and cover the workflow from profile creation through real Qwen3-TTS synthesis. Harden the agent-job HTTP test against completion racing cancellation.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backends): add LongCat video and avatar generation
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web]
* refactor(config): declare model I/O modalities
Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook.
Assisted-by: Codex:GPT-5 [web]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(backends): pin grpcio-tools to the installed grpcio in runProtogen
runProtogen installed grpcio-tools unpinned, so the protoc it bundles
stamped backend_pb2.py with the newest Protobuf gencode (7.35.0). When a
backend caps the protobuf runtime lower -- vLLM pins protobuf to 6.33.6 --
the import-time guarantee runtime >= gencode fails:
google.protobuf.runtime_version.VersionError: Detected incompatible
Protobuf Gencode/Runtime versions ... gencode 7.35.0 runtime 6.33.6
The backend crashes on `import backend_pb2` before it can serve, which
surfaces to the user as "grpc service not ready". It was mis-reported as a
ROCm/gfx1201 failure in #10718 but is not GPU-specific and affects every
vLLM variant (and any backend that caps protobuf below the latest gencode).
Pin grpcio-tools to the grpcio version the backend already installed --
they release in lockstep -- so the generated gencode stays in step with
the protobuf runtime. Falls back to unpinned when grpcio isn't present.
Closes#10718
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The image backends call PIL Image.save(request.dst) without a format, so
Pillow infers the encoder from the file extension. The core passes an
absolute staging path ending in .tmp (e.g. /staging/localai-output-*.tmp),
which Pillow can't map to a format, raising "unknown file extension: .tmp"
and crashing the worker right after a successful GPU inference.
Pass format="PNG" explicitly. LocalAI serves generated images as PNG
regardless of the temporary path, so this is always correct and no longer
depends on the extension of the destination the core happens to allocate.
diffusers is the reported backend (#10727); vllm-omni and tinygrad carry
the identical latent crash for any .tmp staging destination.
Closes#10727
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The nvidia-l4t-cuda-13-arm64 vLLM backend left `vllm` unpinned, so the
prebuilt image drifted onto whatever aarch64 wheel was latest at build
time (0.23.x). On GB10 / DGX Spark (Grace Blackwell, unified memory),
0.23 crashes deterministically during cold model loads with an empty
"Engine core initialization failed" set and pins GPU memory until a host
reboot.
vLLM 0.24.0 carries vllm-project/vllm#45179 ("release cached device
memory under pressure on UMA GPUs during weight loading"), which the
reporter verified fixes the crash on GB10. Pin the L4T requirements to
0.24.0 to match the already-pinned cublas13 build
(requirements-cublas13-after.txt) and keep the image deterministic.
Editing this file also re-triggers the single-arch L4T image build via
the path filter, republishing the gallery image with 0.24.0 (the
single-arch matrix builds again after #10703).
Closes#10722
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Two bugs broke OpenAI-style tool calling on the MLX backend (and any
Python backend sharing backend/python/common), reproduced end-to-end on
LocalAI v4.5.5 with the metal-mlx backend and
mlx-community/Qwen3.5-2B-MLX-8bit.
messages_to_dicts left each tool call's function.arguments as the raw
OpenAI-wire JSON string. HuggingFace chat templates (e.g. Qwen3.5)
iterate arguments as a mapping (.items()), so any request whose history
contained a prior assistant tool_calls message failed with HTTP 500
"Generation failed: Can only get item pairs from a mapping." — breaking
every agent loop on its second turn. Decode the string back into a dict
so the template sees a mapping.
split_reasoning returned ("", text) whenever the opening think tag was
absent. Models like Qwen3.5 open the assistant turn already inside
thinking, so the generated text carries only the closing </think>; the
whole chain-of-thought leaked into content. When the opener is missing
but the closer is present, treat everything before the closer as
reasoning.
Adds platform-independent unit tests under backend/python/common
(stdlib-only, no MLX/venv required, following parent_watch_test.py).
Assisted-by: Claude Code:claude-opus-4-8
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm): install ROCm vLLM from the AMD wheel index on Python 3.12
The rocm-vllm backend crashed at load with "No module named 'vllm'".
requirements-hipblas-after.txt requested a bare `vllm`, which resolves to
the CUDA-only PyPI wheel; that wheel is unusable on an AMD GPU. vLLM's
prebuilt ROCm wheels live on a dedicated index (https://wheels.vllm.ai/rocm/)
and are published only for CPython 3.12, so on the backend's default 3.10
the installer silently falls back to the CUDA wheel.
Add a hipblas branch to backend/python/vllm/install.sh that pins Python to
3.12 and installs vllm from the ROCm wheel index, hiding the bare-`vllm`
after-file so installRequirements installs only the base ROCm
torch/transformers first and does not pull the CUDA wheel.
Fixes#10642
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vllm): drop the dead hipblas-after requirement and its hide dance
requirements-hipblas-after.txt (a bare `vllm`) is never installed for
hipblas: installRequirements only adds requirements-${BUILD_PROFILE}-after.txt
when BUILD_TYPE != BUILD_PROFILE, and for hipblas they are equal. So the file
was dead and the install.sh hide/restore of it was a no-op. Remove both. The
hipblas branch already installs vllm explicitly from the ROCm wheel index, so
deleting the bare-`vllm` file also removes a latent CUDA-wheel trap should the
installRequirements gap ever be closed.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(grpc): self-terminate backend workers when LocalAI dies non-gracefully
Symptom: a backend model-worker subprocess (the per-model gRPC server LocalAI
spawns) can be orphaned and linger — holding VRAM and its listen port — if the
LocalAI process is killed non-gracefully (e.g. a supervisor's graceful-shutdown
grace period elapses and LocalAI is SIGKILLed) before its own teardown runs.
Root cause: LocalAI's graceful teardown (pkg/signals/handler.go installs the
SIGINT/SIGTERM handler; core/cli/run.go registers app.Shutdown ->
ModelLoader.StopAllGRPC -> process.Stop in pkg/model/process.go) only runs when
LocalAI receives a catchable signal and survives long enough to run its
handlers. Backends are spawned via github.com/mudler/go-processmanager v0.1.1,
whose getSysProcAttr() sets Setpgid:true (own process group, so the group can be
signalled) but never PR_SET_PDEATHSIG/Pdeathsig, and exposes no Config field or
option for a caller to inject/extend SysProcAttr. LocalAI fully delegates
spawning to that library (it never builds the exec.Cmd itself), so it cannot set
a kernel parent-death signal at the spawn site. If LocalAI is SIGKILLed, nothing
tells the backend to exit and it is reparented to init.
Fix: add a best-effort, backend-side safety net at the one shared choke point
every out-of-process Go backend routes through — grpc.StartServer / RunServer in
pkg/grpc. On startup it captures getppid() and polls; when the process is
reparented (getppid changes / becomes 1 — the standard POSIX signal the original
parent died) it logs and self-terminates. getppid() reparent detection is
portable (Linux + macOS), unlike Linux-only PR_SET_PDEATHSIG. Toggle via
LOCALAI_BACKEND_PARENT_WATCH (default on; off on Windows) and
LOCALAI_BACKEND_PARENT_WATCH_INTERVAL. This is strictly a backstop alongside the
existing graceful SIGTERM->grace->SIGKILL teardown, which is unchanged.
Scope/limitations: covers Go-based backends (everything using pkg/grpc). The
C++ backends (e.g. llama-cpp) and Python backends do not route through
pkg/grpc and are not covered by this mechanism — they would each need an
equivalent parent-death check (follow-up). The fully general fix is for
go-processmanager to expose SysProcAttr injection so LocalAI can set Pdeathsig
at spawn for every backend regardless of language (suggested upstream follow-up;
out of scope for this LocalAI-only PR).
Test: pkg/grpc/parentwatch_test.go builds a real test -> middle -> grandchild
process tree, lets the middle process exit to orphan the grandchild running the
real watchParentDeath, and asserts it detects the reparent and self-terminates.
Unix-only (build-tagged), runs in CI (Linux).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(process): extend parent-death backstop to C++ and Python backends
The Go parent-death watcher (pkg/grpc/parentwatch.go, commit 772b435d5)
only protects backends that route through pkg/grpc. C++ and Python
backends don't, so the originally-reported case — the llama.cpp gRPC
worker surviving a non-graceful LocalAI death — was still uncovered.
Extend the same best-effort backstop to both languages, reusing the
exact mechanism and semantics:
- capture getppid() at startup, skip if already orphaned (<=1)
- a background thread polls getppid() and self-exits on reparenting
(getppid() != orig || == 1), portable across Linux/macOS, no-op on
Windows
- same env vars: LOCALAI_BACKEND_PARENT_WATCH (default on; falsy
false/0/no/off disable) and LOCALAI_BACKEND_PARENT_WATCH_INTERVAL
(default 2s; accepts Go-style durations like 500ms/2s/1m)
C++: implemented in backend/cpp/llama-cpp (the reported, most-used C++
backend) as a dependency-free header parent_watch.h, wired into
grpc-server.cpp's main() and copied at build time via prepare.sh. C++
backends have no shared server scaffolding, so other C++ backends
(ds4, ik-llama-cpp, privacy-filter, ...) are not yet covered and would
each need the same one-line include+call as follow-ups.
Python: implemented once in the shared common/parent_watch.py and armed
from common/grpc_auth.py's get_auth_interceptors() — the single helper
every one of the 35 Python backends invokes while building its gRPC
server — so all Python backends (and future ones) are covered with no
per-backend edits and no duplicated implementation.
Tests (real process-tree reparent detection, mirroring the Go test):
- backend/cpp/llama-cpp/parent_watch_test.cpp (via run-unit-tests.sh)
- backend/python/common/parent_watch_test.py (python -m unittest)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
fix(vllm): non-streaming tool-call regression after #10351 (native_streaming is a capability flag, not a state flag)
#10351 introduced native streaming via `parser.extract_tool_calls_streaming`
and gated the post-loop `extract_tool_calls` block on `native_streaming and
not native_streaming_error`. That works for streaming requests, but for
non-streaming requests the same flag is still True (it only means "the
parser can stream", not "we actually streamed"), so the block was skipped
and the `elif` cleared `content = ""` — the tool call was silently lost.
Symptom: non-streaming chat.completions with `tools=[...]` returns
`finish_reason: "stop"` with `content: ""` and no `tool_calls`. Streaming
requests are unaffected.
Fix: gate both branches on `streaming` too, so the extract_tool_calls
block runs for non-streaming requests (and for streaming requests that
fell back to the buffered path).
Reproduction (vLLM 0.24, Qwen3-Coder-Next-NVFP4, qwen3_coder parser):
curl -s -X POST http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"coder","stream":false,
"messages":[{"role":"user","content":"7*8 via calc"}],
"tools":[{"type":"function","function":{"name":"calc",
"parameters":{"type":"object",
"properties":{"expression":{"type":"string"}}}}}]}'
Before: finish_reason: "stop", content: "", tool_calls: []
After: finish_reason: "tool_calls", tool_calls[0].function.name: "calc"
Streaming path re-verified in the same setup: delta.tool_calls arrives
token-by-token, finish_reason: "tool_calls", no raw XML in content.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
