* fix(backends): choose the protoc generator from the protobuf runtime, and regenerate stubs after late installs
The vLLM backends still crash on startup with
VersionError: Detected incompatible Protobuf Gencode/Runtime versions when
loading backend.proto: gencode 7.35.0 runtime 6.33.6
despite #10735 and #10944. Three separate defects kept it alive.
1. runProtogen picked the generator from the installed *grpcio* version.
grpcio-tools' version tracks grpcio, but the gencode its bundled protoc
emits tracks *protobuf*, and the two move independently: grpcio-tools
1.82.1 (the version #10735 pins to, matching grpcio 1.82.1) requires
protobuf>=7.35.1 and stamps gencode 7.35.0. Pinning to grpcio could
therefore never constrain the gencode. Constrain the install to the
protobuf already in the venv instead and let the resolver pick the newest
compatible grpcio-tools. That both selects a generator the runtime accepts
and stops protogen from moving the runtime under the backend's other deps.
This is self-correcting, so the hardcoded GRPCIO_TOOLS_VERSION=1.78.0
escape hatch from #10944 is no longer needed and is removed.
2. The stubs were generated too early. Most branches of vllm/install.sh (and
vllm-omni) install vllm *after* installRequirements, and vllm re-resolves
the protobuf runtime as it lands. Stubs generated against the pre-vllm
runtime can end up newer than the runtime that finally ships, which is the
ROCm failure exactly. Regenerate once the dependency set is final.
3. rm -f of the .py sources left __pycache__ behind. CPython validates a .pyc
against source mtime and size, both of which can be unchanged across a
regeneration (the gencode triple is the same width whether it reads 7.35.0
or 6.33.5), so a stale backend_pb2.pyc could shadow the stub just written.
Also fail the build when the generated stub cannot be imported, so a
gencode/runtime mismatch surfaces at image build time instead of reaching
users as an opaque "grpc service not ready".
Verified by driving the real runProtogen through the ROCm install sequence in
a venv harness: before, gencode 7.35.0 against runtime 6.33.6 (reproducing the
reported error verbatim); after, gencode 6.33.5 against runtime 6.33.6 and the
stub imports cleanly.
Closes#10940Closes#10718
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): regenerate protobuf stubs in the other backends that install after installRequirements
Same defect as the vllm change: installRequirements generates the stubs at the
end of its own run, so any backend that installs further packages afterwards can
have the protobuf runtime moved out from under stubs that were already written.
The gencode stamped into backend_pb2.py then exceeds the runtime that ships and
the backend dies at model load with "grpc service not ready".
fish-speech already had this bug and worked around the symptom: it forces
protobuf>=5.29.0 after installRequirements precisely because "transitive deps
(wandb, tensorboard) may downgrade protobuf to 3.x but our generated
backend_pb2.py requires protobuf 5+". Regenerating after the pin addresses the
cause rather than propping up the runtime to match stale stubs.
Applied to the backends whose post-installRequirements step resolves a
dependency graph and can therefore move protobuf:
fish-speech -e . plus an explicit protobuf install
vibevoice pip install . (with deps)
llama-cpp-quantization gguf / GGUF_PIP_SPEC
trl gguf / GGUF_PIP_SPEC
Deliberately not applied to ace-step and chatterbox (both --no-deps, so the
dependency graph cannot change) or voxcpm (pins setuptools only). gguf does not
depend on protobuf today, but it resolves dependencies, and "this package does
not touch protobuf right now" is exactly the assumption that made the earlier
fix ineffective.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): resolve the protoc generator in a throwaway env so it cannot edit the backend's pinned deps
Installing grpcio-tools into the backend's own venv to generate the stubs also
drags its dependencies in: grpcio-tools 1.82.1 requires grpcio>=1.82.1, so a
backend that pinned grpcio==1.78.1 silently shipped 1.82.1 instead. Caught by
building the llama-cpp-quantization image and reading the versions back out of
the artifact:
before grpcio 1.82.1 (requirements.txt pins grpcio==1.78.1)
after grpcio 1.78.1 grpcio-tools absent from the venv entirely
Resolve the generator in a throwaway environment instead, still constrained to
the protobuf the backend ships so the gencode stays compatible. The backend's
dependency set is then exactly what its requirements files declared. protoc's
output is plain Python and carries no dependency on the interpreter that
produced it, so generating from a different env is safe; the import check still
runs under the backend's python, since that is the interpreter that has to load
the stubs at model load.
Verified on the rebuilt image: gencode 7.35.0, runtime protobuf 7.35.1, grpcio
back at its pinned 1.78.1, and the shipped stub imports cleanly against 7.35.1.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): bound the protoc generator by BOTH the installed grpcio and protobuf
The generated stubs impose two independent constraints, and every fix so far,
including the previous commit on this branch, satisfied one while violating the
other:
backend_pb2.py needs protobuf runtime >= gencode
backend_pb2_grpc.py needs installed grpcio >= grpcio-tools
Resolving the generator against protobuf alone picked grpcio-tools 1.82.1 for a
backend holding grpcio at 1.78.1, so the gencode was fine but the gRPC stub was
not:
RuntimeError: The grpc package installed is at version 1.78.1, but the
generated code in backend_pb2_grpc.py depends on grpcio>=1.82.1.
That is also why installing grpcio-tools into the backend venv appeared to work
earlier: it dragged grpcio up to match, which was load-bearing rather than the
regression it looked like. Isolating the generator removed the accidental fix
and exposed the missing constraint.
Bound grpcio-tools from both sides instead and let the resolver find the newest
version satisfying both. The protobuf ceiling makes it back off to an older
generator when the runtime trails, bounding the gencode; the grpcio ceiling
keeps the _grpc stub loadable. Resolved against the four real runtime pairs
observed in built images:
grpcio 1.78.1 / protobuf 7.35.1 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.78.0 / protobuf 6.33.6 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.82.1 / protobuf 6.33.6 -> grpcio-tools 1.81.1, gencode 6.33.5 OK
grpcio 1.82.1 / protobuf 7.35.1 -> grpcio-tools 1.82.1, gencode 7.35.0 OK
Also restore the import check to cover backend_pb2_grpc as well as backend_pb2.
Narrowing it to backend_pb2 is why the image build passed while CI failed: the
guard could not see the constraint that was actually broken.
Verified by running the CI sequence locally for llama-cpp-quantization, the
backend whose test failed:
make -C backend/python/llama-cpp-quantization -> exit 0
make -C backend/python/llama-cpp-quantization test -> exit 0, OK
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Every vibevoice requirements file pulled diffusers straight from
git+https://github.com/huggingface/diffusers. That branch now reports
itself as 0.40.0.dev0 and requires huggingface-hub>=1.23.0,<2.0, while
transformers>=4.51.3,<5.0.0 (which upstream VibeVoice mandates) still
caps huggingface-hub at <1.0. Because a git URL offers the resolver
exactly one candidate version, uv has nothing to backtrack to and the
install fails outright:
Because only diffusers==0.40.0.dev0 is available and diffusers==0.40.0.dev0
depends on huggingface-hub>=1.23.0,<2.0 [...] we can conclude that your
requirements are unsatisfiable.
This broke the vibevoice build on every variant - cpu (amd64/arm64),
cuda 12/13, l4t 12/13, intel and rocm, plus the darwin metal job - and
has been failing the weekly full-matrix rebuild for three weeks. It is
not caught by master pushes because backend builds are path-filtered
there, so it only surfaces on the Sunday cron and on release tags.
Use the PyPI package instead. That is what upstream VibeVoice declares
in its own pyproject.toml, and what every other LocalAI backend already
does - vibevoice was the only one tracking the git branch. With a real
release series available the resolver settles on diffusers 0.39.0 with
huggingface-hub 0.36.2 and transformers 4.57.6, and it can keep
backtracking on its own if upstream shifts again.
Verified with uv pip compile against cpu, cublas12, cublas13, hipblas,
intel, mps and l4t13: all resolve to that same coherent set. l4t12 only
resolves on aarch64, since its Jetson index ships no x86_64 torch wheel.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Bash] [uv]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat: add distributed mode (experimental)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix data races, mutexes, transactions
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix events and tool stream in agent chat
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* use ginkgo
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(cron): compute correctly time boundaries avoiding re-triggering
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* enhancements, refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* do not flood of healthy checks
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* do not list obvious backends as text backends
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* tests fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Drop redundant healthcheck
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* enhancements, refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vibevoice): add ASR support
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(tests): download voice files
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Small fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Small fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Try to run on bigger runner
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* debug
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* CI can't hold vibevoice
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix: install only torch/torchvision from jetson index
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix: use pip for l4t-12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Revert "fix: install only torch/torchvision from jetson index"
This reverts commit 2d2b020078
* chatterbox needs wheel
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vibevoice): add backend
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore: add workflow and backend index
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(gallery): add vibevoice
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Use self-hosted for intel builds
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Pin python version for l4t
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
