* feat(backends): add LongCat video and avatar generation
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web]
* refactor(config): declare model I/O modalities
Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook.
Assisted-by: Codex:GPT-5 [web]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
When a model config has no explicit backend, the model loader greedily
probes every installed backend and binds to the first Load that
succeeds. opus and local-store were the only in-tree backends with no
model artefact to validate, so they accepted anything — an LLM
installed after them could silently bind to the audio codec or the
vector store and then fail at inference with "unimplemented"
(see #9287).
opus now accepts only its own name (what the realtime WebRTC path
sends) or none. local-store namespaces are arbitrary (router caches,
biometrics, user-named stores), so core's StoreBackend now marks
genuine store loads with a store:// prefix on the gRPC model name and
the backend refuses names without it; core and backend ship from the
same release, so the convention upgrades in lockstep.
Also repair the bit-rotted 'make test-stores' bootstrap (the suite
never registered external backends, so BACKENDS_PATH was dead weight)
and add the Load-validation rule to the adding-backends checklist.
Related: #9287
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(vibevoice-cpp): true streaming TTSStream via vv_capi_tts_stream
Replaces the synth-to-tempfile TTSStream hack with a real streaming path:
binds the new vv_capi_tts_stream callback ABI via a single reusable purego
callback (CGO_ENABLED=0-safe, no runtime/cgo), copies each int16 PCM window
into the gRPC results channel after the streaming WAV header.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* test(vibevoice-cpp): real-model streaming integration test with TTFA measurement
Gated behind VIBEVOICE_IT=1, this Ginkgo spec dlopens the engine .so and
drives the exact Go->purego->C TTSStream/TTS path against the real
vibevoice-realtime-0.5B model. It measures time-to-first-audio for the
streaming path versus the batch path and asserts the streaming win:
44-byte WAV header first, >=2 PCM windows, non-silent audio, and
TTFA < total_stream. Without the env var the spec skips so CI and
normal go test are unaffected.
Measured: TTFA 2.38s vs batch deliver-time 39.96s (first audio in 5.9%
of the batch time, ~17x faster), 18 stream chunks, non-silent 24kHz PCM.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vibevoice-cpp): pin streaming-decoder engine build
Bumps VIBEVOICE_CPP_VERSION to the streaming-decoder engine commit that
adds vv_capi_tts_stream (localai-org/vibevoice.cpp#8). Re-pin to the
merged master commit once that PR lands.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vibevoice-cpp): re-pin to merged streaming-decoder commit
localai-org/vibevoice.cpp#8 merged to master as 000e372; move the pin
off the PR branch commit onto the merged master commit.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* test(vibevoice-cpp): check writer errors in TTFA report (errcheck)
golangci-lint errcheck flagged the unchecked fmt.Fprintf calls that
print the streaming TTFA headline. Build the report once with
fmt.Sprintf and write it per destination with an explicitly discarded
error, matching the GinkgoWriter reporting idiom used by the other
backend tests.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-fable-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
C++/ggml transcription + speaker diarization + timestamps backend. Purego
dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp
and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output
into segments with nanosecond timestamps. Adds the importer (surfaces in
GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml,
and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf).
Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues)
passed against the real libmoss-transcribe.so. The pre-commit coverage gate
(full pkg/core + tests/e2e) could not run in the authoring sandbox (no live
models, port 9090 held); CI must enforce it before merge.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Link the f5-tts library into the crispasr backend so CrispASR's native
F5-TTS runtime (SWivid F5-TTS, 22-layer DiT flow-matching + built-in Vocos
vocoder) is compiled in. The single self-contained GGUF auto-detects as
f5-tts through the session router, so no explicit backend selector is
needed. Add the f5-tts-crispasr gallery entry (cstr/f5-tts-GGUF) and an
env-gated e2e synthesis spec.
F5-TTS is voice-cloning only and has no baked speaker: it clones from a
reference WAV plus its transcript, supplied via the voice/voice_text
options. The gallery description documents this bring-your-own-reference
requirement.
Verified e2e on the pinned engine (278fb79): the GGUF auto-detects as
f5-tts, the reference voice loads, and synthesis produces a valid 24 kHz
mono WAV.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(backends): pin grpcio-tools to the installed grpcio in runProtogen
runProtogen installed grpcio-tools unpinned, so the protoc it bundles
stamped backend_pb2.py with the newest Protobuf gencode (7.35.0). When a
backend caps the protobuf runtime lower -- vLLM pins protobuf to 6.33.6 --
the import-time guarantee runtime >= gencode fails:
google.protobuf.runtime_version.VersionError: Detected incompatible
Protobuf Gencode/Runtime versions ... gencode 7.35.0 runtime 6.33.6
The backend crashes on `import backend_pb2` before it can serve, which
surfaces to the user as "grpc service not ready". It was mis-reported as a
ROCm/gfx1201 failure in #10718 but is not GPU-specific and affects every
vLLM variant (and any backend that caps protobuf below the latest gencode).
Pin grpcio-tools to the grpcio version the backend already installed --
they release in lockstep -- so the generated gencode stays in step with
the protobuf runtime. Falls back to unpinned when grpcio isn't present.
Closes#10718
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The image backends call PIL Image.save(request.dst) without a format, so
Pillow infers the encoder from the file extension. The core passes an
absolute staging path ending in .tmp (e.g. /staging/localai-output-*.tmp),
which Pillow can't map to a format, raising "unknown file extension: .tmp"
and crashing the worker right after a successful GPU inference.
Pass format="PNG" explicitly. LocalAI serves generated images as PNG
regardless of the temporary path, so this is always correct and no longer
depends on the extension of the destination the core happens to allocate.
diffusers is the reported backend (#10727); vllm-omni and tinygrad carry
the identical latent crash for any .tmp staging destination.
Closes#10727
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The nvidia-l4t-cuda-13-arm64 vLLM backend left `vllm` unpinned, so the
prebuilt image drifted onto whatever aarch64 wheel was latest at build
time (0.23.x). On GB10 / DGX Spark (Grace Blackwell, unified memory),
0.23 crashes deterministically during cold model loads with an empty
"Engine core initialization failed" set and pins GPU memory until a host
reboot.
vLLM 0.24.0 carries vllm-project/vllm#45179 ("release cached device
memory under pressure on UMA GPUs during weight loading"), which the
reporter verified fixes the crash on GB10. Pin the L4T requirements to
0.24.0 to match the already-pinned cublas13 build
(requirements-cublas13-after.txt) and keep the image deterministic.
Editing this file also re-triggers the single-arch L4T image build via
the path filter, republishing the gallery image with 0.24.0 (the
single-arch matrix builds again after #10703).
Closes#10722
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Allow llama.cpp model configs to select the backend devices used for offload, matching upstream --device behavior so users can exclude a display or debug GPU.
Signed-off-by: rvmzes <rvmzes@rvmzess-MacBook-Pro.local>
Co-authored-by: rvmzes <rvmzes@rvmzess-MacBook-Pro.local>
* ⬆️ Update leejet/stable-diffusion.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(stablediffusion-ggml): pass chroma knobs via model_args after upstream API change
Upstream stable-diffusion.cpp bb849711 removed the dedicated
chroma_use_dit_mask / chroma_use_t5_mask / chroma_t5_mask_pad fields from
sd_ctx_params_t and now reads them from the generic model_args key=value
spec (parse_key_value_args). Assigning the old struct members broke the
gosd.cpp build. Emit the three options into model_args instead so the
existing chroma controls keep working. Verified by building
libgosd-fallback.so against the pinned upstream commit.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
