* feat(backends): add LongCat video and avatar generation
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web]
* refactor(config): declare model I/O modalities
Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook.
Assisted-by: Codex:GPT-5 [web]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add four ready-to-run DFlash speculative-decoding entries for the
llama.cpp backend, now that upstream DFlash support (draft-dflash) is in
the pinned llama.cpp. Each entry bundles a full target model with its
small z-lab block-diffusion drafter and sets spec_type:draft-dflash,
spec_n_max:15, and flash attention (required by DFlash):
- qwen3-4b-dflash (Qwen3-4B + Qwen3-4B-DFlash drafter)
- qwen3.5-9b-dflash (Qwen3.5-9B + Qwen3.5-9B-DFlash drafter)
- qwen3.6-27b-dflash (Qwen3.6-27B dense + drafter)
- qwen3.6-35b-a3b-dflash (Qwen3.6-35B-A3B MoE + drafter)
The 4B pair uses the base Qwen3-4B target (not Qwen3.5-4B): its drafter
reports general.name "Qwen3 4B DFlash" and is the canonical pairing
documented upstream. All drafters were downloaded and verified to carry
GGUF architecture "dflash" (not the fork-only "dflash-draft" /
"DFlashDraftModel") so they load in the upstream backend, and every
drafter SHA256 was confirmed against the downloaded bytes.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs: refresh LocalAI homepage
Reframe the homepage around LocalAI's modular multimodal runtime, native inference engines, deployment range, and built-in platform capabilities. Remove the outdated video in favor of current product visuals and clearer paths into the documentation.
Assisted-by: Codex:gpt-5
* docs: give homepage a full-width canvas
Let the product homepage opt out of Relearn's persistent sidebar and duplicate title while preserving the documentation shell on interior pages. Tighten the responsive bounds for narrow screens.
Assisted-by: Codex:gpt-5
* docs: fit homepage to the Relearn content flow
Remove the full-width shell exception and use a single-column homepage inside the standard documentation layout. This avoids competing scroll containers and the compressed split hero.
Assisted-by: Codex:gpt-5
* docs: hide homepage scroll rail
Preserve Relearn's content scrolling while removing the visible scrollbar beside the landing-page hero.
Assisted-by: Codex:gpt-5
* docs: contain homepage sections within docs column
Prevent landing-page headings, figures, and section grids from widening Relearn's content pane or exposing overflow rails.
Assisted-by: Codex:gpt-5
* docs: remove nested homepage scrollbars
Wrap the quick-start command within its column and suppress component-level scrollbar tracks across the landing page.
Assisted-by: Codex:gpt-5
* docs: remove outdated gallery screenshot
Drop the stale Model Gallery image from the homepage until a current product visual is available.
Assisted-by: Codex:gpt-5
* docs: fix homepage architecture link
Point the homepage CTA at the generated reference/architecture route.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
When a model config has no explicit backend, the model loader greedily
probes every installed backend and binds to the first Load that
succeeds. opus and local-store were the only in-tree backends with no
model artefact to validate, so they accepted anything — an LLM
installed after them could silently bind to the audio codec or the
vector store and then fail at inference with "unimplemented"
(see #9287).
opus now accepts only its own name (what the realtime WebRTC path
sends) or none. local-store namespaces are arbitrary (router caches,
biometrics, user-named stores), so core's StoreBackend now marks
genuine store loads with a store:// prefix on the gRPC model name and
the backend refuses names without it; core and backend ship from the
same release, so the convention upgrades in lockstep.
Also repair the bit-rotted 'make test-stores' bootstrap (the suite
never registered external backends, so BACKENDS_PATH was dead weight)
and add the Load-validation rule to the adding-backends checklist.
Related: #9287
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>
The vibevoice transcription e2e hangs until the go test timeout when
the HF CDN is slow: the ASR Q4_K model is >10 GB, downloadFile capped
every curl attempt at --max-time 600 (needs a sustained ~17 MB/s to
fit), and curl's --retry restarts from byte zero, so no attempt ever
makes forward progress. This killed the job twice on PR #10764 and
previously forced skipping it on release tags (#10567).
Replace the wall-clock cap with stall detection (--speed-limit 1 MiB/s
over --speed-time 120s) and resume from the bytes already on disk with
-C -, retrying from Go because curl does not re-evaluate the resume
offset on its internal retries. Resume against the HF Xet CDN was
verified by killing a transfer mid-flight and confirming the next
invocation appended (114 MB -> 235 MB, GGUF magic intact).
Also parameterize the suite timeout (BACKEND_TEST_TIMEOUT, default
30m) and raise it to 120m for the vibevoice transcription wrapper: a
10 GB download plus 25 specs does not fit in 30m even on a good day,
and the job-level GHA timeout there is already 150m.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vibevoice-cpp): true streaming TTSStream via vv_capi_tts_stream
Replaces the synth-to-tempfile TTSStream hack with a real streaming path:
binds the new vv_capi_tts_stream callback ABI via a single reusable purego
callback (CGO_ENABLED=0-safe, no runtime/cgo), copies each int16 PCM window
into the gRPC results channel after the streaming WAV header.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* test(vibevoice-cpp): real-model streaming integration test with TTFA measurement
Gated behind VIBEVOICE_IT=1, this Ginkgo spec dlopens the engine .so and
drives the exact Go->purego->C TTSStream/TTS path against the real
vibevoice-realtime-0.5B model. It measures time-to-first-audio for the
streaming path versus the batch path and asserts the streaming win:
44-byte WAV header first, >=2 PCM windows, non-silent audio, and
TTFA < total_stream. Without the env var the spec skips so CI and
normal go test are unaffected.
Measured: TTFA 2.38s vs batch deliver-time 39.96s (first audio in 5.9%
of the batch time, ~17x faster), 18 stream chunks, non-silent 24kHz PCM.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vibevoice-cpp): pin streaming-decoder engine build
Bumps VIBEVOICE_CPP_VERSION to the streaming-decoder engine commit that
adds vv_capi_tts_stream (localai-org/vibevoice.cpp#8). Re-pin to the
merged master commit once that PR lands.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vibevoice-cpp): re-pin to merged streaming-decoder commit
localai-org/vibevoice.cpp#8 merged to master as 000e372; move the pin
off the PR branch commit onto the merged master commit.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* test(vibevoice-cpp): check writer errors in TTFA report (errcheck)
golangci-lint errcheck flagged the unchecked fmt.Fprintf calls that
print the streaming TTFA headline. Build the report once with
fmt.Sprintf and write it per destination with an explicitly discarded
error, matching the GinkgoWriter reporting idiom used by the other
backend tests.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-fable-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
C++/ggml transcription + speaker diarization + timestamps backend. Purego
dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp
and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output
into segments with nanosecond timestamps. Adds the importer (surfaces in
GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml,
and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf).
Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues)
passed against the real libmoss-transcribe.so. The pre-commit coverage gate
(full pkg/core + tests/e2e) could not run in the authoring sandbox (no live
models, port 9090 held); CI must enforce it before merge.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Link the f5-tts library into the crispasr backend so CrispASR's native
F5-TTS runtime (SWivid F5-TTS, 22-layer DiT flow-matching + built-in Vocos
vocoder) is compiled in. The single self-contained GGUF auto-detects as
f5-tts through the session router, so no explicit backend selector is
needed. Add the f5-tts-crispasr gallery entry (cstr/f5-tts-GGUF) and an
env-gated e2e synthesis spec.
F5-TTS is voice-cloning only and has no baked speaker: it clones from a
reference WAV plus its transcript, supplied via the voice/voice_text
options. The gallery description documents this bring-your-own-reference
requirement.
Verified e2e on the pinned engine (278fb79): the GGUF auto-detects as
f5-tts, the reference voice loads, and synthesis produces a valid 24 kHz
mono WAV.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(schema): accept reasoning_content as inbound alias for reasoning
Interleaved-thinking clients (cogito, vLLM/DeepSeek-style) emit reasoning_content
on assistant turns. Accept it as an inbound alias so reasoning survives the
tool-result loop; canonical reasoning wins when both are present. Emission is
unchanged (still reasoning).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(schema): pin interleaved reasoning+tool_calls round-trip
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(openai): pin reachedTokenBudget truncation detection
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): add thinking and signature fields to content blocks
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): parse inbound thinking blocks into reasoning
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): emit thinking blocks with synthetic signature on tool turns
Extract buildAnthropicContentBlocks so non-streaming content assembly is
unit-testable, and prepend a thinking block (with an opaque synthetic
signature) before text/tool_use blocks when the request opts into thinking.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): stream thinking_delta and signature_delta before tool_use
Extract anthropicStreamSequence so the streaming block order is unit-testable,
and emit content_block_start(thinking) -> thinking_delta -> signature_delta ->
content_block_stop before the tool_use block sequence when thinking is enabled.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: add interleaved thinking with tool calls guide
Add a features guide describing interleaved thinking: an assistant turn
carrying reasoning and tool_calls together, the reasoning-round-trip
contract (including the reasoning_content inbound alias and Anthropic
thinking blocks with a synthetic signature), per-backend enablement
(reasoning_format for llama.cpp, reasoning_parser/tool_call_parser for
vLLM/SGLang plus the vLLM auto-config hook), a worked request/response
example, and known limitations. Cross-link from model-configuration,
text-generation, and openai-functions.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(backends): pin grpcio-tools to the installed grpcio in runProtogen
runProtogen installed grpcio-tools unpinned, so the protoc it bundles
stamped backend_pb2.py with the newest Protobuf gencode (7.35.0). When a
backend caps the protobuf runtime lower -- vLLM pins protobuf to 6.33.6 --
the import-time guarantee runtime >= gencode fails:
google.protobuf.runtime_version.VersionError: Detected incompatible
Protobuf Gencode/Runtime versions ... gencode 7.35.0 runtime 6.33.6
The backend crashes on `import backend_pb2` before it can serve, which
surfaces to the user as "grpc service not ready". It was mis-reported as a
ROCm/gfx1201 failure in #10718 but is not GPU-specific and affects every
vLLM variant (and any backend that caps protobuf below the latest gencode).
Pin grpcio-tools to the grpcio version the backend already installed --
they release in lockstep -- so the generated gencode stays in step with
the protobuf runtime. Falls back to unpinned when grpcio isn't present.
Closes#10718
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The OIDC verifier was built with a bare oidc.Config{ClientID: ...}, so
go-oidc applied its default of accepting RS256-signed ID tokens only. An
identity provider configured with an EC signing key (e.g. Authentik) issues
ES256-signed tokens, and the callback failed verification with:
failed to verify ID token: oidc: malformed jwt: unexpected signature
algorithm "HS256"; expected ["RS256"]
surfacing to the user as HTTP 500 "failed to fetch user info" (#10677; the
underlying cause became visible after the logging fix in #10679).
Set SupportedSigningAlgs to the standard asymmetric algorithms
(RS256/384/512, ES256/384/512, PS256/384/512, EdDSA). All are verified
against the provider's published JWKS. HS256 is intentionally excluded: it
is symmetric and would validate against the client secret, a different and
security-sensitive trust model.
Tested with a functional spec that signs an ES256 ID token and confirms it
verifies with the configured algorithms and is rejected under go-oidc's
RS256-only default (using oidc.StaticKeySet, no network).
Closes#10677
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Backend log capture into the per-model BackendLogStore (which feeds the
UI "Backend Logs" page and /api/backend-logs) was opt-in and off by
default in single mode, while worker/distributed mode force-enables it
via SetBackendLoggingEnabled(true). There was no CLI flag either, so the
only way to populate the store was the Settings UI toggle - and the page
was silently empty out of the box. Distributed "just worked"; single
mode looked broken.
Default EnableBackendLogging to true in NewApplicationConfig so single
mode matches worker mode. The store is a small in-memory ring buffer, so
the cost is negligible.
Now that the default is on, loadRuntimeSettingsFromFile's usual
"only flip false->true" merge would ignore a persisted false and revert
the UI toggle-off on every restart. There is no env var/CLI flag for
this setting, so an explicit persisted value is now authoritative in
both directions, letting the toggle-off survive a restart.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
