* feat(backend): add vllm-cpp text-generation backend (vllm.cpp)
Wrap https://github.com/mudler/vllm.cpp - the LocalAI-team from-scratch C++20
port of vLLM (paged KV cache, continuous batching, prefix caching, safetensors
+ GGUF loading, no Python at inference) - as a Go gRPC backend over its stable
C ABI (ABI v2) via purego.
Backend (backend/go/vllm-cpp):
- Load -> vllm_engine_load: accepts a .gguf file or a config.json model dir
(anything else is refused, satisfying the greedy-probe rule); context_size
maps to max_model_len, options block_size/num_blocks/max_num_seqs size the
KV cache and scheduler admission.
- Predict -> vllm_complete (blocking); PredictStream -> vllm_complete_stream
with the per-delta C callback bridged into the gRPC stream. The backend
embeds base.Base (not SingleThread): concurrent requests batch continuously
in the engine's shared AsyncLLM scheduler.
- PredictOptions.Grammar -> the ABI's structured_grammar (GBNF), giving
grammar-constrained tool calling at parity with llama-cpp; the ABI also
exposes JSON-schema/regex/choice constraints.
- Hand-mirrored POD structs with layout locked by unit tests
(unsafe.Offsetof vs the C offsets) and a runtime vllm_abi_version gate.
- One portable library per platform (vllm.cpp uses per-file SIMD tiers with
runtime dispatch), so no avx/avx2/avx512 variant builds.
Wiring:
- backend-matrix: CPU amd64+arm64 (per-arch + manifest merge), CUDA 12/13
amd64 (120a;121a Blackwell fat binary), L4T arm64 (121a, GB10/DGX Spark -
the runtime-proven GPU target), Vulkan amd64, and Darwin arm64 Metal.
- backend/index.yaml meta + 12 image entries (latest/development x cpu,
cuda12, cuda13, l4t, vulkan, metal); bump_deps registration for the
VLLM_CPP_VERSION pin; root Makefile registration; test-extra runs the unit
specs (pure Go, no engine build).
- Importers: preference-only swaps - llama-cpp (GGUF) and vllm (safetensors)
advertise vllm-cpp via AdditionalBackends and emit backend: vllm-cpp
without tokenizer templating (the C ABI takes the FINAL prompt; templating
and tool parsing stay LocalAI-side). No auto-detect importer.
- Docs: backends list, top-level README maintained-engines table,
compatibility table.
Verified: 20/20 Ginkgo specs against the real pinned engine and
Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU - blocking + streaming parity, greedy
determinism, stop words, GBNF-constrained generation, and 4 concurrent
streams; plus a dlopen/ABI-gate smoke of the built gRPC server binary.
Upstream ABI v2 + production structured-output wiring landed as
mudler/vllm.cpp@86013f3.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): ride the autoparser code path - engine-side chat templating and tool engagement (ABI v3)
The backend now implements AIModelRich (PredictRich / PredictStreamRich) over
vllm.cpp's ABI v3 chat entry points, so chat and tool calling ride the SAME
code path as the llama.cpp autoparser: the ENGINE renders the model's chat
template, decides when a tool call engages, and parses it - LocalAI receives
pre-parsed ChatDelta / ToolCallDelta protos exactly as it does from llama-cpp.
- With use_tokenizer_template + structured Messages, PredictOptions lowers to
ONE OpenAI chat request JSON (messages, tools, tool_choice, sampling,
stream_options.include_usage) for vllm_chat / vllm_chat_stream. tool_choice
auto lowers engine-side to a LAZY structural-tag decode constraint - free
text until the model emits the tool trigger, then the call is
grammar-constrained; required/named force a call. Tool output is parsed by
the engine's streaming Hermes-style parser; each chat.completion.chunk maps
onto ChatDeltas (content / reasoning_content / tool_calls) which the host
already prefers over Go-side tag extraction. Without structured messages the
plain path (LocalAI templating + optional GBNF grammar) applies unchanged.
- The engine resolves the chat template from the GGUF tokenizer.chat_template
metadata (or tokenizer_config.json); templates beyond its minja subset -
e.g. the full Qwen3.5 namespace()/macro template - degrade engine-side to a
Hermes-aware fallback prompt (tools schemas + <tool_call> instruction) with
a stderr witness, so structural-tag engagement keeps working.
- Importers now emit the same config shape as llama-cpp for vllm-cpp
(use_tokenizer_template: true, no-grammar autoparser flow); only the
llama-cpp-specific use_jinja option and the vllm-python parser options are
dropped.
- Pin bumped to mudler/vllm.cpp@aaed7ec (ABI v3 + chat-prompt resolution).
Verified against the real engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU: full
suite green - blocking chat, streaming deltas concatenating byte-equal to the
blocking answer, a REQUIRED tool call returning schema-valid arguments JSON,
and an AUTO run where the engine itself engages get_weather and streams parsed
tool deltas; plus unit specs for the request lowering, chunk->ChatDelta
mapping, and the C struct mirrors (ABI gate now v3).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): ABI v5 - engine-side parser selection for 30 tool dialects + reasoning
Bump the vllm.cpp pin to the autoparser-parity engine: 30 tool-call dialects
(every pure-text parser in the pinned vLLM registry, each ported 1:1 with its
upstream tests), 7 reasoning parsers, google/minja as the template renderer
(the full Qwen3.5 template now renders engine-side), per-family structural
tags (tool_choice required/named compiles the model's NATIVE syntax where
expressible), and template auto-detection for both parser axes.
Backend changes:
- cModelParams mirrors ABI v5 (tool_parser + reasoning_parser fields,
layout-locked by the offset tests; ABI gate now v5).
- New model options tool_parser:<name> / reasoning_parser:<name> pass through
to the engine; unset means template auto-detection (18-row tool marker
table; [THINK]->mistral, <think>->think_auto for reasoning); "none"
disables the reasoning split; unknown names fail the first chat call.
- Chat chunks parse the `reasoning` field (the pin renamed
reasoning_content), flowing into ChatDelta.ReasoningContent which the host
already prefers.
Live e2e against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU, full suite green: the
real chat template renders (no more fallback), reasoning auto-detection picks
think_auto so markerless answers stay pure content (the live run caught the
deepseek_r1 content-swallow upstream and drove the think_auto fix), required
tool_choice returns schema-valid arguments, auto tool_choice engages
engine-side and streams parsed deltas, and blocking/streaming stay
byte-identical. Turn latency also dropped (proper template EOS behavior).
Upstream program landed as mudler/vllm.cpp 86013f3..5fffe7e (ABI v2-v5,
minja, parser waves B1/B2/B4, reasoning seam, structural-tag registry,
think_auto).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(vllm-cpp): bump the engine pin to the ENG-wave close-out
mudler/vllm.cpp@df8909b: the six engine-backed vLLM tool-parser families
(qwen3-coder/xml/mimo, kimi_k2, glm45/47, minimax_m2, gemma4, seed_oss)
text-reimplemented from their wire formats and held to the upstream test
suites - 39 registered dialects; the pinned vLLM registry is now covered
except the three Rust/Harmony-backed families, descoped by decision. kimi_k2
also gains a full native structural-tag builder; four new template
auto-detection rows land with test-pinned ordering.
Full backend e2e re-run green against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): add the vllm-cpp-development gallery meta
The gallery grew the twelve latest/development image entries but was missing
the separate vllm-cpp-development meta (own capabilities map targeting the
-development image names), which every backend ships so the development
gallery resolves per-platform. Validated: all capability targets in both
metas resolve to existing entries, and every image URI's tag suffix matches
a backend-matrix build.
Also full-stack verified in this change's context (single-node local-ai from
this branch, locally-built backend under --backends-path, Qwen3.5-2B GGUF):
/v1/chat/completions non-stream (clean content + usage), streaming (SSE
deltas), tool_choice auto engaging get_weather engine-side with schema-valid
arguments and finish_reason=tool_calls, and streamed tool-call deltas in the
standard name-first cadence.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): repair the CI backend builds - gcc-14 -Werror + fat-arch Triton
Two distinct failures took down all five vllm-cpp backend builds on the PR:
1. gcc-14 (ubuntu:24.04 CI images; the local toolchain is gcc-13) fails the
engine build with -Werror=maybe-uninitialized in InputBatch::condense - a
false positive through a staging std::optional's raw storage. Fixed
upstream (mudler/vllm.cpp@61f3e85) by moving slot-to-slot directly;
verified BOTH ways under dockerized g++-14.2 (unfixed reproduces CI's two
diagnostics exactly, fixed compiles clean) with the engine's behavior
suites green. Pin bumped to that sha.
2. The amd64 CUDA builds died at CMake configure: the vendored Triton-AOT
cubin trees are per-arch and the engine refuses -DVLLM_CPP_TRITON=ON on a
multi-arch (120a;121a) fat build unless pinned to one tree, which would be
unsound for the other arch. Triton is now enabled only on the single-arch
arm64/GB10 build (where the cubins matter); the fat amd64 binary uses the
engine's non-AOT GDN path.
Backend e2e re-run green at the new pin (Qwen3.5-2B on CPU, full suite).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): cuda-12 images cannot compile compute_121a - target 120a only
The second CI round surfaced a CUDA-version constraint: the cuda-12 (12.8)
image's nvcc rejects 'compute_121a' (GB10 arch support landed with CUDA 13),
killing the amd64 cuda-12 build at nvcc. Gate the architecture list on
CUDA_MAJOR_VERSION (exported by Dockerfile.golang): cuda-12 builds consumer
Blackwell 120a only, cuda-13 keeps the 120a;121a fat binary, arm64/l4t
(cuda-13) keeps single-arch 121a with the Triton cubins. GB10 is arm64, so
the amd64 cuda-12 image never served it - no capability change.
Verified by Makefile dry-run variable dumps for all three combinations
(cuda12 -> 120a; cuda13 -> 120a;121a; cpu -> CUDA off).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): drop the cuda-12 variant - the engine needs the CUDA 13 toolchain
Third CI round, third layer: with the arch list already narrowed to 120a,
the cuda-12 (12.8) build still dies in ptxas compiling the sm_120a NVFP4 MMA
kernels ("Vector type too large, exceeds 128 bit limit") - the Blackwell fp4
path genuinely requires the CUDA 13 toolchain, and vllm.cpp supports
Blackwell-family GPUs only. Shipping a cuda-12 image without the fp4 kernels
would be a crippled build of an engine whose whole GPU story is fp4, so the
variant is dropped instead:
- backend-matrix: cuda-12 vllm-cpp entry removed (cuda-13 amd64, l4t arm64,
cpu, vulkan, metal remain).
- gallery: cuda12 image entries removed; the nvidia capability now resolves
to the cuda13 image in both metas; the nvidia-cuda-12 key is dropped so
older-driver hosts fall back to the CPU image instead of an unrunnable one.
- backend Makefile: BUILD_TYPE=cublas under CUDA_MAJOR_VERSION=12 now fails
fast with a clear message; cuda-13 keeps the 120a;121a fat binary and
arm64/l4t keeps 121a with the Triton cubins.
Verified: Makefile branch dumps for all four combinations (cuda12 loud
error, cuda13 fat, arm64 121a+Triton, cpu off), YAML parses, matrix filter
tests green, gallery capability targets all resolve.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): forward multi-turn tool identity and reasoning to the engine
chatRequestJSON dropped Message.ToolCallId and Message.Name on role="tool"
replies and Message.ReasoningContent on assistant history, so a second
turn after tool execution reached the engine's chat template without the
fields that bind a tool result to the call it answers. Forward all three
(present-only, matching the OpenAI wire shape) and pin vllm.cpp to
6a0bd3e7, where ChatMessage parses/round-trips tool_calls, tool_call_id,
name and reasoning and the minja adapter exposes them to the template
context.
Adds the round-trip request-lowering spec (user -> assistant tool_call ->
tool reply -> lowered request) and re-ran the gated e2e suite against the
new engine pin with a real Qwen3.5 GGUF: chat, reasoning split, streaming
parity, required-tool and auto-tool cases all green.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): bump vllm.cpp for the darwin arm64 i8mm build fix
The darwin-metal CI job was the first build to compile the engine's arm
CPU-quant files on macOS and hit their Linux-only <asm/hwcap.h> /
<sys/auxv.h> includes. vllm.cpp 9e1c9025 detects i8mm per-OS (auxv on
Linux, sysctl on Apple Silicon) with kernels untouched. Gated e2e suite
re-run green against the new pin with a real Qwen3.5 GGUF.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): darwin build - bound cmake parallelism when nproc is absent
The macOS runners have no nproc, so JOBS evaluated empty and
`cmake --build -j$(JOBS)` became bare `-j`: unlimited clang jobs on a
3-core/7GB Mac, which swap-thrashed until the 6h GHA timeout (the log
shows "nproc: Command not found" and 7+ concurrent clang processes being
reaped at the cutoff). Use the same portable fallback chain as the other
darwin backends: nproc, then sysctl hw.ncpu, then 4.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add magpie-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the magpie-tts.cpp ggml port of NVIDIA's
Magpie TTS Multilingual 357M (encoder + autoregressive decoder over NanoCodec
tokens), producing 22.05 kHz mono audio in 5 baked voices (Aria, Jason, John,
Leo, Sofia; case-insensitive names or indices 0-4) across 9+ languages from a
single self-contained GGUF. Mirrors qwen3-tts-cpp / moss-tts-cpp: dlopen the
static-ggml shared library, bind the flat magpie_tts_capi_* C-API via purego
(no local C shim needed, the upstream .so exports it directly), and serve the
gRPC TTS + TTSStream methods behind base.SingleThread (the C context is not
reentrant across synthesize calls).
The backend CMakeLists translates the Makefile's -DGGML_{CUDA,METAL,VULKAN,HIP}
flags into upstream's MAGPIE_GGML_* toggles (upstream FORCE-overwrites the ggml
cache entries from those), pinned to magpie-tts.cpp v0.1.1
(e3f3dd1ebe22b64e7405f93b519f2d1930712568), which statically links ggml into
libmagpie-tts.so (ldd shows only system libs).
Wires the full registration: backend-matrix.yml (CPU amd64/arm64, CUDA 12/13,
Intel SYCL f16/f32, Vulkan amd64/arm64, ROCm, NVIDIA L4T + L4T CUDA 13, and
Darwin metal), backend/index.yaml metas and image entries, the root Makefile
build targets, the changed-backends backend-filter path mapping, the bump_deps
auto-bump matrix, a test-extra per-backend smoke job, the /backends/known
pref-only importer entry, the backend capabilities map (TTS + TTSStream, no
voice cloning), and the README / compatibility-table docs rows.
Verified locally: unit + e2e Ginkgo suites pass against the real q8_0 GGUF
(22.05 kHz mono WAV, RMS > 0.01), a live gRPC LoadModel + TTS round-trip
returns valid non-silent audio, and the pre-commit gates (make lint,
make test-coverage-check) pass, run manually with LOCALAI_TEST_HTTP_PORT
overriding the locally-occupied 9090.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* gallery: add magpie-tts-cpp model entries (q8_0 + f16)
Add the Magpie TTS Multilingual 357M GGUFs from mudler/magpie-tts.cpp-gguf to
the model gallery: q8_0 (~624 MB, near-lossless, fastest decode, recommended)
with an f16 (~784 MB) variant, both served by the magpie-tts-cpp backend.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* magpie-tts-cpp: bump pin to rewritten upstream v0.1.1 SHA
Upstream history was rewritten to purge accidentally committed build
artifacts; v0.1.1 now resolves to 6f7696cf.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The "Bump Backend dependencies" workflow has failed every night for over
ten days. Four upstreams — ced.cpp, moss-transcribe.cpp, voice-detect.cpp
and rf-detr.cpp — moved from the mudler org to localai-org, so the GitHub
API answers 301 for the old slugs. ced.cpp additionally renamed its
default branch to main.
bump_deps.sh fetched without -L or -f and never checked the response, so
the redirect's JSON body was passed straight to sed, which died with
"unterminated `s' command". The loud failure was luck: an error body
without slashes would have been substituted into the Makefile as the new
pin, silently corrupting the version and shipping it in a bump PR.
Point the matrix at the new slugs and branch, and harden the script so a
bad response can never reach sed: follow redirects, fail on HTTP errors,
and require a bare 40-hex SHA before rewriting anything. Also refresh the
now-stale repository URLs in the backend Makefiles, test scripts,
backend/index.yaml and the docs.
Verified all 25 matrix entries resolve to a commit SHA and that the four
previously-failing jobs run end to end against the real API.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(backend): add moss-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the moss-tts.cpp ggml port of the OpenMOSS
MOSS-TTS-Local v1.5 text-to-speech model (GPT-J local transformer decoded through
MOSS-Audio-Tokenizer-v2), producing 48 kHz stereo audio with optional
reference-audio voice cloning. Mirrors the qwen3-tts-cpp backend: dlopen the
static-ggml shared library, bind the moss-tts.cpp C-API via purego, and serve
the gRPC TTS method. A thin C shim holds the pipeline handle and copies engine
PCM into a Go-freeable buffer.
Wires the CI registration: backend-matrix.yml (CPU, CUDA 12/13, Intel SYCL
f16/f32, Vulkan, ROCm, NVIDIA L4T, plus Darwin metal), backend/index.yaml metas
and image entries pointing at mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF, the
root Makefile build targets, and the changed-backends.js path mapping.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: list the moss-tts-cpp backend among the LocalAI-maintained engines
Add moss-tts.cpp to the README "Backends built by us" table, the
Text-to-Speech compatibility table, and the reference-audio voice-cloning
backend list, so the new backend is documented alongside its peers.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): pin moss-tts.cpp to the squashed single-commit release
moss-tts.cpp history was collapsed to a single commit; repoint MOSSTTS_CPP_VERSION
to ee722b8e9205ee9b1b1c398a4e87e4e393e9be41.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): add the moss-tts-cpp-development gallery meta
The gallery had the -development image entries but no matching -development
meta anchor (as locate-anything-cpp and depth-anything-cpp have), so the master
build was not installable as a gallery backend. Add moss-tts-cpp-development
mirroring the production meta with the -development capability image names.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
C++/ggml transcription + speaker diarization + timestamps backend. Purego
dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp
and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output
into segments with nanosecond timestamps. Adds the importer (surfaces in
GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml,
and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf).
Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues)
passed against the real libmoss-transcribe.so. The pre-commit coverage gate
(full pkg/core + tests/e2e) could not run in the authoring sandbox (no live
models, port 9090 held); CI must enforce it before merge.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(voice-detect): add Go purego backend for voice-detect.cpp
Add backend/go/voice-detect implementing the Backend gRPC voice subset
(VoiceEmbed/VoiceVerify/VoiceAnalyze) over libvoicedetect.so via purego,
mirroring the parakeet-cpp / omnivoice-cpp backends.
The flat voicedetect_capi C ABI is dlopen'd cgo-less; malloc'd string and
float-vector returns are owned by Go and released through the matching capi
free functions, with the per-ctx last error surfaced into Go errors. Calls are
serialized via base.SingleThread since the C context is not reentrant.
Proto field mapping:
- VoiceEmbed: VoiceEmbedRequest.audio (path) -> embed_path -> Embedding+Model.
- VoiceVerify: audio1/audio2 + threshold (<=0 falls back to the
verify_threshold option, default 0.25) -> verify_paths -> verified/distance/
threshold/confidence/model/processing_time_ms.
- VoiceAnalyze: audio (path) -> analyze_path_json; the JSON age/gender/emotion
document maps to a single VoiceAnalysis segment (start/end 0; gender "label"
-> dominant_gender with the remaining float scores as the gender map; emotion
label/scores -> dominant_emotion/emotion).
The Makefile pins voice-detect.cpp to 47546430, clones+builds libvoicedetect.so
with ggml static-linked (PIC, GGML_NATIVE off) so dlopen needs no external
libggml/libvoicedetect; ldd on the artifact shows only system libs. Ginkgo
tests cover option parsing and analyze-JSON mapping; embed/verify smoke specs
gate on VOICEDETECT_BACKEND_TEST_MODEL + VOICEDETECT_BACKEND_TEST_WAV.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(voice-detect): wire backend into index, gallery and build
Register the voice-detect.cpp speaker-recognition + voice-analysis
backend (added in Voice-INT-A) into LocalAI's distribution surfaces,
mirroring the ced backend (the closest mudler C++/ggml audio analogue):
- backend/index.yaml: add the &voicedetect meta-backend (capabilities
platform map, no top-level uri) plus the full set of concrete per-arch
image entries (cpu/cuda12/cuda13/metal/rocm/sycl/vulkan/l4t and the
-development variants). Referential integrity audited - every alias
target resolves.
- gallery/index.yaml: add 5 model entries on backend voice-detect -
ECAPA-TDNN, WeSpeaker ResNet34, 3D-Speaker ERes2Net, CAM++ and the
wav2vec2 age/gender/emotion analyze model. The engine architecture is
read from GGUF metadata (voicedetect.arch) at load. GGUF artifacts are
not yet published: each files: entry points at the intended
mudler/voice-detect-gguf location with a TODO to fill sha256 after
upload (no fabricated hashes).
- .github/backend-matrix.yml: add the linux build matrix block + the
darwin metal entry mirroring ced.
- .github/workflows/bump_deps.yaml: track mudler/voice-detect.cpp via
VOICEDETECT_VERSION (pin 47546430, = 4754643).
- core/config/backend_capabilities.go: register voice-detect in the
backend capability map (VoiceVerify/VoiceEmbed/VoiceAnalyze ->
speaker_recognition), mirroring speaker-recognition.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(face-detect): add purego Go backend for face-detect.cpp
Add the LocalAI Go backend that dlopens libfacedetect.so (the flat
facedetect_capi_* C-ABI) via purego, mirroring the sibling voice-detect
backend. Implements the Face subset of the Backend gRPC service:
- Embeddings(PredictOptions): Images[0] base64 -> temp file -> embed_path
-> L2-normalized ArcFace embedding.
- Detect(DetectOptions): src -> detect_path_json -> Detection boxes
(class_name "face", [x1,y1,x2,y2] -> x/y/w/h).
- FaceVerify(FaceVerifyRequest): two images + threshold + anti_spoof ->
verify_paths; best-effort img areas via detect.
- FaceAnalyze(FaceAnalyzeRequest): img -> analyze_path_json -> per-face
age + gender ("M"/"F" normalized to "Man"/"Woman").
The Makefile pins face-detect.cpp to 636a1963 and builds the shared lib
with ggml + vendored libjpeg-turbo static (PIC), so the .so is
ldd-clean (no libggml) and exports only facedetect_capi_* (no jpeg_
symbols). Gated Ginkgo e2e mirrors voice-detect.
Note for the gallery-wiring task: backend registration (index.yaml,
gallery, core/config/backend_capabilities.go) is intentionally not
touched here.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(voice-detect): replace em dashes in net-new descriptions
Project style forbids em/en dashes. Replace the three U+2014 chars
introduced by the voice-detect gallery/index wiring with `-`/`:`.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(face-detect): wire backend into index, gallery and build
Register the face-detect.cpp face detection / embedding / verification /
analysis backend (added in Face-INT-A) into LocalAI's distribution
surfaces, mirroring the voice-detect wiring (the closest mudler C++/ggml
recognition analogue):
- backend/index.yaml: add the &facedetect meta-backend (capabilities
platform map, no top-level uri to avoid the meta-backend gotcha) plus
the full set of concrete per-arch image entries (cpu/cuda12/cuda13/
metal/rocm/sycl-f16/sycl-f32/vulkan/l4t and the -development variants),
22 entries. Referential integrity audited: every alias target resolves.
- gallery/index.yaml: add 4 model entries on backend face-detect -
face-detect-buffalo-l/m/s (insightface SCRFD + ArcFace/MBF, NON-COMMERCIAL)
and face-detect-yunet-sface (OpenCV-Zoo YuNet + SFace, APACHE-2.0, the
commercial-friendly alternative). The detector/embedder architecture is
read from GGUF metadata (facedetect.arch) at load; only the real
verify_threshold option is set (0.35 buffalo, 0.363 sface). GGUF
artifacts are not yet published: each files: entry points at the
intended mudler/face-detect-gguf location with a TODO to fill sha256
after upload (no fabricated hashes).
- core/config/backend_capabilities.go: register face-detect in the
backend capability map (Embedding/Detect/FaceVerify/FaceAnalyze ->
face_recognition), mirroring insightface.
- .github/backend-matrix.yml: add the linux build matrix block + the
darwin metal entry mirroring voice-detect.
- .github/workflows/bump_deps.yaml: track mudler/face-detect.cpp via
FACEDETECT_VERSION (pin 636a1963).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(recon): voice-detect metal build branch + face-detect gallery usecases
Add the missing metal BUILD_TYPE branch to the voice-detect Makefile
forwarding -DVOICEDETECT_GGML_METAL=ON, mirroring face-detect, so the
darwin metal CI artifact is built with the Metal backend instead of
CPU-only.
Expand the 4 face-detect gallery models' known_usecases to
[face_recognition, detection, embeddings] to match the backend
capabilities map and the mirrored insightface-buffalo entries, so
auto-selection for /v1/detect and /embeddings works.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* docs(recon): document voice-detect and face-detect ggml backends
Document the new standalone C++/ggml biometric backends as the
recommended/default option for face and voice recognition, keeping the
existing Python insightface / speaker-recognition backends framed as the
legacy path.
- features/face-recognition.md: add a face-detect (ggml) backend section
with the gallery entries (buffalo-l/m/s non-commercial, yunet-sface
Apache-2.0), licensing, and verify/detect/analyze quickstart.
- features/voice-recognition.md: add a voice-detect (ggml) backend
section with the gallery entries (ecapa-tdnn, wespeaker-resnet34,
eres2net, campplus speaker recognizers; emotion-wav2vec2 non-commercial
analyze head) and quickstart.
- reference/compatibility-table.md: add face-detect.cpp and
voice-detect.cpp rows to the Vision, Detection & Recognition table.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(gallery): publish recon backend GGUF uris + sha256
Fill in the published HuggingFace GGUF uris and verified sha256 for the
9 recon gallery entries (voice-detect-* and face-detect-*), and remove
the TODO publish markers. Correct the eres2net, campplus, and
emotion-wav2vec2 uris to the actual published filenames.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(gallery): re-embed buffalo anti-spoof + add audeering age/gender voice model
Update the 3 buffalo face-detect GGUF sha256 (anti-spoof ensemble now
embedded and re-uploaded under the same filenames/uris) and note the
FaceVerify anti_spoof request flag in each description. Add a new
voice-detect-age-gender-wav2vec2 gallery entry mirroring the emotion
model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(gallery): add face-detect-buffalo-sc and antelopev2 packs
Add gallery entries for two newly-published insightface face packs on
the face-detect backend: buffalo_sc (smallest pack, SCRFD-500M + small
ArcFace) and antelopev2 (higher-accuracy, SCRFD-10G + ArcFace glint360k
R100, 512-d). Both are non-commercial research-only.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(recon): honor LocalAI per-model threads in voice/face-detect backends
LocalAI spawns one backend process per model and serves requests
concurrently, so the engines' own min(hardware_concurrency, 8) default
can oversubscribe cores. Forward the per-model Threads value from the
gRPC LoadModel options into the engine via VOICEDETECT_THREADS /
FACEDETECT_THREADS (read at backend construction) before the capi load.
A non-positive Threads is treated as unset, leaving the engine default.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump backend pins to CPU-optimized engine commits
voice-detect.cpp -> 0d9c1b3 (radix-2 FFT FBank, threads, flash attn + cached
pos-conv); face-detect.cpp -> 523aee1 (thread-gated direct conv, threads).
Brings the CPU optimizations into the LocalAI backend builds. GGUF format and
parity unchanged, so the published HF GGUFs remain valid.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump backend pins to round-2 CPU-optimized engines
voice-detect.cpp -> fe7e6a3 (ERes2Net 1x1->mul_mat, CAM++ layout+context,
wav2vec2 conv-LN, ECAPA capture-drop, AVX512 dispatch opt-in); face-detect.cpp
-> 9c8adb7 (AVX2 Winograd F(2x2,3x3) for SCRFD/ArcFace 3x3 convs, ArcFace
BN-fold). Parity unchanged (cosine=1.0); GGUF format unchanged, HF GGUFs valid.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump backend pins to round-3 Winograd engines
voice-detect.cpp -> 45122ec (Winograd F(2x2,3x3) for WeSpeaker/ERes2Net 3x3
convs, -22%/-20% @8t); face-detect.cpp -> cd5c962 (Winograd F(4x4,3x3) for
SCRFD large maps, -22% @1t on top of F(2x2), more load-stable). Parity held
(cosine=1.0); GGUF format unchanged, HF GGUFs valid.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump backend pins to round-4 Winograd engines (CPU opt complete)
voice-detect.cpp -> d2839ca (CAM++ FCM 2D convs through Winograd, -15.5%/-10.3%);
face-detect.cpp -> c1db23d (AVX2-vectorized Winograd tile transforms, SCRFD
detect -14%/-9.6%). Final CPU optimization round; the conv-kernel lever class is
now exhausted (parity held cosine=1.0; GGUF/parity unchanged, HF GGUFs valid).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump face-detect pin to deep-kernel engine (7ae5c4d)
face-detect.cpp -> 7ae5c4d: register-blocked winograd-domain GEMM microkernel
(2.8x isolated GFLOP/s), AVX-512 zmm evolution behind runtime CPUID dispatch
(ship-safe, AVX2 fallback bit-identical), bias/relu fused into the winograd
output transform, and SFace Conv+BN fold + bias/PReLU fusion. SCRFD detect
~1.4x faster end-to-end vs the round-4 baseline; parity bit-exact; portable
single binary (function-multiversioned, no global -mavx512f).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump voice-detect pin to ECAPA operand-order win (e9c56ae)
voice-detect.cpp -> e9c56ae: weight-as-src0 mul_mat order in ECAPA's F32
conv1d_same (routes through tinyBLAS sgemm); ECAPA embed 1.67x @1t / ~1.3x @8t,
parity cosine=1.0. Isolated to encoder.cpp (ECAPA-only); ERes2Net/CAM++/WeSpeaker
do not call conv1d_same so are provably unaffected.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to FMA-throughput engines (voice f7b9f89, face 2d2d5f0)
face -> 2d2d5f0: route ArcFace 3x3 body convs through the AVX-512 winograd
microkernel (kWinoMinSize 80->14); ArcFace 1.62x @1t, SCRFD detect to 0.966 of
MLAS @1t, no regression. voice -> f7b9f89: runtime-CPUID-dispatched AVX-512
winograd-GEMM microkernel (ship-safe, AVX2 fallback bit-identical); WeSpeaker
1.90x @1t. Parity cosine=1.0 throughout; portable single binaries.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to MLAS-class direct-conv engines (voice 7ecfd07, face be22d67)
Hand-tuned nChw16c AVX-512 register-tiled direct-conv microkernel (~263 GFLOP/s,
within 6-7% of MLAS per-op efficiency), runtime-CPUID-dispatched + AVX2 fallback,
fused bias/relu. voice 7ecfd07: default 3x3-s1 kernel for WeSpeaker (+37%/+32%)
+ ERes2Net, CAM++ pinned to Winograd. face be22d67: shape-gated to the ArcFace
recognizer body (+25-27% @8t); SCRFD detector stays on Winograd (no regression).
Parity cosine=1.0 / detect <=1px on AVX-512 + AVX2 paths. Portable single binaries.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump voice pin to Phase-A blocked backbone (f4e7eef)
WeSpeaker ResNet34 runs as one nChw16c blocked island (2 reorders/forward vs
~60) on AVX-512, default; per-conv directconv fallback on AVX2. +2.9% @1t /
+17-19% @8t vs per-conv directconv, parity cosine=1.0. The conv microkernel is
already FMA-bound near peak (~0.86-0.98x MLAS-implied); residual to MLAS is
sub-peak edge + non-conv tail, documented in docs/cpu-optimization.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to breadth blocked-backbone (voice 7f66871, face d80092b)
voice 7f66871: AVX2-vectorized (ymm) blocked island - AVX2-only hosts now run
the blocked backbone for WeSpeaker (2.3x over per-conv-AVX2, cosine=1.0);
ERes2Net stays per-conv (blocked regresses, opt-in only); CAM++ Winograd-pinned.
face d80092b: ArcFace recognizer blocked island, AVX-512 default (-13% @8t, ~0.90x
MLAS, the closest conv result), auto per-conv on AVX2; SCRFD untouched on Winograd
(0 island invocations during detect). Parity cosine=1.0 / detect <=1px throughout.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to small-spatial + stem conv kernels (voice 99b1804, face 47fdab6)
Measured-gap-driven conv kernels: small-spatial (fill the register tile when
output width <= tile width) + small-IC stem + strided-1x1/downsample recovery.
ArcFace recognizer 0.57 -> 0.70x MLAS @1t (the closest conv model), WeSpeaker
0.65 -> 0.79x @1t. Parity cosine=1.0 / detect <=1px. The OC-block-sharing lever
was a measured dead-end (deep stride-1 is L3-weight-bandwidth bound, not
read-port bound) and was NOT shipped. Kernel ceiling reached; further gap needs
an algorithm-class change (cache-blocked weight-stationary GEMM, or q8 weights).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to GPU persistent-graph + multi-model-safe cache (voice 45d2e6b, face 0a4799a)
GPU wins (CUDA/ggml backend, no CPU-path change): persistent per-shape graph+context
cache in Backend::compute() eliminates the per-call cudaGraph re-instantiation churn
-> wav2vec2 emotion+age-gender now AT GPU parity with torch-cuDNN on GB10 (0.97-0.98x),
CAM++ -5.7ms; bit-identical parity. Cache hardened multi-model-safe (invalidate-on-free
keyed by the ModelLoader weights buffer) so LocalAI multi-model hosting cannot stale-hit.
Conv models still trail cuDNN (im2col-materialization-bound) - cuDNN implicit-GEMM lever next.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to cuDNN-conv-capable engines (voice b6e4356, face 6107a24)
Adds the opt-in cuDNN implicit-GEMM conv path (VOICEDETECT_GGML_CUDNN /
FACEDETECT_GGML_CUDNN, DEFAULT OFF -> zero build/runtime dep until enabled).
On GPU it kills the im2col-materialization bottleneck and reaches torch-cuDNN
parity on the spill-bound convs: SCRFD detect 14.8->6.4ms (2.3x, ~parity),
WeSpeaker ~parity, ERes2Net beats torch (1.10x); ArcFace/CAM++ neutral (no
spill). Parity exact (SCRFD <=1px, cosine=1.0). To USE it in LocalAI, the CUDA
backend build must enable the flag AND bundle libcudnn - deferred until a
cuDNN-bundled GPU image; flag stays OFF here.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(recon): enable cuDNN conv path on arm64+CUDA13 recon backends
The voice-detect.cpp / face-detect.cpp engines have an opt-in cuDNN
implicit-GEMM conv path behind VOICEDETECT_GGML_CUDNN / FACEDETECT_GGML_CUDNN
(default OFF) that kills im2col on the GPU and reaches torch-cuDNN parity
(SCRFD 2.3x, WeSpeaker/ERes2Net parity), measured on the GB10
(arm64, CUDA 13, sm_121a).
Enable it for the CUDA build, but only where cuDNN actually ships: the
arm64 + CUDA 13 image (GB10/Jetson/L4T). x86 CUDA images carry no cuDNN,
so flipping it on globally for BUILD_TYPE=cublas would be a link failure.
The Makefiles gate on CUDA_MAJOR_VERSION=13 + arch (TARGETARCH from the
matrix/Docker build, uname -m fallback for local builds).
backend/Dockerfile.golang already installs the runtime libcudnn9-cuda-13
in the arm64+CUDA13 apt block; add the matching libcudnn9-dev-cuda-13 so
the build-time link resolves.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump voice-detect pin to ERes2Net blocked-default (30beecd)
Defaults VD_ERES2NET_BLOCKED ON: routes the ERes2Net Res2Net body through the
blocked nChw16c AVX-512 directconv island instead of the 1x1 mul_mat fast path
(CONT-transpose + skinny low-K GEMM). On the shipped GGML_NATIVE=OFF build (ggml
mul_mat is AVX2-only) this wins ~2x at every thread count (2.07x@1t, 2.2x@4t,
2.05x@8t); pure-AVX2 fallback still 1.3-1.62x. Parity exact (cosine=1.000000 vs
golden), so registered voices + verify/identify thresholds are unaffected. The
prior default-OFF rested on a stale comment whose 23pct regression only held on
the non-shipping GGML_NATIVE=ON build.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* docs(readme): announce native voice-detect + face-detect backends in Latest News
Add a Latest News entry for the new from-scratch C++/ggml biometric backends
(voice-detect.cpp + face-detect.cpp) that replace the Python insightface and
speaker-recognition backends: no Python/onnxruntime at inference, self-contained
GGUF, bit-exact parity, GPU cuDNN parity. Mirrors the parakeet.cpp /
locate-anything.cpp native-backend news entries. Refs PR #10441.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): re-pin to the squashed engine release commits
The voice-detect.cpp and face-detect.cpp histories were squashed to a single
release commit, which orphaned the previous pins (voice 30beecd, face 6107a24).
Re-pin to the new single-commit SHAs (voice 3d51077, face 06914b0); the tree is
identical, so the backend build is unchanged.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ced): sketch sound-classification backend (CED audio tagger)
Wires ced.cpp (CED, 527-class AudioSet sound-event tagger; baby cry,
footsteps, glass, alarms, dog bark) into LocalAI as a Go/purego backend.
SKETCH (backend skeleton real; core REST wiring + CI/gallery is a checklist
in DESIGN.md):
- backend/backend.proto: new SoundDetection rpc + SoundClass messages
(run `make protogen-go` to regenerate pkg/grpc/proto).
- backend/go/ced: main.go (purego dlopen libced.so + ced_capi.h),
goced.go (Ced gRPC backend: Load + SoundDetection), Makefile
(clone-at-pin CED_VERSION, ggml static-PIC shared build), run.sh,
package.sh, .gitignore.
- DESIGN.md: REST /v1/audio/classification wiring (handler/route/capability
registration checklist), gallery/index + CI registration, and a scoping
note for the realtime/websocket live-recognition path (sliding-window
classify over the existing ws transport + voicegate; the ced C-API
per-PCM entry point is already window-friendly).
Backend code does not compile until protogen-go regenerates the pb types
and a libced.so is built (Makefile clones+builds it).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ced): REST /v1/audio/classification endpoint + capability registration
Wires the ced sound-event classification backend (AudioSet audio tagger)
end to end through the REST surface, mirroring the transcription path.
- Handler: core/http/endpoints/openai/sound_classification.go parses the
multipart audio upload, temp-files it, resolves the model config and
calls the SoundDetection RPC; returns {model, detections[]} JSON.
- Backend wrapper: core/backend/sound_classification.go (ModelSoundDetection)
loads the model and normalizes the proto response into schema types.
- Schema: core/schema/sound_classification.go (SoundClassificationResult).
- gRPC layer: SoundDetection wired through the LocalAI wrapper (interface,
Backend client, Client, embed, server, base default) so the loader-typed
client exposes the RPC; proto regenerated via make protogen-go.
- Route: POST /v1/audio/classification (+ /audio/classification alias) with
the audio/multipart default-model middleware in routes/openai.go.
- Capability surfaces: swagger @Tags/@Router on the handler; FLAG_SOUND_
CLASSIFICATION usecase flag + UsecaseSoundClassification + UsecaseInfoMap +
GuessUsecases + ModalityGroups + GetAllModelConfigUsecases; meta usecase
option; /api/instructions audio area updated; auth RouteFeatureRegistry +
FeatureAudioClassification (APIFeatures, default ON) + FeatureMetas; UI
usecaseFilters, capabilities.js CAP_SOUND_CLASSIFICATION, Models.jsx filter
+ i18n; docs page features/audio-classification.md + whats-new + crosslink.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ced): realtime sound-event detection over the websocket API
When a realtime pipeline configures a sound-classification model, each
VAD-committed utterance (the same window the transcription path produces)
is also run through the CED sound-event classifier and the scored AudioSet
tags are emitted as a new server event. No new backend rpc is needed: the
SoundDetection gRPC method already exists on this branch.
- config: add Pipeline.SoundDetection (yaml/json sound_detection,omitempty)
beside Transcription/VAD.
- realtime: add Model.SoundDetection(ctx, audio, topK, threshold) to the
ModelInterface; implement it on wrappedModel and transcriptOnlyModel by
calling backend.ModelSoundDetection with the session's sound-classification
model config (mirrors how Transcribe dispatches). Load the optional config
in newModel / newTranscriptionOnlyModel; nil config keeps it additive.
- types: add ConversationItemSoundDetectionEvent (item_id, content_index,
detections[]{label,score,index}) with type conversation.item.sound_detection,
its ServerEventType constant and MarshalJSON, mirroring the transcription
completed event.
- realtime: add emitSoundDetection (unary path: classify the committed window,
build the event, t.SendEvent) and wire it at the utterance-commit hook right
after emitTranscription; gated on session.SoundDetectionEnabled (resolved
from Pipeline.SoundDetection at session setup, defaults top_k=5, threshold=0).
Its error is logged via xlog but never aborts the turn.
- test: Ginkgo specs for emitSoundDetection (tags emitted, empty detections,
classifier error) plus a SoundDetection method on the fakeModel double.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ced): implement SoundDetection in nodes backend test doubles
The SoundDetection method added to the grpc backend interface left two
test doubles (fakeBackendClient, fakeGRPCBackend) incomplete, so
core/services/nodes failed to compile under `go vet`/`go test` (go build
missed it: the doubles live in _test.go). Add the method to both,
mirroring their existing Detect mock. Repairs CI for the nodes package.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ced): decouple realtime sound detection from VAD (sound-only sessions)
Sound-event detection must activate on sounds, not speech, so it no longer
runs through the voice VAD/transcription path. A sound-detection-only
pipeline (sound_detection set, no transcription/LLM) now:
- is accepted by prepareRealtimeConfig (sound_detection counts as a pipeline
stage),
- builds a lightweight model via newSoundDetectionOnlyModel (no VAD/STT/LLM/TTS
loaded), and
- defaults the session to turn_detection none (no VAD) with no transcription
stage, so the client drives windowing via input_audio_buffer.commit
(option A: client-side sliding window). The per-PCM C-API already supports
arbitrary windows.
commitUtterance gains a sound-only branch: it emits the
conversation.item.sound_detection event (scored AudioSet tags) and stops -
no transcription, no LLM response. generateResponse is now guarded on a
transcription stage being present, so a sound-only turn never invokes the LLM.
Existing transcription/VAD sessions are unchanged (additive). Added a
commitUtterance sound-only Ginkgo spec asserting it emits the sound event and
neither transcribes nor generates a response. go vet + golangci-lint
(new-from-merge-base) clean; openai suite green.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ced): register sound-classification backend in gallery + CI
Mechanical backend-image registration for the ced sound-event classifier,
mirroring the parakeet-cpp Go/purego backend everywhere it is wired up.
- .github/backend-matrix.yml: add the ced build matrix, field-for-field copies
of the parakeet-cpp entries (cpu amd64/arm64, cublas cuda 12/13 amd64,
l4t cuda-13 arm64, l4t-jetpack cuda-12 arm64, sycl f32/f16, vulkan
amd64/arm64, rocm hipblas, and the metal darwin entry), changing only
backend and tag-suffix. dockerfile stays ./backend/Dockerfile.golang.
- backend/index.yaml: add the &ced meta anchor (capabilities map per platform)
plus ced-development and the per-arch image entries, each uri/mirror
tag-suffix matching the matrix exactly. The model gallery (GGUF) entry is
intentionally deferred pending the HuggingFace publish (TODO note inline).
- scripts/changed-backends.js: add an explicit item.backend === "ced" branch in
inferBackendPath mapping to backend/go/ced/, same mechanism and ordering as
the parakeet-cpp branch (before the generic golang fallthrough).
- .github/workflows/bump_deps.yaml: register mudler/ced.cpp -> CED_VERSION in
backend/go/ced/Makefile so the daily bot bumps the pin.
- swagger/{docs.go,swagger.json,swagger.yaml}: regenerated via make swagger so
the existing /v1/audio/classification annotations land in the generated spec.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ced): server-side windowing for realtime sound detection (option B)
Adds an optional server-driven sliding-window classifier so a sound-only
realtime client only has to stream audio (no input_audio_buffer.commit):
- Pipeline.sound_detection_window_ms / sound_detection_hop_ms config knobs.
When both > 0 on a sound-only session, the server classifies the last
window of streamed audio every hop and emits a conversation.item.sound_
detection event; the input buffer is trimmed to one window so a long
stream stays bounded. When unset, the session stays client-driven
(option A). Runs independent of VAD (sound events are not speech).
- handleSoundWindow (ticker) + classifySoundWindow (one tick, extracted so
it is unit-testable) + writeWindowWAV, which declares the true
InputSampleRate (NewWAVHeaderWithRate) so the classifier resamples
correctly. Goroutine is started after toggleVAD and torn down with the
session (close + wg.Wait).
- Register pipeline.sound_detection (+window_ms/hop_ms) in the config meta
registry; the earlier realtime commit added pipeline.sound_detection
without a registry entry, failing TestAllFieldsHaveRegistryEntries. This
fixes that and covers the two new knobs.
Tests: classifySoundWindow emits an event + trims the buffer to one window,
no-ops on too-little audio; writeWindowWAV declares the given sample rate.
go build/vet + golangci-lint (new-from-merge-base) clean; config + openai
suites green.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ced): add ced-base GGUF model gallery entries (f16 + q8_0)
The ced-base weights are now published at mudler/ced-base-gguf (Apache-2.0,
converted from mispeech/ced-base). Adds gallery/ced.yaml (backend: ced +
known_usecases: sound_classification) and two gallery/index.yaml entries
(ced-base-f16 default, ced-base-q8 smallest) with sha256-pinned files, and
removes the now-resolved TODO from backend/index.yaml.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ced): add tiny/mini/small GGUF model gallery entries
Publishes the rest of the CED family (same architecture, metadata-driven port
verified end-to-end on ced-tiny) to mudler/ced-{tiny,mini,small}-gguf and adds
their f16 + q8_0 gallery entries:
ced-tiny (5.5M, edge/Pi-class) f16 11MB / q8_0 6MB
ced-mini (9.6M) f16 19MB / q8_0 11MB
ced-small (22M) f16 42MB / q8_0 23MB
All sha256-pinned. ced-base remains the accuracy default.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(ced): point gallery entries at the consolidated mudler/ced-gguf repo
All CED quantizations (tiny/mini/small/base, f16/q8_0) now live in a single
HuggingFace repo, mudler/ced-gguf, instead of per-model repos. Repoint the 8
gallery model entries' urls + file uris accordingly. sha256 and filenames are
unchanged.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(ced): bump CED_VERSION to the short-clip fix
Pin the ced backend to ced.cpp 99c6ed3, which fixes a crash on any clip
shorter than target_length (~10.11s): time_pos_embed was added at its full
63-frame grid instead of being sliced to the clip's actual time grid, tripping
ggml_can_repeat in ggml_add. Surfaced by the live realtime e2e (sub-10s
windows) and gated with a short-clip parity test upstream.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(ced): list ced.cpp as a LocalAI-team engine + backend-guide directive
- README.md: add ced.cpp to the "native C/C++/GGML engines developed and
maintained by the LocalAI project" table.
- docs/content/features/backends.md: add a Sound Classification backend
category (sound-event classification / audio tagging) listing ced.cpp.
- .agents/adding-backends.md: add a "Documenting the backend" section and two
verification-checklist items requiring new backends to be documented in the
backends.md category list, and in-house native engines to be added to the
README maintained-engines table. This directive was missing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(ced): repin CED_VERSION to the v0.1.0 release commit
ced.cpp history was squashed into a single release commit (tagged v0.1.0), so
the previous pin (99c6ed3) no longer exists upstream. Pin to c04ac14, the
v0.1.0 release commit, so the backend builds against a commit that exists.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ced): silence gosec G304/G103 + govet unsafeptr on audited paths
- sound_classification.go: os.Create(dst) where dst = temp dir + path.Base of
the upload (no traversal). #nosec G304, matching the depth-anything-cpp handler.
- goced.go: reading a NUL-terminated C string from a libced-owned buffer.
#nosec G103 (gosec) + //nolint:govet (golangci-lint's unsafeptr check), since
the uintptr is a C-owned malloc'd buffer, not Go-GC memory.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add apex-quant (MoE per-tensor/per-layer quantization recipe) to the
"Backends built by us" section as a note after the engines table, since
it is a quantization recipe rather than a native inference engine.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
docs: document the privacy-filter.cpp backend in README and compatibility table
The privacy-filter.cpp backend (#10360) was registered in backend/index.yaml
and referenced from the PII feature docs, but was missing from the backend
catalog surfaces. Add it to the README "Backends built by us" table, the
compatibility table (Utilities & Other, CPU/CUDA 13/Vulkan), and the backend
type list in the backends feature doc.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Bring the Backend & Model Compatibility Table up to the full set of
backends published in backend/index.yaml (60+), organized by modality
with per-backend acceleration targets. Add an "Available Backends"
pointer and expand the backend-type list in the backends feature doc.
Update the README backend count to 60+ and add a "Backends built by us"
section listing the native C/C++/GGML engines maintained by the LocalAI
project (parakeet.cpp, voxtral.c, vibevoice.cpp, rf-detr.cpp,
locate-anything.cpp, depth-anything.cpp, LocalVQE, local-store).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the localai-org/localai-realtime-demo Go client to the README
Examples list and to the realtime docs (integrations + realtime feature
page). Refresh the Latest News section with June 2026 highlights pulled
from history since v4.3.0: realtime pipeline streaming, the parakeet.cpp
and CrispASR speech work, new backends (locate-anything.cpp, Ideogram4,
llama.cpp video input), and distributed-mode hardening.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Add an opt-in `local-ai chat` command for testing chat models directly from the terminal without manually sending curl requests.
The command connects to a running LocalAI server, lists available models through the existing OpenAI-compatible API, streams chat completions, and supports interactive commands such as `/models`, `/model`, `/clear`, and `/exit`.
Keep `local-ai run` focused on the server lifecycle so the web UI, API clients, and multiple chat terminals can coexist against the same server.
Document the new command and terminal workflow in the README and CLI docs.
Tests:
- go test -count=1 ./core/cli/chat
- go test -count=1 ./core/cli
Assisted-by: Codex:GPT-5
Signed-off-by: Ching Kao <0980124jim@gmail.com>
* docs: add 'how LocalAI works' architecture diagram
Add a blueprint-style architecture diagram: clients -> small core (API,
router, WebUI, agents) -> gRPC -> backend processes pulled on demand as
OCI images. Place it on the overview page and replace the stale external
architecture image on the reference page.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: add blueprint diagrams across feature, distributed & getting-started docs
Add 24 architecture/flow/comparison diagrams (PNG + HTML source) under
docs/static/images/diagrams/, wired into their docs pages, from an
impact-vs-effort audit of the docs. Broaden the API surface on the
overview architecture diagram (OpenAI, Anthropic, ElevenLabs, Ollama,
and LocalAI's own API) and move the gRPC boundary label clear of the arrows.
Pages: distributed mode (architecture, scheduling, ds4 layer-split),
distributed inferencing, MLX, realtime, quantization, MCP, agents,
mitm & cloud proxy, middleware, reverse-proxy TLS, VRAM, voice & face
recognition, reranker, function calling, fine-tuning (recipe + jobs),
diarization, audio transform, quickstart, model resolution.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: add composable-core diagram to README hero
Commit the composable-core card (small core + on-demand backend tiles)
alongside the other diagrams and reference it from the README hero via a
repo-relative path, so it renders on GitHub.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: fix composable-core connectors/badge and federated-vs-worker layout
- composable-core: thicken the plug-in connectors so they read clearly, and
widen the SEPARATE IMAGE badge so its text no longer overflows the box.
- federated-vs-worker: shorten the WHOLE/SPLIT REQUEST pills to fit, and
replace the tangled node-to-node activation arrows with a clean fan-out
(request split across all sharded nodes), mirroring the federated panel.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Reframe the README hero and docs (homepage, overview, FAQ) around the
composable architecture: a small core, with backends built as dedicated
gRPC services around best-in-class engines, shipped as separate OCI
images and pulled on demand. Lead from strength: drop the "36+ backends"
kitchen-sink framing and the "All-in-One Complete AI Stack" / "single
binary that gives you everything" lines that read as a monolith.
- README: small-core differentiator; composable + open/extensible bullets
- _index.md: composable tagline; install only what you use
- overview.md: core vs on-demand backends; gRPC/OCI mechanics as benefits;
bring-your-own model and backend
- faq.md: "Do I need to install all the backends?" and
"Can I bring my own model or backend?"
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Walk down the release history and add per-release one-liners for 4.3.0,
4.2.0, 4.1.0, and 4.0.0 in the Latest News section, leading with the
headline win for each release. Move Prem into a collapsible "Past
sponsors" block under the active sponsors row.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.7 [claude-code]
Update README and docs to attribute maintenance to the LocalAI team
(Ettore Di Giacinto and Richard Palethorpe) and drop the autonomous
AI dev team section.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude-code:claude-opus-4-7 [Edit] [Bash]
* feat(ui): Add dynamic model editor with autocomplete
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(docs): Add link to longformat installation video
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Updated README to include a guided tour section with links to various assets and details about agents and usage metrics.
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
* feat: add distributed mode (experimental)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix data races, mutexes, transactions
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix events and tool stream in agent chat
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* use ginkgo
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(cron): compute correctly time boundaries avoiding re-triggering
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* enhancements, refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* do not flood of healthy checks
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* do not list obvious backends as text backends
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* tests fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Drop redundant healthcheck
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* enhancements, refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
AIO images are behind, and takes effort to maintain these. Wizard and
installation of models have been semplified massively, so AIO images
lost their purpose.
This allows us to be more laser focused on main images and reliefes
stress from CI.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(mlx-distributed): add new MLX-distributed backend
Add new MLX distributed backend with support for both TCP and RDMA for
model sharding.
This implementation ties in the discovery implementation already in
place, and re-uses the same P2P mechanism for the TCP MLX-distributed
inferencing.
The Auto-parallel implementation is inspired by Exo's
ones (who have been added to acknowledgement for the great work!)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* expose a CLI to facilitate backend starting
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat: make manual rank0 configurable via model configs
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add missing features from mlx backend
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Apply suggestion from @mudler
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
docs: make examples repository link more prominent in README
Add a badge-style button link to the examples repository in the main
README and expand examples/README.md with example categories and
quick-start links to help new users discover the examples repo.
Co-authored-by: localai-bot <localai-bot@noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* docs: Update model compatibility documentation with missing backends
Added the following backends to README.md and compatibility-table.md:
- vllm-omni: Multimodal vLLM with vision and audio support
- nemo: NVIDIA NeMo framework for speech models
- outetts: OuteTTS with voice cloning capabilities
- faster-qwen3-tts: Faster Qwen3 TTS implementation
- qwen-asr: Qwen automatic speech recognition
- voxcpm: VoxCPM speech understanding model
- whisperx: Enhanced Whisper with word-level transcription
These backends exist in the codebase (backend/index.yaml) but were missing
from the documentation. This update ensures accurate reflection of currently
supported backends in LocalAI.
* Apply suggestion from @mudler
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
---------
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
Co-authored-by: localai-bot <localai-bot@example.com>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
docs: fix README screenshot references and clean up alt text
The Talk Interface was incorrectly using screenshot_tts.png (same as
Generate Audio) instead of screenshot_talk.png. Also standardized
alt text across all screenshot references for consistency.
Signed-off-by: localai-bot <localai-bot@noreply.github.com>
Co-authored-by: localai-bot <localai-bot@noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
feat: add MIT license badge to README.md header
- Add Shields.io license badge showing 'License: MIT'
- Place badge in header section with other badges
- Link badge to LICENSE file
- Follows existing badge format (for-the-badge style)
Co-authored-by: localai-bot <localai-bot@noreply.github.com>
docs: add Table of Contents to README.md for easier navigation
- Add collapsible TOC with anchor links to all major sections
- Include H2 sections and important H3 subsections
- Place TOC after main description, before Local Stack Family
- Use proper markdown anchor link format
Signed-off-by: localai-bot <localai-bot@noreply.github.com>
Co-authored-by: localai-bot <localai-bot@noreply.github.com>
* feat: add standalone and agentic functionalities
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* expose agents via responses api
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: add autonomous development team section to README
- Add blog post link to Media, Blogs, Social section
- Add new section about autonomous AI agent maintenance team
- Include links to reports.localai.io and project board
- Reference the experiment blog post
* Apply suggestion from @mudler
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
---------
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
Co-authored-by: team-coding-agent-1 <team-coding-agent-1@localai.dev>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
* feat(musicgen): add ace-step and UI interface
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Correctly handle model dir
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Drop auto-download
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add to models, fixup UIs icons
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Update docs
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* l4t13 is incompatbile
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* avoid pinning version for cuda12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Drop l4t12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): Add audio conversations
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(realtime): Vendor the updated API and modify for server side
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(realtime): Update to the GA realtime API
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore: Document realtime API and add docs to AGENTS.md
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat: Filter reasoning from spoken output
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(realtime): Send delta and done events for tool calls and audio transcripts
Ensure that content is sent in both deltas and done events for function call arguments and audio transcripts. This fixes compatibility with clients that rely on delta events for parsing.
💘 Generated with Crush
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(realtime): Improve tool call handling and error reporting
- Refactor Model interface to accept []types.ToolUnion and *types.ToolChoiceUnion
instead of JSON strings, eliminating unnecessary marshal/unmarshal cycles
- Fix Parameters field handling: support both map[string]any and JSON string formats
- Add PredictConfig() method to Model interface for accessing model configuration
- Add comprehensive debug logging for tool call parsing and function config
- Add missing return statement after prediction error (critical bug fix)
- Add warning logs for NoAction function argument parsing failures
- Improve error visibility throughout generateResponse function
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.5 via Crush <crush@charm.land>
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
exllama2 development has stalled and only old architectures are
supported. exllamav3 is still in development, meanwhile cleaning up
exllama2 from the gallery.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
