mirror of
https://github.com/mudler/LocalAI.git
synced 2026-08-02 11:30:44 -04:00
fix/vllm-cpp-l4t-cuda12-fallback
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mudler's LocalAI [bot]
|
4baa36ddd8 |
feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100)
* feat(backend): add vllm-cpp text-generation backend (vllm.cpp) Wrap https://github.com/mudler/vllm.cpp - the LocalAI-team from-scratch C++20 port of vLLM (paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, no Python at inference) - as a Go gRPC backend over its stable C ABI (ABI v2) via purego. Backend (backend/go/vllm-cpp): - Load -> vllm_engine_load: accepts a .gguf file or a config.json model dir (anything else is refused, satisfying the greedy-probe rule); context_size maps to max_model_len, options block_size/num_blocks/max_num_seqs size the KV cache and scheduler admission. - Predict -> vllm_complete (blocking); PredictStream -> vllm_complete_stream with the per-delta C callback bridged into the gRPC stream. The backend embeds base.Base (not SingleThread): concurrent requests batch continuously in the engine's shared AsyncLLM scheduler. - PredictOptions.Grammar -> the ABI's structured_grammar (GBNF), giving grammar-constrained tool calling at parity with llama-cpp; the ABI also exposes JSON-schema/regex/choice constraints. - Hand-mirrored POD structs with layout locked by unit tests (unsafe.Offsetof vs the C offsets) and a runtime vllm_abi_version gate. - One portable library per platform (vllm.cpp uses per-file SIMD tiers with runtime dispatch), so no avx/avx2/avx512 variant builds. Wiring: - backend-matrix: CPU amd64+arm64 (per-arch + manifest merge), CUDA 12/13 amd64 (120a;121a Blackwell fat binary), L4T arm64 (121a, GB10/DGX Spark - the runtime-proven GPU target), Vulkan amd64, and Darwin arm64 Metal. - backend/index.yaml meta + 12 image entries (latest/development x cpu, cuda12, cuda13, l4t, vulkan, metal); bump_deps registration for the VLLM_CPP_VERSION pin; root Makefile registration; test-extra runs the unit specs (pure Go, no engine build). - Importers: preference-only swaps - llama-cpp (GGUF) and vllm (safetensors) advertise vllm-cpp via AdditionalBackends and emit backend: vllm-cpp without tokenizer templating (the C ABI takes the FINAL prompt; templating and tool parsing stay LocalAI-side). No auto-detect importer. - Docs: backends list, top-level README maintained-engines table, compatibility table. Verified: 20/20 Ginkgo specs against the real pinned engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU - blocking + streaming parity, greedy determinism, stop words, GBNF-constrained generation, and 4 concurrent streams; plus a dlopen/ABI-gate smoke of the built gRPC server binary. Upstream ABI v2 + production structured-output wiring landed as mudler/vllm.cpp@86013f3. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vllm-cpp): ride the autoparser code path - engine-side chat templating and tool engagement (ABI v3) The backend now implements AIModelRich (PredictRich / PredictStreamRich) over vllm.cpp's ABI v3 chat entry points, so chat and tool calling ride the SAME code path as the llama.cpp autoparser: the ENGINE renders the model's chat template, decides when a tool call engages, and parses it - LocalAI receives pre-parsed ChatDelta / ToolCallDelta protos exactly as it does from llama-cpp. - With use_tokenizer_template + structured Messages, PredictOptions lowers to ONE OpenAI chat request JSON (messages, tools, tool_choice, sampling, stream_options.include_usage) for vllm_chat / vllm_chat_stream. tool_choice auto lowers engine-side to a LAZY structural-tag decode constraint - free text until the model emits the tool trigger, then the call is grammar-constrained; required/named force a call. Tool output is parsed by the engine's streaming Hermes-style parser; each chat.completion.chunk maps onto ChatDeltas (content / reasoning_content / tool_calls) which the host already prefers over Go-side tag extraction. Without structured messages the plain path (LocalAI templating + optional GBNF grammar) applies unchanged. - The engine resolves the chat template from the GGUF tokenizer.chat_template metadata (or tokenizer_config.json); templates beyond its minja subset - e.g. the full Qwen3.5 namespace()/macro template - degrade engine-side to a Hermes-aware fallback prompt (tools schemas + <tool_call> instruction) with a stderr witness, so structural-tag engagement keeps working. - Importers now emit the same config shape as llama-cpp for vllm-cpp (use_tokenizer_template: true, no-grammar autoparser flow); only the llama-cpp-specific use_jinja option and the vllm-python parser options are dropped. - Pin bumped to mudler/vllm.cpp@aaed7ec (ABI v3 + chat-prompt resolution). Verified against the real engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU: full suite green - blocking chat, streaming deltas concatenating byte-equal to the blocking answer, a REQUIRED tool call returning schema-valid arguments JSON, and an AUTO run where the engine itself engages get_weather and streams parsed tool deltas; plus unit specs for the request lowering, chunk->ChatDelta mapping, and the C struct mirrors (ABI gate now v3). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vllm-cpp): ABI v5 - engine-side parser selection for 30 tool dialects + reasoning Bump the vllm.cpp pin to the autoparser-parity engine: 30 tool-call dialects (every pure-text parser in the pinned vLLM registry, each ported 1:1 with its upstream tests), 7 reasoning parsers, google/minja as the template renderer (the full Qwen3.5 template now renders engine-side), per-family structural tags (tool_choice required/named compiles the model's NATIVE syntax where expressible), and template auto-detection for both parser axes. Backend changes: - cModelParams mirrors ABI v5 (tool_parser + reasoning_parser fields, layout-locked by the offset tests; ABI gate now v5). - New model options tool_parser:<name> / reasoning_parser:<name> pass through to the engine; unset means template auto-detection (18-row tool marker table; [THINK]->mistral, <think>->think_auto for reasoning); "none" disables the reasoning split; unknown names fail the first chat call. - Chat chunks parse the `reasoning` field (the pin renamed reasoning_content), flowing into ChatDelta.ReasoningContent which the host already prefers. Live e2e against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU, full suite green: the real chat template renders (no more fallback), reasoning auto-detection picks think_auto so markerless answers stay pure content (the live run caught the deepseek_r1 content-swallow upstream and drove the think_auto fix), required tool_choice returns schema-valid arguments, auto tool_choice engages engine-side and streams parsed deltas, and blocking/streaming stay byte-identical. Turn latency also dropped (proper template EOS behavior). Upstream program landed as mudler/vllm.cpp 86013f3..5fffe7e (ABI v2-v5, minja, parser waves B1/B2/B4, reasoning seam, structural-tag registry, think_auto). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(vllm-cpp): bump the engine pin to the ENG-wave close-out mudler/vllm.cpp@df8909b: the six engine-backed vLLM tool-parser families (qwen3-coder/xml/mimo, kimi_k2, glm45/47, minimax_m2, gemma4, seed_oss) text-reimplemented from their wire formats and held to the upstream test suites - 39 registered dialects; the pinned vLLM registry is now covered except the three Rust/Harmony-backed families, descoped by decision. kimi_k2 also gains a full native structural-tag builder; four new template auto-detection rows land with test-pinned ordering. Full backend e2e re-run green against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): add the vllm-cpp-development gallery meta The gallery grew the twelve latest/development image entries but was missing the separate vllm-cpp-development meta (own capabilities map targeting the -development image names), which every backend ships so the development gallery resolves per-platform. Validated: all capability targets in both metas resolve to existing entries, and every image URI's tag suffix matches a backend-matrix build. Also full-stack verified in this change's context (single-node local-ai from this branch, locally-built backend under --backends-path, Qwen3.5-2B GGUF): /v1/chat/completions non-stream (clean content + usage), streaming (SSE deltas), tool_choice auto engaging get_weather engine-side with schema-valid arguments and finish_reason=tool_calls, and streamed tool-call deltas in the standard name-first cadence. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): repair the CI backend builds - gcc-14 -Werror + fat-arch Triton Two distinct failures took down all five vllm-cpp backend builds on the PR: 1. gcc-14 (ubuntu:24.04 CI images; the local toolchain is gcc-13) fails the engine build with -Werror=maybe-uninitialized in InputBatch::condense - a false positive through a staging std::optional's raw storage. Fixed upstream (mudler/vllm.cpp@61f3e85) by moving slot-to-slot directly; verified BOTH ways under dockerized g++-14.2 (unfixed reproduces CI's two diagnostics exactly, fixed compiles clean) with the engine's behavior suites green. Pin bumped to that sha. 2. The amd64 CUDA builds died at CMake configure: the vendored Triton-AOT cubin trees are per-arch and the engine refuses -DVLLM_CPP_TRITON=ON on a multi-arch (120a;121a) fat build unless pinned to one tree, which would be unsound for the other arch. Triton is now enabled only on the single-arch arm64/GB10 build (where the cubins matter); the fat amd64 binary uses the engine's non-AOT GDN path. Backend e2e re-run green at the new pin (Qwen3.5-2B on CPU, full suite). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): cuda-12 images cannot compile compute_121a - target 120a only The second CI round surfaced a CUDA-version constraint: the cuda-12 (12.8) image's nvcc rejects 'compute_121a' (GB10 arch support landed with CUDA 13), killing the amd64 cuda-12 build at nvcc. Gate the architecture list on CUDA_MAJOR_VERSION (exported by Dockerfile.golang): cuda-12 builds consumer Blackwell 120a only, cuda-13 keeps the 120a;121a fat binary, arm64/l4t (cuda-13) keeps single-arch 121a with the Triton cubins. GB10 is arm64, so the amd64 cuda-12 image never served it - no capability change. Verified by Makefile dry-run variable dumps for all three combinations (cuda12 -> 120a; cuda13 -> 120a;121a; cpu -> CUDA off). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): drop the cuda-12 variant - the engine needs the CUDA 13 toolchain Third CI round, third layer: with the arch list already narrowed to 120a, the cuda-12 (12.8) build still dies in ptxas compiling the sm_120a NVFP4 MMA kernels ("Vector type too large, exceeds 128 bit limit") - the Blackwell fp4 path genuinely requires the CUDA 13 toolchain, and vllm.cpp supports Blackwell-family GPUs only. Shipping a cuda-12 image without the fp4 kernels would be a crippled build of an engine whose whole GPU story is fp4, so the variant is dropped instead: - backend-matrix: cuda-12 vllm-cpp entry removed (cuda-13 amd64, l4t arm64, cpu, vulkan, metal remain). - gallery: cuda12 image entries removed; the nvidia capability now resolves to the cuda13 image in both metas; the nvidia-cuda-12 key is dropped so older-driver hosts fall back to the CPU image instead of an unrunnable one. - backend Makefile: BUILD_TYPE=cublas under CUDA_MAJOR_VERSION=12 now fails fast with a clear message; cuda-13 keeps the 120a;121a fat binary and arm64/l4t keeps 121a with the Triton cubins. Verified: Makefile branch dumps for all four combinations (cuda12 loud error, cuda13 fat, arm64 121a+Triton, cpu off), YAML parses, matrix filter tests green, gallery capability targets all resolve. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): forward multi-turn tool identity and reasoning to the engine chatRequestJSON dropped Message.ToolCallId and Message.Name on role="tool" replies and Message.ReasoningContent on assistant history, so a second turn after tool execution reached the engine's chat template without the fields that bind a tool result to the call it answers. Forward all three (present-only, matching the OpenAI wire shape) and pin vllm.cpp to 6a0bd3e7, where ChatMessage parses/round-trips tool_calls, tool_call_id, name and reasoning and the minja adapter exposes them to the template context. Adds the round-trip request-lowering spec (user -> assistant tool_call -> tool reply -> lowered request) and re-ran the gated e2e suite against the new engine pin with a real Qwen3.5 GGUF: chat, reasoning split, streaming parity, required-tool and auto-tool cases all green. Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): bump vllm.cpp for the darwin arm64 i8mm build fix The darwin-metal CI job was the first build to compile the engine's arm CPU-quant files on macOS and hit their Linux-only <asm/hwcap.h> / <sys/auxv.h> includes. vllm.cpp 9e1c9025 detects i8mm per-OS (auxv on Linux, sysctl on Apple Silicon) with kernels untouched. Gated e2e suite re-run green against the new pin with a real Qwen3.5 GGUF. Assisted-by: Claude Code:claude-fable-5 [Bash] [Read] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): darwin build - bound cmake parallelism when nproc is absent The macOS runners have no nproc, so JOBS evaluated empty and `cmake --build -j$(JOBS)` became bare `-j`: unlimited clang jobs on a 3-core/7GB Mac, which swap-thrashed until the 6h GHA timeout (the log shows "nproc: Command not found" and 7+ concurrent clang processes being reaped at the cutoff). Use the same portable fallback chain as the other darwin backends: nproc, then sysctl hw.ncpu, then 4. Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]
|
40d35c0385 |
docs: onboarding overhaul, dedup, and error docs (#7711) (#10895)
* docs: fix CPU image tag (latest, not latest-cpu) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: use canonical localai/localai registry in models guide Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: replace dead llama-stable backend with llama-cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct mitm-proxy intercept config and redaction tier Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fix text-to-audio endpoint and broken notice block Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fix VAD example, stale FAQ, broken link, CLI list, whats-new dump Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: render advanced/reference section indexes (consolidate _index) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: remove duplicate getting-started build/kubernetes pages Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fold container image reference into installation/containers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: remove stale advanced fine-tuning page (superseded by features/fine-tuning) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fold distribution/longcat/sound pages into their parents Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: make getting-started index accurate and complete Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: carry one concrete model through the getting-started path Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add end-to-end 'build your first agent' walkthrough Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add runtime errors reference; consolidate troubleshooting from FAQ Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add agent actions catalog Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: agent-scoped MCP, skills walkthrough, agentic disambiguation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add concrete gallery install lines to media feature pages Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: merge installation into getting-started (URLs preserved via aliases) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add Operations section; move operator pages and P2P API reference Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: journey-ordered top nav and grouped feature sections Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add docs-with-code process gate (PR template + agent instructions) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: remove em/en dashes from documentation prose Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]
|
b00422e45f |
feat(backends): add LongCat video and avatar generation (#10792)
* feat(backends): add LongCat video and avatar generation Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web] * refactor(config): declare model I/O modalities Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook. Assisted-by: Codex:GPT-5 [web] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]
|
94bdc825dc |
feat(backend): add moss-transcribe-cpp backend (MOSS-Transcribe-Diarize) (#10756)
C++/ggml transcription + speaker diarization + timestamps backend. Purego dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output into segments with nanosecond timestamps. Adds the importer (surfaces in GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml, and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf). Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues) passed against the real libmoss-transcribe.so. The pre-commit coverage gate (full pkg/core + tests/e2e) could not run in the authoring sandbox (no live models, port 9090 held); CI must enforce it before merge. Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]
|
600dafd20b |
feat(ced): sound-event classification backend (CED audio tagger) (#10425)
* feat(ced): sketch sound-classification backend (CED audio tagger) Wires ced.cpp (CED, 527-class AudioSet sound-event tagger; baby cry, footsteps, glass, alarms, dog bark) into LocalAI as a Go/purego backend. SKETCH (backend skeleton real; core REST wiring + CI/gallery is a checklist in DESIGN.md): - backend/backend.proto: new SoundDetection rpc + SoundClass messages (run `make protogen-go` to regenerate pkg/grpc/proto). - backend/go/ced: main.go (purego dlopen libced.so + ced_capi.h), goced.go (Ced gRPC backend: Load + SoundDetection), Makefile (clone-at-pin CED_VERSION, ggml static-PIC shared build), run.sh, package.sh, .gitignore. - DESIGN.md: REST /v1/audio/classification wiring (handler/route/capability registration checklist), gallery/index + CI registration, and a scoping note for the realtime/websocket live-recognition path (sliding-window classify over the existing ws transport + voicegate; the ced C-API per-PCM entry point is already window-friendly). Backend code does not compile until protogen-go regenerates the pb types and a libced.so is built (Makefile clones+builds it). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): REST /v1/audio/classification endpoint + capability registration Wires the ced sound-event classification backend (AudioSet audio tagger) end to end through the REST surface, mirroring the transcription path. - Handler: core/http/endpoints/openai/sound_classification.go parses the multipart audio upload, temp-files it, resolves the model config and calls the SoundDetection RPC; returns {model, detections[]} JSON. - Backend wrapper: core/backend/sound_classification.go (ModelSoundDetection) loads the model and normalizes the proto response into schema types. - Schema: core/schema/sound_classification.go (SoundClassificationResult). - gRPC layer: SoundDetection wired through the LocalAI wrapper (interface, Backend client, Client, embed, server, base default) so the loader-typed client exposes the RPC; proto regenerated via make protogen-go. - Route: POST /v1/audio/classification (+ /audio/classification alias) with the audio/multipart default-model middleware in routes/openai.go. - Capability surfaces: swagger @Tags/@Router on the handler; FLAG_SOUND_ CLASSIFICATION usecase flag + UsecaseSoundClassification + UsecaseInfoMap + GuessUsecases + ModalityGroups + GetAllModelConfigUsecases; meta usecase option; /api/instructions audio area updated; auth RouteFeatureRegistry + FeatureAudioClassification (APIFeatures, default ON) + FeatureMetas; UI usecaseFilters, capabilities.js CAP_SOUND_CLASSIFICATION, Models.jsx filter + i18n; docs page features/audio-classification.md + whats-new + crosslink. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): realtime sound-event detection over the websocket API When a realtime pipeline configures a sound-classification model, each VAD-committed utterance (the same window the transcription path produces) is also run through the CED sound-event classifier and the scored AudioSet tags are emitted as a new server event. No new backend rpc is needed: the SoundDetection gRPC method already exists on this branch. - config: add Pipeline.SoundDetection (yaml/json sound_detection,omitempty) beside Transcription/VAD. - realtime: add Model.SoundDetection(ctx, audio, topK, threshold) to the ModelInterface; implement it on wrappedModel and transcriptOnlyModel by calling backend.ModelSoundDetection with the session's sound-classification model config (mirrors how Transcribe dispatches). Load the optional config in newModel / newTranscriptionOnlyModel; nil config keeps it additive. - types: add ConversationItemSoundDetectionEvent (item_id, content_index, detections[]{label,score,index}) with type conversation.item.sound_detection, its ServerEventType constant and MarshalJSON, mirroring the transcription completed event. - realtime: add emitSoundDetection (unary path: classify the committed window, build the event, t.SendEvent) and wire it at the utterance-commit hook right after emitTranscription; gated on session.SoundDetectionEnabled (resolved from Pipeline.SoundDetection at session setup, defaults top_k=5, threshold=0). Its error is logged via xlog but never aborts the turn. - test: Ginkgo specs for emitSoundDetection (tags emitted, empty detections, classifier error) plus a SoundDetection method on the fakeModel double. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): implement SoundDetection in nodes backend test doubles The SoundDetection method added to the grpc backend interface left two test doubles (fakeBackendClient, fakeGRPCBackend) incomplete, so core/services/nodes failed to compile under `go vet`/`go test` (go build missed it: the doubles live in _test.go). Add the method to both, mirroring their existing Detect mock. Repairs CI for the nodes package. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): decouple realtime sound detection from VAD (sound-only sessions) Sound-event detection must activate on sounds, not speech, so it no longer runs through the voice VAD/transcription path. A sound-detection-only pipeline (sound_detection set, no transcription/LLM) now: - is accepted by prepareRealtimeConfig (sound_detection counts as a pipeline stage), - builds a lightweight model via newSoundDetectionOnlyModel (no VAD/STT/LLM/TTS loaded), and - defaults the session to turn_detection none (no VAD) with no transcription stage, so the client drives windowing via input_audio_buffer.commit (option A: client-side sliding window). The per-PCM C-API already supports arbitrary windows. commitUtterance gains a sound-only branch: it emits the conversation.item.sound_detection event (scored AudioSet tags) and stops - no transcription, no LLM response. generateResponse is now guarded on a transcription stage being present, so a sound-only turn never invokes the LLM. Existing transcription/VAD sessions are unchanged (additive). Added a commitUtterance sound-only Ginkgo spec asserting it emits the sound event and neither transcribes nor generates a response. go vet + golangci-lint (new-from-merge-base) clean; openai suite green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): register sound-classification backend in gallery + CI Mechanical backend-image registration for the ced sound-event classifier, mirroring the parakeet-cpp Go/purego backend everywhere it is wired up. - .github/backend-matrix.yml: add the ced build matrix, field-for-field copies of the parakeet-cpp entries (cpu amd64/arm64, cublas cuda 12/13 amd64, l4t cuda-13 arm64, l4t-jetpack cuda-12 arm64, sycl f32/f16, vulkan amd64/arm64, rocm hipblas, and the metal darwin entry), changing only backend and tag-suffix. dockerfile stays ./backend/Dockerfile.golang. - backend/index.yaml: add the &ced meta anchor (capabilities map per platform) plus ced-development and the per-arch image entries, each uri/mirror tag-suffix matching the matrix exactly. The model gallery (GGUF) entry is intentionally deferred pending the HuggingFace publish (TODO note inline). - scripts/changed-backends.js: add an explicit item.backend === "ced" branch in inferBackendPath mapping to backend/go/ced/, same mechanism and ordering as the parakeet-cpp branch (before the generic golang fallthrough). - .github/workflows/bump_deps.yaml: register mudler/ced.cpp -> CED_VERSION in backend/go/ced/Makefile so the daily bot bumps the pin. - swagger/{docs.go,swagger.json,swagger.yaml}: regenerated via make swagger so the existing /v1/audio/classification annotations land in the generated spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): server-side windowing for realtime sound detection (option B) Adds an optional server-driven sliding-window classifier so a sound-only realtime client only has to stream audio (no input_audio_buffer.commit): - Pipeline.sound_detection_window_ms / sound_detection_hop_ms config knobs. When both > 0 on a sound-only session, the server classifies the last window of streamed audio every hop and emits a conversation.item.sound_ detection event; the input buffer is trimmed to one window so a long stream stays bounded. When unset, the session stays client-driven (option A). Runs independent of VAD (sound events are not speech). - handleSoundWindow (ticker) + classifySoundWindow (one tick, extracted so it is unit-testable) + writeWindowWAV, which declares the true InputSampleRate (NewWAVHeaderWithRate) so the classifier resamples correctly. Goroutine is started after toggleVAD and torn down with the session (close + wg.Wait). - Register pipeline.sound_detection (+window_ms/hop_ms) in the config meta registry; the earlier realtime commit added pipeline.sound_detection without a registry entry, failing TestAllFieldsHaveRegistryEntries. This fixes that and covers the two new knobs. Tests: classifySoundWindow emits an event + trims the buffer to one window, no-ops on too-little audio; writeWindowWAV declares the given sample rate. go build/vet + golangci-lint (new-from-merge-base) clean; config + openai suites green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add ced-base GGUF model gallery entries (f16 + q8_0) The ced-base weights are now published at mudler/ced-base-gguf (Apache-2.0, converted from mispeech/ced-base). Adds gallery/ced.yaml (backend: ced + known_usecases: sound_classification) and two gallery/index.yaml entries (ced-base-f16 default, ced-base-q8 smallest) with sha256-pinned files, and removes the now-resolved TODO from backend/index.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add tiny/mini/small GGUF model gallery entries Publishes the rest of the CED family (same architecture, metadata-driven port verified end-to-end on ced-tiny) to mudler/ced-{tiny,mini,small}-gguf and adds their f16 + q8_0 gallery entries: ced-tiny (5.5M, edge/Pi-class) f16 11MB / q8_0 6MB ced-mini (9.6M) f16 19MB / q8_0 11MB ced-small (22M) f16 42MB / q8_0 23MB All sha256-pinned. ced-base remains the accuracy default. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): point gallery entries at the consolidated mudler/ced-gguf repo All CED quantizations (tiny/mini/small/base, f16/q8_0) now live in a single HuggingFace repo, mudler/ced-gguf, instead of per-model repos. Repoint the 8 gallery model entries' urls + file uris accordingly. sha256 and filenames are unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): bump CED_VERSION to the short-clip fix Pin the ced backend to ced.cpp 99c6ed3, which fixes a crash on any clip shorter than target_length (~10.11s): time_pos_embed was added at its full 63-frame grid instead of being sliced to the clip's actual time grid, tripping ggml_can_repeat in ggml_add. Surfaced by the live realtime e2e (sub-10s windows) and gated with a short-clip parity test upstream. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(ced): list ced.cpp as a LocalAI-team engine + backend-guide directive - README.md: add ced.cpp to the "native C/C++/GGML engines developed and maintained by the LocalAI project" table. - docs/content/features/backends.md: add a Sound Classification backend category (sound-event classification / audio tagging) listing ced.cpp. - .agents/adding-backends.md: add a "Documenting the backend" section and two verification-checklist items requiring new backends to be documented in the backends.md category list, and in-house native engines to be added to the README maintained-engines table. This directive was missing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): repin CED_VERSION to the v0.1.0 release commit ced.cpp history was squashed into a single release commit (tagged v0.1.0), so the previous pin (99c6ed3) no longer exists upstream. Pin to c04ac14, the v0.1.0 release commit, so the backend builds against a commit that exists. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): silence gosec G304/G103 + govet unsafeptr on audited paths - sound_classification.go: os.Create(dst) where dst = temp dir + path.Base of the upload (no traversal). #nosec G304, matching the depth-anything-cpp handler. - goced.go: reading a NUL-terminated C string from a libced-owned buffer. #nosec G103 (gosec) + //nolint:govet (golangci-lint's unsafeptr check), since the uintptr is a C-owned malloc'd buffer, not Go-GC memory. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]
|
13f59f0822 |
docs: document the privacy-filter.cpp backend (#10386)
docs: document the privacy-filter.cpp backend in README and compatibility table The privacy-filter.cpp backend (#10360) was registered in backend/index.yaml and referenced from the PII feature docs, but was missing from the backend catalog surfaces. Add it to the README "Backends built by us" table, the compatibility table (Utilities & Other, CPU/CUDA 13/Vulkan), and the backend type list in the backends feature doc. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]
|
9b57dcb721 |
docs: document all available backends and add "built by us" list (#10376)
Bring the Backend & Model Compatibility Table up to the full set of backends published in backend/index.yaml (60+), organized by modality with per-backend acceleration targets. Add an "Available Backends" pointer and expand the backend-type list in the backends feature doc. Update the README backend count to 60+ and add a "Backends built by us" section listing the native C/C++/GGML engines maintained by the LocalAI project (parakeet.cpp, voxtral.c, vibevoice.cpp, rf-detr.cpp, locate-anything.cpp, depth-anything.cpp, LocalVQE, local-store). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
7e0b73deaa |
fix(docs): fix broken references to distributed mode
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
bf5a1dd840 |
feat(voxtral): add voxtral backend (#8451)
* feat(voxtral): add voxtral backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * simplify Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
26a374b717 |
chore: drop bark which is unmaintained (#8207)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
2cc4809b0d |
feat: docs revamp (#7313)
* docs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small enhancements Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Enhancements * Default to zen-dark Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |