1248 Commits

Author	SHA1	Message	Date
Ettore Di Giacinto	f88444f8e3	feat(backend): add buun-llama-cpp fork (DFlash + TCQ KV-cache) ... spiritbuun/buun-llama-cpp is a fork of TheTom/llama-cpp-turboquant that adds two independent features on top: DFlash block-diffusion speculative decoding (via a dedicated DFlashDraftModel GGUF arch) and two extra TCQ KV-cache variants (turbo2_tcq, turbo3_tcq) on top of TurboQuant's turbo2/turbo3/turbo4. Follows the turboquant thin-wrapper pattern — reuses backend/cpp/llama-cpp grpc-server sources verbatim, patches only the build copy to extend the KV allow-list and wire up buun-exclusive tree_budget / draft_topk options. DraftModel is already wired end-to-end (proto field 39 → params.speculative), so DFlash activation only needs the existing options passthrough (spec_type:dflash) plus the drafter path in draft_model. CacheTypeOptions now surfaces the five turbo* values so the React UI dropdown shows them — benefits turboquant too (previously users had to type them in YAML manually). Assisted-by: Claude:Opus-4.7 [Read] [Edit] [Bash] [WebFetch] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-30 11:04:29 +00:00
mudler's LocalAI [bot]	c4617265b9	fix(ci): trim two pieces of per-PR work that buy nothing (#11219) ... Measured over the week to 2026-07-30, 97% of CI wall-clock is queueing and 3% is execution: a median 5-hour queue against a 4-20 minute median job. With the queue saturated, throughput is concurrency divided by service time, so cutting execution time raises the drain rate directly. Two steps stood out as paying nothing for what they cost. test.yml: drop the free-disk-space step (~3.1min per run, ~22 h/week). That action exists to make room for docker buildx layers and this job runs no buildx step. It was also sized for a `make test` that downloaded multi-GB GGUF/whisper fixtures and built llama-cpp/whisper/stablediffusion-ggml; the test-suite reorg moved all of that into tests/e2e-backends and tests/e2e-aio, as the Makefile test target already records. Its tool-cache:true wipe was additionally deleting /opt/hostedtoolcache, forcing setup-go and setup-node to re-download toolchains that ship preinstalled on the runner. build-test.yaml: build only the host target on pull_request. The three-platform cross-compile (linux/amd64, linux/arm64, darwin/arm64) is the bulk of that job's ~6.6min median, ~47 h/week, and nothing consumes a PR's binaries. goreleaser's --single-target still runs every before-hook (protogen-go, react-ui, go mod tidy), so the "is the release build broken" signal is unchanged. master pushes and tags keep building all three. Also record why the Linux Go workflows pass cache: false to actions/setup-go, since it reads as an oversight and is not. Set up Go has a median of 11 seconds on those runners, so there is nothing to win, and the repo already sits at GitHub's 10 GB Actions cache ceiling with 31 entries, where each setup-go entry is 222-375 MB on Linux and up to 1.4 GB on macOS. Re-enabling it would evict something that is earning its space. Assisted-by: Claude:opus-5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-30 12:17:14 +02:00
mudler's LocalAI [bot]	47097041ff	fix(vllm): apply Options[] engine flags before engine init (#11147) ... fix(vllm): apply Options[] engine flags before engine init (#11130) CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`, `--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could still reach the engine. Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine is constructed. Names are normalized the way vLLM's CLI spells them (`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to the target field's type (bare flag -> True for booleans), and unknown or uncoercible flags warn and are skipped instead of failing the load, since Options[] is a bag shared with backend-level settings. Field types come from the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not mistaken for a float. Precedence is typed proto fields -> `options:` -> `engine_args:`. The production engine_args defaults seeded in hooks_vllm.go therefore skip any key the user already set as an option, otherwise the later engine_args pass would silently override it. Parser lookups now accept both spellings, so `--reasoning-parser:qwen3` selects LocalAI's parser as well. The helper's tests are stdlib-only and run in the lint workflow's dependency-light job via `make test-python-helpers`. Assisted-by: Claude:claude-opus-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>	2026-07-30 12:13:00 +02:00
mudler's LocalAI [bot]	9c85cacfe3	feat(audio-cpp): add the audio.cpp native backend (#11141) ... * backend(audio-cpp): add the native build scaffold Links 0xShug0/audio.cpp engine_runtime through its public framework headers and serves Health/Status. Model loading and the audio RPCs follow. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): keep the build-tree rpath at $ORIGIN Upstream sets CMAKE_BUILD_WITH_INSTALL_RPATH in its own directory scope, so CMake was appending its build-tree library dir to our target and baking an absolute build-host path into the shipped binary. Set BUILD_WITH_INSTALL_RPATH on the target so a package that forgets to bundle libggml*.so fails on the build machine too, instead of only on a user's box. Also document why EXCLUDE_FROM_ALL must stay on the add_subdirectory call, correct the claim that Ubuntu ships no gRPC CMake config, stop the pin comment from repeating the assignment token that bump_deps.sh rewrites, and make test-engine fail rather than pass when no test is registered. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): parse namespaced model options Splits option entries on the first colon so path values survive, and routes load./session. prefixes to the upstream load and session option maps. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): reject out-of-range numeric model options std::atoi is undefined once the digits exceed long and in practice wraps, so device:2147483648 was accepted and handed the ggml backend selector a device index of -2147483648 from a function whose error text promises a non-negative integer. Parse with strtol and reject on ERANGE, on a value above INT_MAX, and on any unconsumed trailing input. The error strings are unchanged. Name the whole entry in the unknown-key error too: an entry like ':value' has an empty key and left the user nothing to grep for in their YAML. Tests look keys up through a helper instead of map::at, so a prefix off-by-one fails one named check rather than aborting the binary and skipping the rest of the suite, and cover the overflow, negative and non-numeric paths. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): route LocalAI RPCs onto audio.cpp tasks Task-major resolution over the family's advertised capability set, with the voice-reference and instructions signals selecting cloning and voice design, and a streaming-to-offline fallback for server-streaming transcription only. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): use upstream's 'spk' task name and pin the preference order The SpeakerRecognition short name was 'spkrec', which audio.cpp neither prints nor parses; a name copied out of audio.cpp was rejected and a pinned 'spkrec' would not survive the engine boundary. Emit 'spk', keep 'spkrec' as an input-only alias, and correct the known-tasks lists. Three assertions were vacuous because their fixtures advertised a single task, so reversing a preference order or dropping the RPC name and the attempted pairs from the capability error all passed. Give them fixtures that can tell the orderings apart. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): convert sample, time and PCM units Integer nanosecond conversion so 44.1 kHz stays exact, float seconds for the VAD and diarization messages, and saturating s16le encode so an overshooting sample cannot wrap to the opposite sign. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): harden seconds_to_samples against NaN and overflow seconds_to_samples is the one entry point fed by untrusted-shaped input: a float-seconds timestamp off the wire, or a boundary from a model that diverged. Its guard covered only the low side, so NaN and out-of-range values fell through to an undefined double-to-int64 cast and came back as INT64_MIN. A hugely negative sample index used later as an offset or a length is a wild pointer rather than merely a wrong timestamp. Reject NaN with the !(x > 0) form and saturate before the cast. Also round instead of truncating there. These functions exist to cross the float seconds boundary the VAD and diarize messages use, and truncation lost a sample about half the time on the samples-to-seconds-and-back round trip, starting at n=1. Pin the decode scale at INT16_MIN, pin nanosecond truncation on a nonzero fraction, and record why the clamp argument order in f32_to_s16le is load-bearing for NaN. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): map NaN PCM samples to silence explicitly f32_to_s16le relied on std::min argument order to keep a NaN sample away from std::lround, whose result is unspecified for NaN. That was too subtle to rest on a comment, and the comment was itself wrong: it warned against a spelling that the outer std::max already catches, while three real spellings leak, including std::clamp, which is the idiomatic C++17 way to write the same clamp and so the likeliest future edit. Divert NaN before the clamp and encode it as 0. A NaN sample rendered as a full-scale click is worse audio than a dropped one, and this unit converts audio that may have originated off the wire. Pin it with an exact-value check rather than a range check, since all three outcomes the plausible spellings produce are finite and inside full scale, plus an invalid-operation check that fails unless the NaN is diverted before any ordered comparison. That second check is what catches modernizing the clamp and dropping the guard together. Also bound the seconds round-trip comment, which claimed unconditionally what holds only below roughly 2^23 samples, and document NaN, saturation and that bound in the header. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): assemble transcripts from runtime spans The top-level transcript text is TaskResult.text_output verbatim. audio.cpp carries text nowhere else: speech_segments, speaker_turns and word_timestamps hold spans and labels only, so deriving the text from them empties the transcript for any producer that omits word timing, VibeVoice diarized ASR included. Fixtures cover every observed producer shape. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): keep a nested speaker turn's own label A segment sourced from speaker_turns re-derived its speaker by greatest overlap. A turn's overlap with its own span is the largest possible, so a turn nested inside another speaker's turn could only tie with the container, and the tie went to whichever came first. sortformer_diar binarizes each speaker independently and sorts by start sample, so the container always comes first and the interjecting speaker was silently erased from DiarizeSegment.speaker. choose_segment_spans now carries the label out with the span. Also pins the nearest-segment fallback against measuring from either endpoint or from segment position, which a trailing-only stray word could not do, and exercises the empty-word guard in join_words. Two fixtures that pin a rule but do not mirror any pinned family are relabelled defensive. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serialize runs with a wedge-aware guard audio.cpp sessions are not reentrant and a wedged CUDA call cannot be cancelled, so a plain mutex would pile every worker thread behind a stuck GPU. Callers waiting past the configured bound, or arriving while the holder has already overrun it, fail fast instead. A caller that queues behind a healthy run deliberately does not stamp the clock: only the thread that takes the lock does. Stamping on arrival would restart the wedge clock on every request and hide a stuck run from everyone behind it, which is the pile-up this guard exists to prevent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serialize inference through an InferenceLane One audio.cpp model is loaded per backend process and its sessions are not reentrant, so concurrent gRPC handlers have to take turns. Serialization alone is not enough: a wedged GPU call cannot be cancelled from userspace, so an unbounded queue behind one stuck run would swallow every gRPC worker thread until the process is useless. InferenceLane gives handlers a lane with room for one runner. LaneEntry occupies it for a scope and gives it back on every exit, including an exception, and is the only way to take the lane at all: occupy/vacate are private with LaneEntry as the sole friend, so a caller cannot acquire without holding something that releases. LaneEntry is immovable on purpose, because a moved-from entry would have to stop releasing while the lane still recorded it as occupied. A caller either waits indefinitely or brings a millisecond budget. A bounded caller that cannot get in fails instead of waiting on, and a bounded caller whose budget is already shorter than the age of the run in the lane fails immediately, which is what stops a queue forming behind a wedged run. The two failures carry different text: one names the wait it exhausted, the other states the measured age of the run without claiming to know why it is long, since a short budget meeting a legitimately long run lands there too. The run's age is stamped only after acquisition. A waiter that published itself as holder would restart the measurement and hide a genuinely stuck holder from every caller behind it. Budget negotiation and the overrun decision are pure functions taking their inputs explicitly, so both are covered without threads or sleeping. The per-model ceiling arrives as an int of milliseconds; a request may tighten it and may never loosen it. Replaces the previous run_guard unit, which was a derivative of an Apache-2.0 file upstream and could not stay in an MIT tree. Written from a behaviour contract with no reference to the removed code. Tests: 65 checks, standard library only, single translation unit, clean under -Wall -Wextra. Mutation tested at 23/23 killed; two of those mutants exposed missing coverage and the tests were extended until they died. ThreadSanitizer clean. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the B10 test able to fail, and document LaneEntry Review of the previous commit found the B10 test could not fail for the reason it was named. It aged the in-flight run to about 120 ms and then tried two budgets, 30 ms and 60 ms, both under that age, so both callers took the fail-fast path. "The two failure modes do not share one message" was comparing two fail-fast messages that differ only in the budget they print, and the timeout path was never reached. The second budget is now 400 ms, well over the run's age, so that caller queues and times out, and a new check asserts which path each caller took instead of inferring it from inequality. A mutant that makes the fail-fast path emit the timeout message previously died only on B4 and B8 checks; it now also dies on B10. Comment-only changes elsewhere. LaneEntry now says it is not reentrant and does not detect reentrancy: a second entry on a thread that already holds the lane surfaces as LaneUnavailable with a positive budget, but parks silently in unbounded mode, which matters because a handler may hold one across a whole stream. The immovability note now names the shapes that work, an optional emplaced in place or a unique_ptr, rather than saying to hold the entry indirectly without saying how; all three documented forms were compiled before being written down, which is how the note came to say that an optional of an immovable type cannot itself be returned. The header's explanation of why fail-fast exists is reworded. Two clauses traced back to a specification written after reading the Apache-2.0 upstream header, and while that was judged de minimis, this unit was rewritten precisely to carry no upstream expression at all. The margin table in the report was also wrong about which wall-clock margins are load-sensitive: there are four, not one, and the tightest is the B3 arrival check, which is now flagged at the call site. No margin value changed and none moved across 65 runs. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): gate model loading on the audio.cpp family Refuses any GGUF without an audiocpp.model_spec.family key and any non-GGUF path without an explicit family option, so the model loader's greedy backend probe cannot bind an unrelated llama.cpp GGUF to this backend (#9287). Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): load models and cache sessions per task Loads one ILoadedVoiceModel and creates an IVoiceTaskSession lazily per (task, mode), so the same model serves both the unary and streaming RPCs. LoadModel derives the family from GGUF metadata or an explicit option and fails with INVALID_ARGUMENT otherwise, so a failed load is a gRPC error the backend probe can see. audiocpp_backend::Task mirrors engine::runtime::VoiceTaskKind positionally, and drift there is silent: every unit still compiles and every test still passes while the backend runs a different task. Two mechanisms pin it. The static_asserts in loaded_model.cpp catch an insertion or a reorder, and -Werror=switch on that one file turns an appended upstream enumerator into a build failure rather than a warning in a 600 file log. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop aborting the process on SIGTERM The signal handler called grpc::Server::Shutdown directly. Shutdown takes an absl::Mutex, which is not async-signal-safe: the handler can interrupt a thread already holding that mutex, and abseil's deadlock detector responds by aborting. Every SIGTERM therefore ended in exit 134 and a 'dying due to potential deadlock' stack rather than a drained shutdown. The handler now sets a lock-free atomic and returns. Server::Wait moves to a helper thread so the main thread can poll that flag and call Shutdown itself, outside any signal context. A condition variable would not have helped, because notifying one from a handler is not async-signal-safe either. SIGTERM and SIGINT both exit 0 with no stack trace, where both previously exited 134. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): correct the status, lifetime and state contracts of LoadedModel An environment fault during session creation was reported as UNIMPLEMENTED. A missing libggml-cpu-*.so surfaced to the client as 'family silero_vad advertises vad/offline but refused to create the session: Failed to initialize CPU backend', which tells LocalAI the model cannot do this and must never be retried, and sends an operator hunting a capability bug instead of a packaging one. A throw from create_task_session is now a plain runtime_error, so it maps to INTERNAL. Only a null return, where the family genuinely declined, stays a CapabilityError. The model.'s task: option was parsed and then dropped: it lived in a local that died at the end of LoadModel and had no route to RequestShape::pinned_task. LoadedModel now keeps it and exposes pinned_task(). The global model becomes a shared_ptr reached through snapshot(). An audio RPC runs for seconds and cannot hold g_model_mu for its duration, so under a unique_ptr a Free arriving mid-request would destroy the model underneath it. Handlers now take a counted reference and whichever finishes last does the teardown, outside the lock. session_for documents the streaming state contract rather than resetting the session itself. Resetting on a cache hit was tried first and is not possible: silero_vad throws 'session prepare() must be called before Silero VAD reset()', so it would turn an ordinary second fetch into a hard error. start_stream's base implementation is already a reset, so a caller that runs prepare then start_stream per stream gets a clean session; a probe against the bundled silero_vad confirms an identical replay when it does and a carried-over stream when it does not. Also: an unknown backend: name is rejected before the model loads rather than after; MainGPU is parsed instead of passed through std::atoi, which turned 'gpu1' into device 0 silently; and device carries a device_set flag, because 0 is both the default and a real device index, so MainGPU was overriding an explicit device:0 that the neighbouring threads: handling promises will win. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the VAD and Diarize RPCs Both emit float seconds, converted from the runtime's sample-index spans, and both take a counted reference to the loaded model through snapshot() and hold it for the whole call: a Free arriving mid-request drops only the global's reference, so whichever request finishes last destroys the model instead of one of them running on freed weights. An AddressSanitizer build reproduces exactly that heap-use-after-free inside ggml_vec_dot_f32 when the handler keeps a raw pointer instead, which is why the shape is what it is. The inference lane is taken before session_for, not after. session_for reads and writes an unsynchronised session cache and the offline run calls prepare(), which mutates the session, so both belong inside the lane. Diarize routes before it reads the input file, so a family that cannot diarize at all says so rather than complaining about the audio first. Its per-segment text stays empty because audio.cpp's SpeakerTurn carries a span and a speaker label only, and nested or overlapping turns are passed through untouched: a sortformer turn inside another speaker's turn is correct output for overlapped speech, and LocalAI is overlap-tolerant downstream. Duration counts frames rather than floats, so a stereo input does not report twice its length. Verified end to end against upstream's bundled silero_vad, which needs no download, using the bundled 16 kHz speech asset: a synthetic tone returns nothing, correctly, because silero detects speech and a sine is not speech. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): enforce ModelIdentity on VAD and Diarize audio-cpp was the only C++ backend without the model-identity guard, and no later task in the plan added it. pkg/grpc/server.go enforces checkModelIdentity on exactly these two RPCs, for the reason #10952 records: in distributed mode a worker can recycle a stopped backend's gRPC port for another model's backend, and the controller's liveness-only probe cannot tell a stale cached route from a live one. Without this guard a stale route gets a different model's VAD or diarization answer back with a 200. The loaded identity lives on LoadedModel rather than in a separate global, which is where this differs from llama-cpp. A handler holding the model through snapshot() then necessarily judges against the identity that model was loaded with, and a concurrent reload cannot swap one without the other. The refusal is NOT_FOUND carrying the verbatim grpcerrors.ModelMismatchSentinel substring. session_for and run_offline now take a const LaneEntry & proof-of-holding parameter. The rule that both must run under the inference lane was prose, which is exactly how the plan came to specify the inverted order; it is now a compile error. Restoring the inverted order fails to build rather than racing on an unsynchronised session map with a mutating prepare(). Diarize's speaker-hint comment claimed the dropped hints were "not a silent failure". From the caller's side that is what they are, and backend.proto documents num_speakers as forcing, so the comment now says plainly that the forwarding is dead for sortformer and that the family which lands must either honour num_speakers or refuse it. read_audio_file inspects the error_code from exists(), so an unsearchable parent directory no longer reports as a missing file. The VAD handler records the stimulus that actually works, since silero correctly ignores synthetic tones and the next task would otherwise rediscover that. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the lane and identity guards structural Two hardenings ahead of the eleven handlers still to be written, both of which get harder to retrofit later. The lane proof-of-holding parameter was a const reference, which binds to a temporary, so session_for(rpc, shape, model->acquire(0)) compiled. Each such temporary dies at the end of its own full-expression, releasing the lane between two calls that must share one: precisely the split the parameter exists to prevent, and the form a future author is most likely to reach for because it reads as tidy. A non-const reference requires an lvalue, so the temporary form now fails to compile while the named-local handlers build unchanged. The header comment no longer implies the check is total either: it proves a lane was taken, not that it is this model's lane. The identity check was two lines each handler had to remember, with nothing failing if a new one forgot them and no C++ equivalent of model_identity_modalities_test.go to notice. snapshot() becomes snapshot_unchecked(), whose only legitimate caller is Status, since HealthMessage carries no ModelIdentity. Handlers go through snapshot_for(), which takes the counted reference, refuses when nothing is loaded, and runs the identity check before anything can route. Every handler already has to call something to obtain the model, so the guarded call is now the shortest path and skipping it means deliberately typing snapshot_unchecked. A convention that has to be remembered can rot; this cannot. Verified: the temporary-argument and inverted-order forms each fail to compile with the expected diagnostic, the real handlers build, and bypassing the guard in Diarize alone turns the identity test red on that RPC while VAD stays green. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the AudioTranscription RPC Adds result_map, the engine-to-proto boundary, and wires the offline transcription RPC. The handler branches on the ROUTED task: for Asr the request's prompt is whisper-style decoding context and becomes a request option, for Alignment the same field IS the transcript to align and becomes the text input. Routing has already decided which. The result text is TaskResult.text_output verbatim and is never derived from the segments. audio.cpp carries transcript text in text_output and nowhere else, so deriving it returns an empty transcript for every producer that reports segments without word timing. transcript_assembly already enforces that; this commit's job is not to undo it at the proto boundary, and result_map_ctest pins it there. read_audio_file now takes the sample rate the caller needs. Both file-fed speech handlers ask for 16 kHz mono, for two reasons: silero_vad and sortformer_diar refuse anything else outright, which turned an ordinary 44.1 kHz upload into INTERNAL, and nemotron_asr emits word timestamps in its own 16 kHz feature domain whatever the input was, so only a 16 kHz buffer makes the emitted nanoseconds right. Zero keeps the file's native rate and channels, which is what source separation will need. LoadedModel::check_can_serve answers a capability refusal before the lane is taken and before the input file is read. Routing is a pure read of the immutable capabilities, so a model that cannot serve an RPC no longer waits out somebody else's run to say so. VAD and Diarize use it too. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop linking sentencepiece's vendored protobuf engine_runtime links sentencepiece, whose default SPM_PROTOBUF_PROVIDER builds the protobuf-lite 3.14.0 sources it vendors. The generated backend.pb.cc is built against the toolchain's protobuf 3.21.12. Both ended up in the binary: 476 google::protobuf:: symbols came from the archive, 278 of them also defined by libprotobuf.so, and the archive won, because once ld pulls a member in for sentencepiece's own code every reference binds to the definitions that member carries. The visible symptom is one function. ParseContext::ParseMessage(MessageLite*, const char*) is what a generated _InternalParse calls for a submessage field and for nothing else, so flat messages parsed and nested ones did not: a TranscriptResult carrying segments serialized to correct bytes that the same process could not read back, and TranscriptLiveRequest, a oneof of submessages, could not have been parsed at all. Underneath that, 3.21 generated code was running 3.14 arena, ArenaStringPtr and ExtensionSet code. -Wl,--exclude-libs does not fix it. It makes those symbols LOCAL in .dynsym and the parse still fails, because the binding was decided at static link time and no visibility flag revisits it. Setting SPM_PROTOBUF_PROVIDER to "package" before add_subdirectory points sentencepiece at the protobuf the generated code was already built against. Zero google::protobuf:: definitions remain in the executable afterwards, every nested message round trips, and citrinet_asr, which parses a SentencePiece ModelProto at load time and would break first if this were wrong, still tokenizes and transcribes correctly. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): fix the segment text a transcription response is built from Segment text is not decoration. core/http/endpoints/openai/transcription.go routes response_format text, srt, vtt and lrc through schema.TranscriptionResponse, which builds the entire body out of Segments[].Text and never reads the top-level text. So for those four formats the segment text IS the response. nemotron_asr emits one word_timestamp per SentencePiece token, and the word boundary is carried as a LEADING SPACE on the piece ("So", "me", " call"). join_words inserted a space unconditionally, so response_format=text returned "So me call me na ture ," while the correct sentence sat unread in the top-level field. The separator is now chosen from the words themselves: whole words are space-joined, subword pieces are concatenated, and one leading space anywhere selects the latter. Concatenating the real nemotron pieces reproduces text_output exactly, verified end to end. This does not touch the top-level text, which is still text_output verbatim. The rule that forbids deriving the transcript from the segments is about the direction segments -> text; segment text has no source other than its words. Two smaller corrections in the same area: timestamp_granularities ["word"] set only "word_timestamps", a key no family in the pinned upstream reads. It now sets "return_timestamps", which qwen3_asr does read and which both runs its forced aligner and shortens its chunk window, so asking for word granularity no longer silently returns nothing. The request-option comment claimed more than it delivered. prompt, translate and temperature are read by no ASR family, and are forwarded only so a family adopting them works unchanged; the comment now says so per key, and gives TranscriptRequest.diarize the same explicit treatment threads already had. Also: the shipping target now carries -Wall -Wextra -Wpedantic, which it never did, so "the build is clean" starts meaning something; and fill_transcript_result no longer swallows a null response pointer, since answering OK with an empty transcript is the one failure mode this unit exists to prevent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the AudioTransform RPC Covers voice conversion, singing voice conversion, speech to speech and source separation, the four tasks LocalAI's AudioTransform can represent. AudioTransformResult carries one dst while htdemucs and mel_band_roformer produce several named stems from a single run, so inference runs ONCE, every stem is written as a sibling file <dst-stem>.<name>.<ext>, and params[stem] selects which one dst receives, defaulting to vocals and falling back to the first output. An unknown stem name is INVALID_ARGUMENT listing the real stem names rather than a silent substitution, and the selection happens before the first write so a refused request leaves no files behind. params[stem] is consumed here and is not forwarded into the engine's request options. The stem decision lives in stem_selection, which is stdlib only and therefore tested by backend/cpp/run-unit-tests.sh. It also validates the names, because they come from the model (htdemucs reads them from the GGUF's config.sources) and each becomes a component of a path this backend writes: a name carrying a path separator would escape the caller's output directory, and two stems sharing a name would silently overwrite one another. Both files are read at their native rate and channel count. Separation forces it, since demucs and roformer refuse any rate but 44.1 kHz and lose the stereo image that separates a centred vocal from a wide mix. The conversion families all resample internally (seed_vc, vevo2, miocodec, chatterbox were each checked), so passing the file through unchanged is also strictly better than band limiting it to 16 kHz first. Verified end to end against htdemucs f16 on a 44.1 kHz stereo mix: four stems plus dst, dst byte identical to the selected stem, params[stem] selecting a different one, an unknown stem refused with no files written, and mono input preserved as mono output. Also against miocodec for the single output path, where params[stem] is refused rather than ignored. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): refuse an impossible stem early, and stop blaming the caller for a failed write Four fixes from the first review of the AudioTransform RPC. check_can_serve now returns the resolved route, so params[stem] on a route that is not source separation is refused from the route instead of after a full inference: 11 ms rather than the 4.5 s a miocodec conversion costs, and far worse on seed_vc or vevo2. The post-run refusal stays as the backstop for a separation-routed family that returns no stems anyway. The typo'd-stem-name case still needs the run, since no framework header publishes the stem names before one. Stem names carrying control bytes are refused. GGUF strings are length prefixed and demucs reads its sources from JSON, so an embedded NUL survives to here: two names differing only after the NUL are distinct std::strings, so the duplicate check passes them, and then path::c_str() truncates both and they open the same file. That is exactly the silent overwrite the duplicate check exists to prevent, with the .wav lost as well. A failed write is now INTERNAL rather than INVALID_ARGUMENT. The destination is LocalAI's own generated-content directory, not anything the caller named, so a full disk or a permission fault there is a server fault and is worth retrying, which is the opposite of what INVALID_ARGUMENT tells a client. An empty output path stays INVALID_ARGUMENT. Two comment corrections and one clarification: the separators' required rate is their checkpoint's declared samplerate rather than a hardcoded 44100, seed_vc resamples with soxr and falls back to sinc-hann, and the "no files left behind" guarantee covers a refused request, not a write that fails partway through the loop. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(audio-transform): stop folding every upload to 16 kHz mono, and name the separation stems Two defects that made source separation unusable through LocalAI's own API, even though the backend served it correctly over gRPC. /audio/transform normalized every upload to 16 kHz mono s16 through utils.AudioToWav, with no way past it. htdemucs and mel_band_roformer refuse any rate but their checkpoint's own and separate a centred vocal from a wide mix using the stereo image, so every separation request through the HTTP API died with "HTDemucs prepare() sample rate mismatch: expected 44100, got 16000" while the same call over gRPC worked. The fold is not wrong, it is backend-specific: LocalVQE's echo cancellation genuinely wants 16 kHz mono and needs the reference in the same shape. So it becomes a declaration, BackendCapability.AudioTransformInputMono16k, set for localvqe and for nothing else. A backend that declares nothing gets its upload unchanged, which means no backend has to opt in to work. utils.AudioToWavPreservingShape is the non-folding conversion: a 16-bit PCM WAV passes through byte for byte at any rate and channel count, anything else is transcoded to WAV with its rate and channel layout kept. The other defect is that the run-once stem design bought nothing. A separation backend writes every stem beside dst from one inference, but AudioTransformResult carried only dst, so the other three were files no caller could find and a caller wanting all four had to run four separations. AudioTransformResult grows a repeated AudioTransformStem, the backend fills it, core/backend validates that each path really is inside the generated-content directory it handed over, and the endpoint publishes them as an X-Audio-Stems JSON header beside the existing X-Audio-Input-Url. JSON because a stem name is the model's own string and could contain any separator a hand-rolled format would use. Verified end to end through the HTTP endpoint with htdemucs f16 on a 44.1 kHz stereo file: 200 with a 44.1 kHz stereo body, all four stems named and fetchable through /generated-audio/, body byte identical to the selected stem, and params[stem]=drums returning a different one. The same upload sent to a model whose backend is localvqe still reaches the backend as 16 kHz mono, confirmed both by the engine's own rate refusal and by the persisted input file. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(audio-transform): reject extensible WAV from the passthrough, escape stem URLs, convert stems with dst Four fixes from the second review, plus one bug they made visible. isPCM16Wav tested only the bit depth, and go-audio's IsValidFile never looks at the format tag, so a 16-bit WAVE_FORMAT_EXTENSIBLE (0xFFFE) upload was passed through untouched where the old fold would have transcoded it. audio.cpp's WAV reader accepts 16-bit only when the tag is 1, so such a file died with "unsupported WAV encoding". Extensible is what many DAWs and Windows tools write and music files are this endpoint's new headline input, so it is a first-contact failure rather than a corner. The check now requires tag 1, with a spec that fails against the old implementation. Stem URLs are percent-escaped. A stem name is the model's own string and legally contains a space, a '#', a '?' or a '%'; an unescaped '#' truncates the URL before the request is even sent. The name field keeps the raw name. sample_rate and response_format are applied to the stems as well as to dst. Applying beat documenting: dst IS one of those stems, so leaving them alone broke the "dst duplicates the selected stem" invariant the whole design rests on, and both conversions are no-ops when unset. A stem whose conversion fails is dropped from the header rather than advertised in the wrong shape. Verifying that turned up why it had never been noticed: the two fields were never bound at all. The request arrives as multipart/form-data and echo's binder falls back to the FIELD NAME without a form tag, matching only case-insensitively, so "SampleRate" never matched "sample_rate" and "Format" never matched "response_format". Both were documented in the endpoint table and silently ignored. Two form tags fix it, and with them the conversion is observable end to end. Docs: audio-transform.md now documents what LocalAI does to an upload before the backend sees it, which backend gets the 16 kHz mono fold and why, params[stem], and the X-Audio-Stems header with a worked example. Also records the known limitation that the fold lookup is on the bare backend name, so pinned variants (vulkan-localvqe) do not match, and points at IsLlamaCppBackend as the suffix-tolerant precedent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the TTS and SoundGeneration RPCs TTSRequest.voice is treated as a speaker reference clip when it names an existing regular file, which makes routing prefer VoiceCloning, and as a named preset otherwise, in which case it travels as VoiceReference::cached_voice_id. Both the clip and SoundGenerationRequest.src are read at the file's own rate and channel count: upstream's own CLI and server do exactly that, every consuming family resamples internally and mostly with a better resampler than ours, and ace_step and stable_audio resample their input per channel, so a downmix here would delete the stereo image they are built to consume. The request builders live in their own unit rather than in grpc-server.cpp's anonymous namespace so they can be tested; grpc-server.cpp has a main() and cannot be linked into a test binary. The option keys are the whole point of these functions, so each one was grepped against the pinned upstream and the accounting is written down beside it. instructions maps to "instruct", which is what upstream's own server maps the OpenAI field to and what qwen3_tts and omnivoice read, and to "caption" for irodori_tts; the style tag is spelled "instruct" too, because "instructions" is looked up nowhere. duration maps to "duration_seconds", read by all three generation families, with the proto's own name kept only as a forward-tolerant alias. Keys that no family reads say so. Both handlers answer a capability refusal before taking the lane and before any file read, so a model that cannot synthesise does not queue behind somebody else's run to be told no. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop emitting an empty style language, and name the missing clip StyleCondition::language was set whenever has_language() was true, with no !empty() guard, while the language option twelve lines below had one. core/backend/tts.go sets Language unconditionally, so has_language() is true on every request LocalAI sends and carries "" when the caller named none. An engaged-but-empty style language is worse than an absent one: supertonic reads text_input->language behind its own !empty() guard and then overrides it from style->language with no guard at all, so "" replaced its "en" default and its tokenizer threw "invalid Supertonic language: ". Every /v1/audio/speech request that set instructions and no language would have been an INTERNAL against a supertonic model. A plain request never saw it, because the style condition only exists when instructions are non-empty, which is why the chatterbox end to end run did not catch it. TTS also stops discarding the Route that check_can_serve already returns. A family routed to voice cloning without a reference clip used to be refused from inside its own prepare(), which meant an INTERNAL naming neither the RPC nor the field to set; chatterbox advertises clon and no tts, so that was every preset-only request to it. It is now an INVALID_ARGUMENT naming TTSRequest.voice, answered in about 4 ms, and it cannot misfire because has_voice_reference is what selected cloning in the first place. Reading CapabilitySet::supports_speaker_reference to generalise this stays a follow-up. The src read carries a written caveat rather than a family blocklist, because ace_step's editing routes legitimately need src: setting src on a stable_audio model corrupts the heap and aborts the process in the pinned upstream, and the only thing keeping that off the network is that schema.ElevenLabsSoundGenerationRequest has no field for it. Nobody reading that Go schema would know why, so the reason is recorded where the field is read. build_tts_shape is extracted so TTSStream cannot describe the same request differently, and it arrived untested: two mutations of it survived until a test_tts_shape case was added. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the TTSStream and AudioTranscriptionStream RPCs TTSStream leads with a streaming WAV header carrying 0xFFFFFFFF sizes, matching the convention backend/go/vibevoice-cpp established, so an HTTP client can start playback before the full PCM exists. Its chunks are read from StreamEvent::named_audio_outputs and not audio_output: supertonic, omnivoice and voxcpm2 all put their streamed audio there and leave audio_output empty until the very end, so reading the obvious field yields a stream with no audio in it. The finish_stream result is the family's own merged whole rather than a tail, so it is emitted only when nothing was streamed. Streaming transcription sends incremental deltas and degrades to a single delta plus the final result on families that offer no streaming ASR, which is the same message sequence with fewer deltas. The four streaming ASR families disagree on what partial_text means: nemotron_asr, vibevoice_asr and higgs_audio_stt report incremental fragments while voxtral_realtime reports the whole hypothesis and reports it twice, so the reconciliation lives in one tested unit rather than in the handler. nemotron_asr reports only through the stream event sink, and only from inside finalize, so the audio driver installs one and clears it again before returning: the session is cached and a sink left holding the caller's frame is a use after free waiting for the next stream. begin_stream is now the only implementation of the streaming state obligation, prepare then start_stream. Streaming sessions are cached, and what clears the previous stream is start_stream's reset; a family override that dropped it would break every call site with no compile error, so there is one call site. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): keep streaming deltas on UTF-8 boundaries, refuse dtypes that abort TranscriptStreamResponse.delta is a proto3 string, whose wire format requires valid UTF-8. voxtral_realtime reports its hypothesis as a concatenation of raw token BYTES (tokenizer_text.cpp:171-183), so the cumulative difference between two consecutive reports is eventually a lone continuation byte, and the C++ runtime serializes that with only a logged warning while the Go runtime refuses to unmarshal it: the client loses the remaining deltas AND the final_result. Measured on a trace of a non-ASCII sentence, 11 of 31 messages failed to unmarshal and every accented character was lost. TranscriptDeltaTracker now holds back an incomplete trailing sequence and merges it into the next fragment; reconcile flushes it, which it always can because the final text is complete. The same trace now unmarshals in full with zero failures. A streaming buffer whose float count is not a whole number of frames is refused rather than truncated. The integer division dropped the tail floats from the fed audio and therefore from the transcript, with no diagnostic; vibevoice_asr refuses the same thing from the other side of the call. A supertonic GGUF whose weights are not f32 is refused at load. It reaches ggml_concat with mismatched operand types and ggml_abort takes the whole backend process down on the first request, so nothing downstream can report it: the model loads, then every request kills the process. Attributed rather than assumed, the unary TTS path aborts identically, and upstream records that package as untested. The refusal names the orig package and says what to run before deleting the guard. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop a repeated lead byte from orphaning the next delta The first UTF-8 fix closed the cumulative half only. Rule 2 discards a fragment the known text already starts with, and when that fragment is the LEAD BYTE of a new character it looks exactly like a repeat of an older character beginning with the same byte. It was discarded rather than held, its continuation bytes then arrived alone and began the next delta, and utf8_complete_prefix_length only ever inspected the trailing sequence, so a delta invalid at the FRONT went out whole. Through a real Go proto.Unmarshal the review's four-character repro gave 3 deltas, 2 unmarshal failures and a lost transcript. Reachable from the incremental families, not only from voxtral: nemotron_asr's decoder cuts at a byte offset and vibevoice_asr's common_prefix_size compares bytes, so both split characters. Measured over 30,000 randomized incremental traces, 53.28% of Japanese traces and 9.52% of French ones carried at least one delta the Go runtime refuses. Two changes. Rule 2 no longer judges a fragment that ends mid-character, so the lead byte is held instead of swallowed and the character survives intact; the cost is a few duplicated bytes in a shrinking cumulative report, which no pinned family produces. release() additionally drops leading orphan continuation bytes, so no delta can begin mid-character whatever the rules above it decide. Losing a byte keeps the stream alive; emitting one ends the RPC and takes the final_result with it. Post-fix all 60,000 traces produce zero unmarshal failures, and the cumulative streams plus both pure-ASCII incremental streams are byte-identical to the previous commit, so nothing changed for the families already working. The weight-dtype allow list moves to family_gate, where it is stdlib-only and pinned by a test rather than only by a comment. Two comment citations corrected. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): read only an exact repeat as a repeat, not any prefix Rule 2 discarded any partial the known text merely started with. For a cumulative family that is a duplicate; for an incremental family it is an ordinary short fragment that happens to coincide with the start of the transcript, and it was dropped, silently corrupting the text. Pure ASCII, no multi-byte character anywhere: the fragments "pure ", "ascii ", "trans", "c", "ri", "p", "t" left the client holding "pure ascii transcrit". Over 5,000 randomized traces per transcript, 9.50% of pure-ASCII and 29.12% of French traces ended with the client holding something other than final_result.text, with a 200 and no diagnostic. Both incremental families emit fragments that small routinely, since nemotron_asr cuts at a byte offset and vibevoice_asr at a common prefix. Narrowing rule 2 to an exact repeat drives that to zero on all six transcripts and changes no cumulative stream at all: 30,000 randomized cumulative traces are byte-identical to the previous commit. What rule 2 guarded was established from upstream rather than from its own comment. The only duplicate any pinned family produces is voxtral_realtime's, where process_available_stream_chunks feeds each event to the sink from inside its loop and returns the last of the batch, so that event arrives twice with byte-equal text. A duplicate is an exact repeat, so equality still covers it. The case given up is a cumulative report that SHRINKS, which no pinned family can produce: voxtral decodes a token vector that is only push_back'ed and cleared by reset(), so within a stream it can only grow. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the AudioTranscriptionLive RPC The one bidirectional stream this backend serves. The client sends a TranscriptLiveConfig, then TranscriptLiveAudio frames; the server acknowledges with ready, emits deltas as the audio arrives, and sends final_result once the read side closes. There is no offline fallback: live transcription has to consume audio incrementally, so a family with no streaming ASR is refused rather than served a batch run, which is what this RPC's Streaming-only mode_candidates list already says. The driver is a new sibling of run_streaming_audio, run_streaming_live, because the audio does not exist yet: instead of slicing a buffer it pulls frames from the caller until the read side closes. It installs the same ScopedStreamSink in the same order, which is not optional, since nemotron_asr returns a bare event from process_audio_chunk and reports every partial through the sink from inside finalize(). It buffers the wire's frames up to the family's own preferred window rather than feeding whatever size the client's audio callback produced, and it does not call finish_stream at all when no audio arrived, because nemotron_asr throws "finalize requires streamed audio" and an empty transcript is the truthful answer to transcribing nothing. Three things the handler had to get right and one it cannot: - The audio contract. A live request carries no samples, but nemotron_asr's streaming prepare() throws without an audio contract, and build_preparation_request derives it from TaskRequest::audio_input, so that field is an EMPTY buffer holding only the rate and the channel count. - 16 kHz or a refusal. The families express their spans in their own 16 kHz feature domain whatever the input was, and live frames cannot be resampled on the way in the way a file can, so an 8 kHz session would return timestamps 2x off with a 200. core/backend hardcodes 16000 anyway. - A mid-stream Config is refused. backend.proto calls it a decoder reset, but deltas already on the wire cannot be retracted, so a reset would leave the final text contradicting the transcript the client assembled. Ignoring the message would hand a client that believes it reset the decoder a transcript that silently continues the audio it thought it discarded. - The stale-route identity check cannot run here: TranscriptLiveRequest carries no ModelIdentity in either arm of its oneof, so snapshot_for does not instantiate for it. snapshot_unchecked's comment now names that as a second legitimate class of caller and says the fix is a proto change. eou and eob stay false. They exist for cache-aware models that emit end-of-utterance and end-of-backchannel tokens; audio.cpp's StreamEvent has no equivalent signal, and a client uses eou to decide the speaker yielded the turn, so a guess inferred from silence cuts people off mid-sentence. The lane is held for the whole stream, which is as long as the user keeps talking: the streaming session is stateful and cached, so a concurrent run would interleave two callers' audio and corrupt both transcripts. Verified against nemotron_asr over a real connection with a 14 s WAV in 512-sample frames: ready first, 59 incremental deltas with no repeated prefix, concat(deltas) equal to final_result.text, word timestamps in nanoseconds, eou and eob false. citrinet_asr answers UNIMPLEMENTED naming the family and listing asr/offline. A config followed by a close returns an empty final_result rather than hanging, and a first message that is not a config is INVALID_ARGUMENT. Two concurrent streams both return the complete transcript. Two cleanups on lines Task 12 touched, folded in. The DtypeAllowList terminator is now asserted at compile time: the reported out-of-bounds read did not exist, the single entry does terminate, but the loops have no other bound and any edit that widened an entry would walk off the end. And the dtype guard now short-circuits on "is there a table entry" through a new predicate rather than on the emptiness of the description string, which would have skipped the check on an entry with an empty allow list, i.e. on precisely the entry that refuses every dtype. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): bound the lane a live stream can hold AudioTranscriptionLive holds the model's inference lane for the whole stream, which is correct (the streaming session is stateful and a concurrent run would interleave two callers' audio) and newly dangerous. Every other RPC holds the lane across compute, or across a write to a slow reader, and both of those terminate on their own. A live stream instead blocks in a client-driven read, and a peer that goes silent WITHOUT closing the stream never terminates anything: the lane stays taken and every other request against that model queues behind a client that stopped speaking. live_watchdog is a one-shot idle timer that ends the stream when no frame has arrived inside a window. It is standard library only, so it is unit tested without an engine. gRPC's synchronous Read has no timeout and cannot be given one, so the only way to unblock it is ServerContext::TryCancel, which decides the wire status itself: the client sees CANCELLED rather than the DEADLINE_EXCEEDED the handler returns, the reason is logged, and the lane coming back is the point. When it fires the read loop throws rather than reporting end-of-input, so the driver does not go on to finalize a decode nobody is waiting for. It is armed only after the lane is taken and disarmed as soon as the read side closes, and both ends matter. Arming earlier would cover acquire(), which legitimately blocks while another live stream runs, so a queued caller would be cancelled for waiting its turn. Disarming later would cover our own decode, where a window overrun is not a peer going quiet and cancelling would throw away the transcript the client is waiting for. The window is the new live_idle_timeout_ms option, 30 s by default, 0 meaning no limit. core/http/endpoints/openai/realtime.go drives a 300 ms ticker and feeds every tick that produced new audio while a turn is open, so 30 s of silence is a hundred ticks that delivered nothing. It is also longer than any pause a speaker takes mid-utterance, which is the case that must never be cut off, and backend.proto lets one stream span many utterances, so a client that pauses longer between them raises the option rather than discovering it. Two smaller corrections in the same handler: - check_can_serve now runs BEFORE the sample rate check. pkg/grpc/grpcerrors/errors.go degrades to the file path on UNIMPLEMENTED and on nothing else, so a live-incapable model asked at a wrong rate was answering INVALID_ARGUMENT and costing the caller its fallback. - a negative sample rate is refused instead of silently becoming 16000. Zero still means 16000, which is what the proto documents; -1 is malformed rather than absent and gets the same refusal every other bad rate gets. And one thing recorded rather than changed, at the handler: "live" here means incremental INPUT, not low latency, and with the pinned families it does not yet mean incremental OUTPUT either. nemotron_asr's process_audio_chunk only appends to its buffer, so its whole decode and every delta happen inside finalize(), after the client closes its send side. The policy-window buffering is inert for that family and matters only for vibevoice_asr and higgs_audio_stt. Verified on the wire with live_idle_timeout_ms:3000. A silent client acked at 371 ms and was cancelled at 3.371 s; a second live stream opened one second later received its ack 2.37 s in, i.e. at the instant the first was cancelled, and then transcribed successfully on the same cached session. Without the watchdog it would still be waiting. Re-ran the live transcription (ready first, 59 incremental deltas, concat equal to the final text, word timestamps in nanoseconds, eou and eob false), the citrinet refusal at both a right and a wrong rate (UNIMPLEMENTED either way now), and Task 12's AudioTranscriptionStream on nemotron_asr, which is unchanged. Mutation testing the watchdog found a weakness in its own test: the destructor test slept past the window inside the watched scope, so a destructor that DETACHED the thread instead of joining it passed unnoticed. The test now uses a window longer than the scope, which kills that mutant, and says why. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): refuse the unsupported RPCs with a reason AudioEncode, AudioDecode, AudioTransformStream, AudioToAudioStream and VoiceEmbed have no counterpart in audio.cpp's VoiceTaskKind. Each now returns UNIMPLEMENTED naming the loaded family, what that family does support, and the upstream limitation, instead of the generated base class's bare status. The reasons live in a table in capability_routing.cpp so they are data rather than literals copied into five handlers, and so a test can assert every one of them. The five claims this was planned against were re-read at the pinned upstream e800d435d130dc776baf6f3e6129bb62b1495c89, and one did not hold. "audio.cpp streams tts and asr only" is false: silero_vad advertises vad with RunMode::Streaming. The refusal stands on the narrower claim that survives, that no family advertises streaming for any task AudioTransform routes to, and a test asserts the refuted wording does not come back. VoiceEmbed is the one refusal whose request carries a ModelIdentity, so it runs the #10952 check before answering: a stale route must get NOT_FOUND and the router's sentinel, not "audio.cpp cannot embed speakers" about a model that is not loaded here. It cannot use snapshot_for, whose no-model branch would tell the caller to load a model when no model can help, so it takes the reference through snapshot_unchecked and checks identity itself. That function's comment now names three classes of caller instead of two. The two bidirectional surfaces refuse without reading their stream, verified with a client that writes a config and eight frames first and gets the status rather than hanging. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): correct the vevo2 clause, and assert the absences Review found a false clause in the AudioToAudioStream refusal. It said s2s is "offline voice conversion ... which converts one clip into another speaker's voice", which is true of miocodec and false of vevo2: vevo2's s2s route is `editing` and only `editing` (default_route_for_task and route_matches_task in src/models/vevo2/session.cpp), documented as "Edit source speech into new target text while using the target voice" and requiring --target-text, so it rewrites what was said. vevo2's voice conversion is its separate vc task. It now reads "offline clip-to-clip processing against a target voice, declared only by miocodec (voice conversion) and vevo2 (speech editing)", and a test asserts the miscast cannot come back. The conclusion is unchanged: neither family converses. That defect was undetectable on the wire, since vevo2 does not load here, which is the argument for upstream_absence_ctest.cpp. It links engine_runtime purely to interrogate make_default_registry() and asserts the five premises the refusal reasons rest on: no codec task kind, no family advertising spk, no streaming for sep/vc/svc/s2s, miocodec advertising exactly vc and s2s, and s2s advertised by exactly miocodec and vevo2. The last two are exact sets, so an addition fails here rather than leaving a message stale. A positive control proves the registry is populated and the query works before any absence is believed, and every assertion has a reproduced negative control. This turns an AUDIO_CPP_VERSION bump from "remember to re-read five prose paragraphs" into a test failure. unsupported_surface now switches over UnsupportedRpc with no default label, so -Wswitch reports a sixth enumerator added without a row at build time; the runtime bounds guard it replaces is deleted. The AudioTransformStream reason had a true premise and an overreaching conclusion: an offline sep family could be buffered into a stream, as other LocalAI backends do. It now says this backend declines to offer a buffered offline call in disguise, rather than implying impossibility. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the missing-switch-case diagnostic fatal unsupported_surface() switches UnsupportedRpc onto the table row that explains it, with no default label, so -Wswitch reports an enumerator nobody handled. As a warning that is not enough: adding a sixth enumerator and building the shipping target gives exit 0, a binary and one warning, and the trailing `return surfaces[0];` then answers the new RPC with AudioEncode's codec reason. That is a confident, specific and false statement about audio.cpp on the wire, on the one code path whose entire job is to be truthful about what this backend cannot do, and it is worse than the runtime fallback it replaced, which at least named itself as a bug in this file. capability_routing.cpp therefore joins loaded_model.cpp on the existing -Werror=switch pin, whose comment already made this argument for the engine enum. The comment now covers both files. The pin stays per-file rather than project-wide because upstream's own ace_step/vae_decoder.cpp has unhandled -Wswitch cases of its own. Verified: a sixth enumerator now fails `make grpc-server` with exit 2 and no binary; appending a 14th VoiceTaskKind upstream still fails loaded_model.cpp, so the two pins fire independently; both reverted clean. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): package the backend image Bundles the dependency closure for the from-scratch image, the dlopened ggml CPU-variant shared objects that ldd cannot see, and upstream's bundled silero_vad and marblenet_vad assets so VAD works with no download. The bundled loader sits in the package ROOT rather than at lib/ld.so. run.sh execs it, which makes /proc/self/exe name the loader, and this backend has two consumers of that path: ggml discovers the libggml-cpu-*.so by listing dirname(/proc/self/exe), and resolve_model_path expands bundled:<name> under the same directory. Rooting the loader makes the binary, the ggml objects and assets/ share the one directory all three resolution mechanisms agree on. llama-cpp's lib/ld.so layout would need assets/ moved into lib/ as well. The image builds against apt gRPC and protobuf, like Dockerfile.ds4 and unlike Dockerfile.privacy-filter. The from-source gRPC that install-base-deps.sh and the base-grpc-* images supply vendors protobuf 26, which pulls abseil into message_lite.h; with SPM_PROTOBUF_PROVIDER=package that collides with sentencepiece's vendored mini-abseil and every absl::internal reference becomes ambiguous. Noble's protobuf 3.21.12 predates the abseil dependency and is the pair every earlier verification of this backend ran against. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): exempt the driver libraries from the packaging gate package.sh already left libcuda.so* and libnvidia-* to the host when copying, because the driver has to match the kernel module on whatever host runs the image, but the validation gate had no matching exemption. With BUILD_TYPE=cublas ggml is static and links CUDA::cuda_driver, so grpc-server carries DT_NEEDED libcuda.so.1 and the gate would have rejected the very absence the copy loop created, failing every cublas build in CI. One regex now feeds both. Building a control for that found a second defect: ld.so --list refuses to trace an object with an unresolvable dependency at all, exiting 127 without emitting a per-library line, so the "=> not found" rule was dead code and no exemption could have applied to it. The gate now traces with LD_TRACE_LOADED_OBJECTS and LD_LIBRARY_PATH, which reports the missing name and exits 0, and which is also what run.sh does at run time. Adds a layout assertion so a future move of the loader into lib/ fails the build instead of shipping a package that resolves bundled: models into lib/assets and finds no ggml CPU backend, and records for Task 16 that the Darwin script must not be a straight copy of privacy-filter-darwin.sh, which never calls package.sh and would silently drop assets/. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): register the backend with CI and the gallery Adds the five Linux matrix entries (cpu amd64/arm64 sharing a tag-suffix so the manifest merge fires, cuda 12, cuda 13, vulkan), the path-filter case that keeps later PRs touching backend/cpp/audio-cpp/ from getting zero CI jobs, the bump-bot entry pointing at the AUDIO_CPP_VERSION pin in the backend Makefile, the gallery meta plus its -development variant and the image entries for every variant, and the Makefile docker-build wiring. The matrix entries carry base-image only, with no builder-base-image, unlike the llama-cpp and privacy-filter blocks they sit next to. The prebuilt quay.io/go-skynet/ci-cache:base-grpc-* images ship a from-source gRPC whose protobuf v26 depends on abseil, and this backend's sentencepiece is built with SPM_PROTOBUF_PROVIDER=package, so it sees real abseil's absl::lts_20240116:: internal alongside its own vendored plain absl::internal and every absl::internal:: reference becomes ambiguous. Building against base-grpc-amd64 fails at sentencepiece-static.dir/error.cc.o with "reference to 'internal' is ambiguous". Dockerfile.audio-cpp installs apt's gRPC/protobuf 3.21.12 itself, which is also the pair every unit and end-to-end run of this backend has been verified against, and the CUDA toolkit therefore has to come from base-image. No Darwin matrix entry and no metal gallery entries: the Metal build needs scripts/build/audio-cpp-darwin.sh, a backends/audio-cpp-darwin make target and a routing step in backend_build_darwin.yml, none of which exist yet, so an entry added now would be routed to build-darwin-go-backend and look for backend/go/audio-cpp/. The inferBackendPathDarwin case and the DARWIN_BESPOKE_BUILDERS membership are in place, inert, so that adding the entry later is a one-line change that cannot be claimed by the generic Go path. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): pin the CUDA architectures, drop the vulkan variant Upstream sets CUDA_ARCHITECTURES to `native` on the engine_runtime target whenever CMAKE_CUDA_ARCHITECTURES is unset at root scope, and docs/build/ linux.md says so outright. ggml's own default does not rescue it: it list(APPEND)s in the ggml subdirectory scope, which never reaches the root scope where the engine_runtime property is decided. No CI runner has a GPU for `native` to enumerate, so both cublas entries would have gone red on the very commit that first turns a CUDA build on. Pin the list in backend/cpp/audio-cpp/Makefile, selected by CUDA_MAJOR_VERSION, which Dockerfile.audio-cpp now forwards from the CI build-arg it was previously discarding. The values are copied from ggml's own version guards rather than invented, so engine_runtime and ggml compile for the same set: CUDA 12 keeps the Maxwell/Pascal/Volta virtual archs and stops at 120a-real, CUDA 13 drops them and adds 121a-real. The `a` suffix is used rather than `f` because the latter needs CMake 3.31.8 and Ubuntu Noble ships 3.28.3. Verified by driving CMake 3.28.3's own CUDA architecture validator over both lists, with 120f-virtual as the rejected control. Drop the vulkan matrix entry, its two gallery entries, the vulkan capability key on both metas and the Vulkan tag. Every other vulkan backend gets its Mesa ICD drivers from .docker/install-base-deps.sh, which package-gpu-libs.sh then bundles; Dockerfile.audio-cpp calls neither and installs only libvulkan-dev and glslc, so the image would ship a Vulkan loader that finds no GPU. No CI job runs a vulkan image against real hardware, so that would have passed green and failed in users' hands. BUILD_TYPE=vulkan stays supported for local builds. Also note on the cublas entries that cuda-major-version now selects the architecture list and that cuda-minor-version and the base-image tag encode the same toolkit, and correct the stale entry counts on matrixEntryKey. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): build for Darwin Metal Bespoke C++ Darwin path like ds4 and privacy-filter: an includeDarwin matrix entry, a backends/audio-cpp-darwin make target, a gated workflow step, and the metal image entries plus metal/metal-darwin-arm64 capability keys in the backend gallery. The build script deliberately does NOT reassemble the package the way privacy-filter-darwin.sh does. It runs the backend's own `make package` and copies the result, so the Darwin package keeps the root-level layout the Linux one has: grpc-server, run.sh, the ggml objects and assets/ in one directory, with lib/ for the dylib closure. Hand-assembling would drop assets/, and assets/ is what makes the bundled: model paths resolve with nothing downloaded. The dylib walk is a full transitive closure rather than the single level ds4 and llama-cpp do, because Homebrew's grpc++ pulls libgrpc, abseil, upb, cares and OpenSSL that grpc-server does not link itself, and a level-1 walk ships a package that only works on a machine that already has Homebrew grpc. Two fixes folded in, both in the backend Makefile: - an EMPTY CUDA_MAJOR_VERSION fell through to the CUDA 12 architecture list, which contains 120a-real and so needs nvcc >= 12.8. A local BUILD_TYPE=cublas build on a 12.0-12.7 host failed to compile where upstream's documented default (native) worked. EMPTY now maps to native, 12 and 13 keep their lists, and any other non-empty value is an error on cublas builds. CI always passes a major, so CI is unaffected. - the Darwin branch now points CMake at Homebrew's keg-only libomp. AppleClang ships no OpenMP runtime and nothing is symlinked into /opt/homebrew, so FindOpenMP finds neither the library nor the header, and audio.cpp calls find_package(OpenMP REQUIRED) whenever ENGINE_ENABLE_OPENMP is on. Without the hint the macOS build would have died at configure time. If the keg is absent the build disables OpenMP instead of failing. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the Darwin fallbacks loud and the rpath walk complete Review follow-up on the Darwin Metal build. The OpenMP fallback was silent. If brew --prefix libomp ever comes back empty, CI produced a green Metal package with 108 #pragma omp directives across ~30 files compiled out, and clang says nothing about an ignored omp pragma without -Wsource-uses-openmp, so the only trace was one absent flag inside a set -x cmake line. That regression would have been blamed on Metal. It now warns. The @rpath arm of the dylib walk had no live candidate when it was written, on the reasoning that a Metal build links ggml statically. The OpenMP fix in the same commit made libomp.dylib one, and whether Homebrew records it as an absolute opt path or as @rpath/libomp.dylib is not observable from Linux. The walk now expands @rpath, @loader_path and @executable_path against the object's own LC_RPATH entries, and only fails when nothing on disk answers, printing the rpath list with the error so a failure on a machine nobody can attach to explains itself. Also: ADDITIONAL_LIBS now go through the closure rather than a bare cp, so they are deduplicated and their own dependencies bundled; build/darwin/lib is created explicitly instead of relying on package.sh pre-creating it; the libomp probe uses nested ifneq rather than $(and ...), which needs GNU make 3.81 and would otherwise expand empty and take the OFF branch on an older make; and -DOpenMP_ROOT is quoted like its CUDA sibling. Verified with a Linux harness that runs the script verbatim against a stubbed otool: a level-2 transitive dep, an @rpath dep reachable only through LC_RPATH, and an ADDITIONAL_LIBS dep are all bundled, a dependency cycle terminates, system libraries are skipped, the packaged tree has assets/ at the root beside grpc-server with the dylibs in lib/, and both failure paths exit non-zero. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make bundled: reachable from a model YAML resolve_model_path() tested the bundled: prefix on `candidate`, which prefers ModelFile and falls back to Model. LocalAI fills ModelFile by joining ModelPath onto the configured model string (pkg/model/loader.go, LoadModelWithFile), and only sets it from a managed artifact otherwise, so a model YAML saying `model: bundled:silero_vad` arrives as ModelFile "/models/bundled:silero_vad" and Model "bundled:silero_vad". The prefix therefore never matched through the normal load path: it matched only for a hand-written LoadModel call that left ModelFile empty, which is exactly how task 15 verified it, and every model YAML using the form failed with "model path does not exist: /models/bundled:silero_vad". Both fields are now checked, Model first, so the zero-download VAD path the package ships assets for is reachable the way it is documented. A caller that puts the form in ModelFile still works, so task 15's verification stands. Compiled clean; the runtime check could not run on this host, whose system libprotobuf/libre2 have gone missing (the pre-existing grpc-server binary no longer resolves its libraries either), so it wants a container run. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): advertise the backend and document its options Registers audio-cpp as preference-only in /backends/known: the family lives in GGUF metadata that an importer cannot read from a remote repo, and one repo hosts thirty families, so there is no honest auto-detect signal. Modality is a single string and the import form chips on a fixed key set, so it registers as tts with the other modalities named in the description rather than under an invented key the UI would bucket as "other". Adds a features page covering the option namespacing, the routing table per endpoint, the RPCs this backend declines and why, the bundled VAD path, the separation stem behaviour, and the family gotchas (supertonic needs the orig package; chatterbox advertises cloning and no plain tts; nemotron_asr defers its whole decode to finalize so live transcription emits nothing until the client half-closes, unlike higgs_audio_stt and voxtral_realtime). Every option name and family capability in it was read off the pinned upstream checkout. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): test resolve_model_path, and correct the family names The bundled: fix in 842443cd7 shipped without a test, which is how the bug got there: task 15 verified the form with a hand-written LoadModel that left ModelFile empty, and that is the one shape the server never produces. Four cases in streaming_driver_ctest, which already links loaded_model.cpp, pin the PRODUCTION shapes instead. The first fails against the pre-fix source (returns the joined /models/bundled:silero_vad); the other three are the branches the bundled: lookup now runs in front of and must fall through for. Three family names in the docs were the source directory rather than the registered family, on pages whose whole argument is that these names cannot be guessed: demucs is htdemucs (demucs/loader.cpp:22), roformer is mel_band_roformer (roformer/assets.h:15), and moss is TWO families, moss_tts_local and moss_tts_nano. The hyphenated ASR names are underscored to match, here and in the compatibility table. The supertonic dtype note claimed more than the evidence carries. The f16 abort is a local observation, identical through TTS and TTSStream; upstream's docs/gguf.md leaves the 16-bit column untested and records q8_0 as "No (unsupported weight dtype)", which says unusable rather than fatal. Both are still refused, because the allow list is what the family can run. Corrected in family_gate.h, family_gate.cpp and the docs together, since the docs inherited the wording from the code. The importers tripwire says in the file that it is a tripwire: it exercises no audio-cpp behaviour, and the registration assertion lives in backend_test.go. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: add audio.cpp models covering every served RPC One representative model per RPC group of the audio-cpp backend, plus the two bundled VAD models, which need no download at all because the assets ship inside the backend package. Every hash was computed with sha256sum on the downloaded file. Quantizations come from upstream's tested-status table in docs/gguf.md rather than a default of q8_0: supertonic ships the orig package (its q8_0 is recorded as an unsupported weight dtype and its f16 aborts in ggml_concat), and nemotron_asr and htdemucs ship f16 because their q8_0 builds are recorded with drift while 16-bit is a clean pass. Diarization and separation use the diarization and audio_transform usecases, not transcript: /v1/audio/diarization and /audio/transform filter the default model on FLAG_DIARIZATION and FLAG_AUDIO_TRANSFORM respectively, so a transcript flag would have hidden both models from their own endpoints. The forced aligner sets parameters.language, which the transcription endpoint uses as the fallback when no language form field is sent, because the family requires both a transcript and a language. All ten entries were run twice: once against the raw gRPC server, and once installed with local-ai models install and called through the HTTP endpoint. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: correct the audio.cpp entries' licenses Swept all ten entries against the real upstream named in audio.cpp's tools/model_manager.py rather than against the audio.cpp repo's own license. Three were wrong: supertonic apache-2.0 -> openrail weights come from mlx-community/supertonic-3-mlx, and both it and Supertone/supertonic are openrail citrinet apache-2.0 -> other pulled from NGC nvidia/nemo/stt_en_citrinet_256, governed by the NGC Terms of Use sortformer other -> cc-by-nc-4.0 nvidia/diar_sortformer_4spk-v1 is CC BY-NC 4.0, and the gallery already uses that exact string, so there is no reason to obscure a non-commercial bar The license field is one word, so citrinet and sortformer also gained a sentence saying why they are restricted. The other seven were confirmed correct against their sources. Also drops an unverified claim from the nemotron description. It said the model drives the realtime transcription session; that endpoint actually calls TranscribeStream, and the live RPC reaches LocalAI only through realtime_semantic_vad.go. Neither path was exercised here, so the description now states only the two calls that were. MarbleNet gains the NeMo upstream under urls: for parity with silero. No sha256, quantization, usecase or model choice changed. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(audio-transform): bound sample_rate, keep same-named uploads apart Four defects the whole-branch review found on the Go side, plus two comment corrections. sample_rate is a disk-exhaustion hazard. The branch added the `form:` tag that makes the field bind for the first time, so the resample path went from dead to live, and utils.AudioResample interpolates the int straight into ffmpeg's -ar with no bound. Measured with ffmpeg 7: -ar 999999999 on a 0.01 s clip writes 20 MB and exits 0, which scales linearly to the reported 3.9 GB for one second, into a GeneratedContentDir nothing sweeps, and convertStems repeats it once per separation stem. Clamped to 8000..192000 in the handler, before the temp dir and before the model is touched, and rejected with a 400 outside it. The low end was reported as "a 0-byte file". It is not: -ar 1 writes a 78-byte header with no audio behind it, whose declared data size still claims 70 bytes, so go-audio parses it as a 35 SECOND file and a size check does not see it. The guard therefore compares the declared data chunk against the bytes actually on disk, and AudioResample now fails rather than returning a WAV carrying nothing. Both parts of a transform request land in one temp dir, and the raw copy was named only after the client's basename, so `-F audio=@mic/clip.wav -F reference=@loopback/clip.wav` wrote "raw-clip.wav" twice. Since AudioToWavPreservingShape hardlinks an already-PCM16 WAV rather than copying it, the reference part's os.Create truncated the inode audio.wav pointed at: mic and reference came out identical, which makes an echo canceller null everything and return near-silence with a 200. The raw copy now carries the form field name. audio-cpp had no BackendCapabilities entry, so VoiceCloningForModel returned nil before it ever consulted the model's tts.voice_cloning override and every `voice: "profile:<id>"` request was refused with a 400, on a backend that ships audio-cpp-chatterbox whose family serves cloning and not plain TTS. Registered with its RPCs, usecases and the reference-audio contract, and deliberately without the 16 kHz mono fold, which its separation families cannot survive. GetBackendCapability was exact-match only, so every pinned gallery variant read as an unknown backend: vulkan-localvqe lost the 16 kHz mono fold that used to be unconditional and started failing inside LocalVQE, and the usecase gate does not stand in for it because BuildFilteredFirstAvailableDefaultModel returns early once the client names a model. Lookup now falls back to the meta name by stripping the gallery's hardware prefix and release-channel suffix, exact match first so nothing can be shadowed. Same class as #10945. Also corrected: the AudioTransformRequest comment claimed echo's binder falls back to the field name, which it does not in either direction (bindData binds ONLY tagged fields and `continue`s otherwise; `model` arrives from setModelNameFromRequest's c.FormValue). And the stable_audio `src` heap corruption caveat now lives on ElevenLabsSoundGenerationRequest, where the Go developer who would add the field can see it, instead of only in C++. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(audio-cpp): refuse a task pin the RPC cannot serve, and stop empty frames holding the lane The model's `task:` option is copied into the request shape by all nine handlers, which is correct, but resolve_route then replaced the RPC's candidate list with the pin WHOLESALE and never asked whether the pin was something that RPC routes to. One pin therefore bled across all nine surfaces, and because the family still supported the pinned task the result was a wrong 200 rather than an error. Reproduced live: nemotron with task:asr made Vad return 200 with zero segments after a full ASR decode, so 14 seconds of speech was reported as silence, and Diarize did the same; silero_vad with task:vad made AudioTranscription return 200 with empty text and four segments whose spans were VAD segments, which combined with response_format in {text,srt,vtt,lrc} building the body solely from Segments[].Text yields a well formed SRT of four timed EMPTY cues. It also contradicted the documented contract, that a family which cannot serve a request is refused rather than rerouted. A pin is now checked against the RPC's admissible task set before it is adopted, and the refusal names both the pin and the RPC. The set is derived from task_candidates with every shape flag set rather than restated, so a task added to an RPC's candidates cannot become inadmissible by omission. Every legitimate pin survives, and the test asserts all fifteen of them alongside the eight crossings that must not. The live watchdog was defeated by empty frames. idle.touch() ran on ANY message, before the has_audio and pcm.empty() filters, so a peer writing unset-oneof or zero-length frames faster than the window held the lane indefinitely while feeding the decoder nothing. There is one lane per model and one model per process, so that is a single client denying the whole backend, which is what the watchdog exists to prevent, and the thrown text already said "no audio frame arrived". The touch moved below the filters, which are now a named predicate so the distinction is testable rather than a call order nobody can see. Three comments corrected against measurement rather than reasoning: - CMakeLists claimed zero google::protobuf:: definitions remain in the executable. nm -C --defined-only reports 2515, and that is expected: they are generated code, sentencepiece::ModelProto's own _InternalParse among them. The claim that holds, and the one the ABI fix is actually about, is that no vendored protobuf RUNTIME is linked and ParseContext::ParseMessage is UNDEFINED in the executable, resolving to libprotobuf.so. - refuse_cloning_without_a_clip's "cannot misfire" paragraph had its reasoning backwards. Routing picks VoiceCloning as the FALLBACK when there is no clip, which is the case being caught; chatterbox, which ships in the gallery, advertises clon and no tts at all, so every voice-less request lands there. - audio_units read "2.1 min at 96 kHz" for index 11289602, which is 1.96 min. 2.1 min is 96 kHz's OWN first failure at 12288002. Both were remeasured and the note is now a per-rate table. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(audio-cpp): exclude the upstream checkout from the C++ gate, harden the darwin walk run-unit-tests.sh pruned */llama.cpp/* but not */audio.cpp/*. It is safe today only by luck: upstream's 44 tests all put "test" at the FRONT of the filename (17 test-*.cpp, 27 test_*.cpp, zero *_test.cpp), so the glob misses every one of them, and nothing enforces that. This gate runs on every PR for every backend and compiles each match as a standalone translation unit with nothing but nlohmann/json on the include path, so the day upstream adds or renames one test the gate goes red repo-wide on an Apache-2.0 file nobody here wrote. audio-cpp-darwin.sh now logs the raw otool -L output and the parsed LC_RPATH list unconditionally, before the walk. Both awk filters in that script assume a column layout nobody working on this can observe, since it runs only on the CI Mac, and a green first Darwin run proves nothing about the assumption: an awk that silently matched nothing yields an empty dependency list, which reads exactly like "no non-system dependencies" and packages happily. Both filters otherwise feed process substitutions, so their input never reached the log. It also lists every symlink in the package and fails on one that cannot resolve inside the image. A dangling link does not fail anything else here, because every assertion tests with -e, which follows links; it fails at dlopen on a user's Mac. Links are NOT banned outright, which the review suggested but which would break the libggml.dylib -> libggml.0.dylib chain the `cp -a` above exists to preserve. What is banned is a link that resolves on the build host and will not resolve in the image: a broken one, or an absolute one pointing outside the package. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(audio-cpp): drop em dashes from the audio-cpp capability entry Follow-up to a84b3c4b9, no behaviour change. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): key the voice-cloning model rule on the resolved backend Making GetBackendCapability strip the gallery hardware prefix and release channel fixed pinned variants of /audio/transform, but VoiceCloningForModel kept keying its per-backend switch on the caller's spelling. A pinned name therefore resolved the capability by stripping and then missed every case in the switch, falling through to the permissive default: cuda12-vibevoice-cpp advertised voice cloning for the realtime 0.5B model, metal-coqui for tacotron2, cuda12-crispasr for a pure ASR model, cpu-qwen3-tts-cpp for CustomVoice. Each of those is a model that cannot clone, so /v1/audio/speech accepted a profile: voice it had to fail on inside the backend rather than rejecting it with a 400, and the UI advertised the capability too. resolveBackendCapability now returns the key the entry was found under, and callers that branch on backend identity use that key instead of the name they were handed. The exact-match-first order is unchanged, so a backend genuinely registered under a variant-looking name still keys on its own name. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * gallery(audio-cpp): declare audio_transform on the chatterbox entry Chatterbox advertises VoiceCloning AND VoiceConversion (src/models/chatterbox), and the entry's own description already said so, but known_usecases listed only tts. /audio/transform selects its default model by FLAG_AUDIO_TRANSFORM, so voice conversion was reachable only by naming the model explicitly and was invisible to every usecase-driven surface. It is the one audio.cpp task with a shipped gallery model and no way to find it. Verified against the real model rather than inferred from the capability list: AudioTransform with chatterbox-q8_0, speech as audio_path and a speaker clip as reference_path, returns a 5.08 s 24 kHz mono WAV at -25.5 dB mean and zero stems, which is the single-output shape voice conversion should have. The description now says which endpoint reaches that half and warns that installing this next to a source-separation model gives /audio/transform two candidates, so the model should be named rather than defaulted. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * gallery(audio-cpp): add voice-design and singing-voice-conversion entries Two of the three audio.cpp task kinds that had no gallery model now have one. Both were driven end to end against the real weights through the backend before being written, not inferred from the capability tables. audio-cpp-irodori-voicedesign covers vdes. TTS carrying `instructions` routes to the vdes task, so the voice is described in words rather than supplied as a clip. Verified: "a calm elderly woman speaking slowly with a warm, gentle tone" over an 8.76 s 48 kHz mono render at -16.8 dB mean, and a closed-loop citrinet pass recovers the sentence with the accent drift expected from a Japanese-first model read by an English recogniser. audio-cpp-seedvc-singing covers svc, and pins task:svc because nothing else can reach it. seed_vc advertises svc and ordinary voice conversion, no request signal means "this input is singing", and auto-routing resolves the tie to voice conversion every time. Verified with the pin: 5.04 s 44.1 kHz output whose closed-loop citrinet transcription is exact. s2s deliberately has no entry, and the reason is not effort. miocodec is the only upstream family whose speech-to-speech route needs no text, and it returned audio with correct duration and level but no recoverable speech in four independent attempts: the stale build, v2 q8_0, v2 orig (the variant upstream records as a clean Pass), both tasks, and matched 44.1 kHz inputs on both sides. vevo2's route refuses with "Vevo2 text/prosody route requires text_input or target_text", and session.cpp:897 fills target_text only from request.text_input, which AudioTransform has no field to carry. The same vevo2 weights convert voice correctly through the default route with an exact ASR round trip, so the model and the plumbing are both healthy; it is the s2s route specifically that this RPC cannot express. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * backend(audio-cpp): carry transform text through params, add the s2s entry AudioTransform is audio-in / audio-out and its proto message has no text field, but not every task it routes to is audio-only. vevo2's speech-to-speech route is a text and prosody route: session.cpp:897 fills refs.target_text from request.text_input and nowhere else, and the run refuses without one with "Vevo2 text/prosody route requires text_input or target_text". The params map is the only channel this RPC has that reaches the engine, so the text travels through it and apply_transform_text_input unpacks it after the params have been copied into task.options. Before this, s2s was not awkward to reach through /audio/transform, it was unreachable, and it was the last audio.cpp task kind with a real model and no way to get to it. target_text is canonical and text is its alias, the order vevo2's own option table declares them in, so a request setting both gets the canonical one rather than whichever the map happened to store first. An empty value falls through to the next candidate instead of ending the search. language rides along only when a text was found: on its own it conditions nothing, and manufacturing a text_input for it would route a plain separation request carrying a language hint through the text path. The keys are left in task.options rather than erased, because vevo2's loader advertises target_text as a request option and a family reading it there keeps working. Nine tests, all confirmed failing on behaviour against a stub that returned false before the implementation was written. Verified end to end afterwards: vevo2-q8_0 with task:s2s and params[text] returns a 5.12 s 24 kHz output whose closed-loop citrinet transcription is exact, and htdemucs separation with no text param still returns its four stems, with and without params[stem]. audio-cpp-vevo2-speech-to-speech ships that route. Every audio.cpp task kind with a loadable family now has a gallery entry; spk remains the only gap and has no family upstream at all. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * docs(audio-cpp): document params[text] and the pinned transform tasks The text channel and the two task pins are both invisible from the endpoint contract alone: nothing in the AudioTransform form tells a reader that a speech-to-speech model needs the line it is resynthesising, and nothing says that asking for singing voice conversion without task:svc silently gets plain voice conversion instead. Both are the kind of thing a user only discovers from a refusal or, worse, from output that looks right and is not. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(utils): annotate the two G304 sites this branch introduced gosec flags os.Open on a variable path, and both new call sites in ffmpeg.go are its alerts on this PR. Neither is reachable by an outside caller: isPCM16Wav opens the exact path it is about to hand ffmpeg as input, which in the upload path is a server-created temp file named from path.Base of the client name so no traversal survives, and wavAudioBytes opens AudioResample's own dst, a name this package derives from src and has just had ffmpeg write. Annotated in the repo's existing style rather than restructured, with the reason spelled out, because a bare suppression is worth nothing to the next reader. The three other G304 sites in this file, in passthroughWAV and isTargetWav, predate the branch and are left untouched. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(audio-cpp): build arm64 with gcc-14 for the armv9.2 SME variants The arm64 CPU image failed to build: cc1: error: invalid feature modifier 'sme' in '-march=armv9.2-a+dotprod+fp16+sve+i8mm+sve2+sme' ggml's CPU_ALL_VARIANTS table includes armv9.2 variants compiled with +sme, and Ubuntu Noble's default gcc-13 rejects that feature modifier. Every entry in the table has to compile even though a host only ever dlopens the one its own CPU supports, so a single unbuildable variant fails the whole image. gcc-14 accepts it, which is exactly the fix llama-cpp already carries in .docker/llama-cpp-compile.sh; this is the same problem reached by a different Dockerfile. Applied to every arm64 BUILD_TYPE rather than to the CPU one alone, and that differs from llama-cpp on purpose. llama-cpp needs it only for its pure-CPU image because its GPU builds run llama-cpp-fallback, which builds no variant table. This backend's Makefile turns ENGINE_ENABLE_CPU_ALL_VARIANTS on for every non-Darwin build, GPU included, so an arm64 GPU image would hit the identical error. The matrix has no arm64 GPU entry today, which is precisely why gating on an empty BUILD_TYPE would leave the trap armed for whoever adds the first one. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-30 12:11:56 +02:00
localai-org-maint-bot	9bfd71387b	feat(stores): add Valkey Search vector store backend (#11196) ... * feat: add Valkey Search vector store backend Add a new built-in Go gRPC store backend 'valkey-store' that implements the four Stores RPCs (Set/Get/Delete/Find) against the Valkey Search module (FT.*) using the pure-Go github.com/valkey-io/valkey-go client. It is selected via the existing per-request 'backend' field on /stores, so there is no proto or HTTP API change, and it mirrors the in-memory local-store while adding persistence across restarts and opt-in HNSW. Each vector is a Valkey HASH keyed by hex(little-endian float32); the index is created lazily on first Set (FLAT+COSINE by default), cosine similarity is derived as 1-distance, and namespaces get a collision-resistant token. Includes unit tests (valkey-go mock) and env-gated integration tests against valkey/valkey-bundle, plus build/matrix/gallery wiring and docs. Assisted-by: Kiro:claude-opus-4.8 golangci-lint Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Address review feedback: recover persisted index dimension, harden Find - Load now recovers the persisted vector DIM from FT.INFO (not just index existence), so a post-restart Set/Find validates against the real DIM instead of silently re-learning a wrong one and dropping mismatched vectors from the index. This also restores Find's dimension check after a restart. - StoresFind treats a dropped/missing index as an empty store (empty result, no error) and clears the stale indexCreated flag, matching local-store's empty-store behaviour. - StoresSet reuses checkDims for its per-key length check so the four RPCs share one dimension-guard implementation. - Add unit tests for FT.INFO dimension recovery, loadIndexState, and the dropped-index Find path. Assisted-by: Kiro:claude-opus-4.8 Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Address review feedback: TLS ServerName/CA, Find nil-check, config fail-fast Addresses external review comments on the valkey-store backend: - StoresFind now rejects a nil/empty query Key before dereferencing it, so a malformed gRPC request can no longer panic the backend. - TLS: derive ServerName (SNI) from the VALKEY_ADDR host so certificate verification works for IP-addressed endpoints, and add VALKEY_TLS_CA_CERT (custom CA bundle) and VALKEY_TLS_SKIP_VERIFY (testing-only) knobs. - Config integer parsing now fails fast on a malformed value (e.g. VALKEY_HNSW_M=1x6) instead of silently defaulting, matching the fail-fast behaviour of the index-algo/distance-metric validation. - Add VALKEY_DB (SELECT n) support for logical-DB isolation. - Cap the human-readable part of a namespace token at 64 chars so a very long model name cannot produce an unbounded key prefix / index name (the appended short hash keeps distinct namespaces collision-free). - Document the KNN-query injection-safety invariant (fields are constants) and why StoresGet uses a single aggregate DoMulti deadline for reads. - Unit tests for the Find nil/empty-key guard, fail-fast HNSW parsing, and VALKEY_DB parsing/validation; docs + .env updated for the new vars. Assisted-by: Kiro:claude-opus-4.8 golangci-lint Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Address review feedback: configure valkey-store via model config richiejp asked that the valkey-store backend take its configuration from a model config rather than process-wide VALKEY_* environment variables, so multiple stores can each have their own Valkey config within one LocalAI process. This removes every env access from the backend and routes config through the model-config seam every other backend uses. - config.go: loadConfig(opts *pb.ModelOptions) now parses the model config `options:` list (key:value strings, split on the first ':') instead of os.Getenv. Option keys mirror the old VALKEY_* names without the prefix (addr, index_algo, distance_metric, ...). Defaults, fail-fast validation and the mandatory client name are unchanged. - store.go: Load threads opts into loadConfig; TLS comments/errors renamed off the VALKEY_* names. - core/backend/stores.go: StoreBackend and NewVectorStore take a *config.ModelConfigLoader, resolve the per-store ModelConfig by store name, and pass its Options (and Backend when unset) to the backend via WithLoadGRPCLoadModelOpts. No config -> default backend + built-in defaults, preserving the zero-config experience. - Endpoints/routes/application: thread the config loader to StoreBackend. - Unit + integration tests: configure via options; the integration test passes addr through the model-config path (VALKEY_ADDR is now only the test harness locating the server). - docs + .env: document the model-config options, drop the env var table. Assisted-by: Kiro:claude-opus-4.8 Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Remove valkey-store informational comment from .env The backend is configured via model config, not env vars — the comment was unnecessary noise in .env. The configuration is already documented in docs/content/features/stores.md. Signed-off-by: Daria Korenieva <daric2612@gmail.com> * feat(valkey-store): gate Load on NamespacePrefix to refuse autoload probing Mirror local-store's pattern: reject model names without store.NamespacePrefix so the model loader's greedy autoload probe cannot bind an arbitrary model name to the vector store backend (the #9287 failure mode). Also adds unit tests for the gate covering: prefixed namespace, prefix alone, unprefixed model name, empty model, and nil opts. Signed-off-by: Daria Korenieva <daric2612@gmail.com> * feat(valkey-store): add username_env/password_env credential indirection Add support for resolving Valkey credentials from environment variables named in the model config, mirroring cloud-proxy's api_key_env pattern. This keeps secrets out of model YAML files and lets distinct store configs each reference their own credentials. Options: username_env / password_env name the env var holding the value. The direct username / password options still work and take precedence when both are set (backward compatible). Includes 5 unit tests and updated stores.md documentation. Signed-off-by: Daria Korenieva <daric2612@gmail.com> * fix: correct rebase artifacts in backend-matrix.yml and Makefile Fix two issues introduced by the conflict-resolution script during the rebase onto master: 1. .github/backend-matrix.yml: valkey-store entries were merged INTO the cloud-proxy entries (duplicate keys in same YAML map items) instead of being separate list items. This broke cloud-proxy Linux builds and the cloud-proxy darwin entry lost its build-type/lang. Fixed by making them standalone entries and restoring cloud-proxy exactly as on master. 2. Makefile: duplicated .NOTPARALLEL and docker-build-backends lines. Collapsed to single lines that are master's current content plus the valkey-store additions. Also adds the three optional pickups from #10801: - /valkey-store in .gitignore (the built binary) - valkey-store row in docs/content/reference/compatibility-table.md - valkey-store line in backend/README.md Signed-off-by: Daria Korenieva <daric2612@gmail.com> --------- Signed-off-by: Daria Korenieva <daric2612@gmail.com> Co-authored-by: Daria Korenieva <daric2612@gmail.com>	2026-07-29 20:12:29 +02:00
Richard Palethorpe	9058a2bb46	feat: Add 3d generation UI/API and trellis2cpp backend (#10979) ... * feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint Adds the plumbing for image-conditioned 3D asset generation (binary glTF / GLB output), modeled on the video generation path: - backend.proto: Generate3D RPC + Generate3DRequest (staged image src, glb dst, seed/step/cfg_scale/texture_steps, quality and background enums, params map for backend-specific extras) - pkg/grpc: thread Generate3D through client, server, embed, base and the backend interfaces; connection-evicting and distributed-node wrappers (in-flight tracking + file staging) included - core/config: FLAG_3D usecase (guessed only for the trellis2cpp backend), '3d' canonical usecase string mapped to the Generate3D method, and a '3d' output modality - REST: POST /v1/3d/generations (+ unversioned alias) returning OpenAIResponse with a /generated-3d URL or b64_json; conditioning image accepted as URL, base64, or data URI; quality/background validated at the edge; .glb served as model/gltf-binary - auth: '3d' route feature (default ON); /api/instructions entry Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(trellis2cpp): add the trellis2.cpp image-to-3D backend Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2, pbr-textures branch) as a Go+purego backend, following the stablediffusion-ggml pattern: - backend/go/trellis2cpp: purego bindings to the flat C ABI (v9, asserted at startup), eager pipeline load with model-set validation (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured exactly like the upstream demo), Generate3D via t2_generate + t2_bake_glb writing a binary glTF to dst. Weight-free unit tests cover resolution/validation/param mapping — CI never downloads the multi-GB GGUF set or runs inference. - CPU SIMD variants build into per-variant directories (the shared libggml sonames collide across variants, unlike sd-ggml's flat renamed-.so scheme); run.sh picks one via /proc/cpuinfo. - CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta + latest/master image entries, bump_deps tracking of the pbr-textures branch, changed-backends.js mapping, top-level Makefile targets. - Importer: auto-detects trellis GGUF repos/URIs (registered before llama-cpp so the .gguf match isn't stolen) and expands any trellis URI to the full 10-file component set spanning the three LocalAI-io HF repos. - Gallery: trellis2-4b (full PBR + 1024 cascade) and trellis2-4b-geometry (512 untextured) with verified sha256s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(ui): 3D generation page with native GLB viewer and IndexedDB history Adds a Studio tab + /app/3d page for the new image-to-3D endpoint: - GlbViewer ports the trellis2cpp demo's dependency-free WebGL2 renderer (quaternion trackball, metallic-roughness PBR, ACES, hidden-line wireframe with a bounded index budget) and pairs it with a minimal GLB parser for the two forms t2_bake_glb emits — dense vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as normalized integers) and the opt-in UV-atlas textured form. Parsing happens before any GL so stats and errors render without WebGL2. - use3DHistory stores past generations (params, input thumbnail, and the GLB blob itself) in IndexedDB with keep-newest-20 eviction — GLBs are multi-MB binaries localStorage can't hold — and the page offers a download button for the active GLB. - Wiring: CAP_3D capability constant (FLAG_3D — the exact string /api/models/capabilities serves), threeDApi, router entries, Studio tab, vite dev proxy, en locale keys. - e2e: render-smoke entry plus a focused spec that feeds a real one-triangle vertex-PBR GLB through the parser/viewer and exercises IndexedDB persistence, selection, deletion, and API errors. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(3d): address API correctness and UX issues Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it. Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add previewable print remeshing Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download. Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(trellis2cpp): centralize remesh dependency pins Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement Generate3D stub for new proto RPC The Generate3D RPC added to backend.proto for the trellis2cpp backend made tonic's generated Backend trait require generate3_d, breaking the kokoros-grpc build. Return unimplemented like the other unsupported modalities. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>	2026-07-29 16:15:04 +02:00
mudler's LocalAI [bot]	4baa36ddd8	feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100) ... * feat(backend): add vllm-cpp text-generation backend (vllm.cpp) Wrap https://github.com/mudler/vllm.cpp - the LocalAI-team from-scratch C++20 port of vLLM (paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, no Python at inference) - as a Go gRPC backend over its stable C ABI (ABI v2) via purego. Backend (backend/go/vllm-cpp): - Load -> vllm_engine_load: accepts a .gguf file or a config.json model dir (anything else is refused, satisfying the greedy-probe rule); context_size maps to max_model_len, options block_size/num_blocks/max_num_seqs size the KV cache and scheduler admission. - Predict -> vllm_complete (blocking); PredictStream -> vllm_complete_stream with the per-delta C callback bridged into the gRPC stream. The backend embeds base.Base (not SingleThread): concurrent requests batch continuously in the engine's shared AsyncLLM scheduler. - PredictOptions.Grammar -> the ABI's structured_grammar (GBNF), giving grammar-constrained tool calling at parity with llama-cpp; the ABI also exposes JSON-schema/regex/choice constraints. - Hand-mirrored POD structs with layout locked by unit tests (unsafe.Offsetof vs the C offsets) and a runtime vllm_abi_version gate. - One portable library per platform (vllm.cpp uses per-file SIMD tiers with runtime dispatch), so no avx/avx2/avx512 variant builds. Wiring: - backend-matrix: CPU amd64+arm64 (per-arch + manifest merge), CUDA 12/13 amd64 (120a;121a Blackwell fat binary), L4T arm64 (121a, GB10/DGX Spark - the runtime-proven GPU target), Vulkan amd64, and Darwin arm64 Metal. - backend/index.yaml meta + 12 image entries (latest/development x cpu, cuda12, cuda13, l4t, vulkan, metal); bump_deps registration for the VLLM_CPP_VERSION pin; root Makefile registration; test-extra runs the unit specs (pure Go, no engine build). - Importers: preference-only swaps - llama-cpp (GGUF) and vllm (safetensors) advertise vllm-cpp via AdditionalBackends and emit backend: vllm-cpp without tokenizer templating (the C ABI takes the FINAL prompt; templating and tool parsing stay LocalAI-side). No auto-detect importer. - Docs: backends list, top-level README maintained-engines table, compatibility table. Verified: 20/20 Ginkgo specs against the real pinned engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU - blocking + streaming parity, greedy determinism, stop words, GBNF-constrained generation, and 4 concurrent streams; plus a dlopen/ABI-gate smoke of the built gRPC server binary. Upstream ABI v2 + production structured-output wiring landed as mudler/vllm.cpp@86013f3. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vllm-cpp): ride the autoparser code path - engine-side chat templating and tool engagement (ABI v3) The backend now implements AIModelRich (PredictRich / PredictStreamRich) over vllm.cpp's ABI v3 chat entry points, so chat and tool calling ride the SAME code path as the llama.cpp autoparser: the ENGINE renders the model's chat template, decides when a tool call engages, and parses it - LocalAI receives pre-parsed ChatDelta / ToolCallDelta protos exactly as it does from llama-cpp. - With use_tokenizer_template + structured Messages, PredictOptions lowers to ONE OpenAI chat request JSON (messages, tools, tool_choice, sampling, stream_options.include_usage) for vllm_chat / vllm_chat_stream. tool_choice auto lowers engine-side to a LAZY structural-tag decode constraint - free text until the model emits the tool trigger, then the call is grammar-constrained; required/named force a call. Tool output is parsed by the engine's streaming Hermes-style parser; each chat.completion.chunk maps onto ChatDeltas (content / reasoning_content / tool_calls) which the host already prefers over Go-side tag extraction. Without structured messages the plain path (LocalAI templating + optional GBNF grammar) applies unchanged. - The engine resolves the chat template from the GGUF tokenizer.chat_template metadata (or tokenizer_config.json); templates beyond its minja subset - e.g. the full Qwen3.5 namespace()/macro template - degrade engine-side to a Hermes-aware fallback prompt (tools schemas + <tool_call> instruction) with a stderr witness, so structural-tag engagement keeps working. - Importers now emit the same config shape as llama-cpp for vllm-cpp (use_tokenizer_template: true, no-grammar autoparser flow); only the llama-cpp-specific use_jinja option and the vllm-python parser options are dropped. - Pin bumped to mudler/vllm.cpp@aaed7ec (ABI v3 + chat-prompt resolution). Verified against the real engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU: full suite green - blocking chat, streaming deltas concatenating byte-equal to the blocking answer, a REQUIRED tool call returning schema-valid arguments JSON, and an AUTO run where the engine itself engages get_weather and streams parsed tool deltas; plus unit specs for the request lowering, chunk->ChatDelta mapping, and the C struct mirrors (ABI gate now v3). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vllm-cpp): ABI v5 - engine-side parser selection for 30 tool dialects + reasoning Bump the vllm.cpp pin to the autoparser-parity engine: 30 tool-call dialects (every pure-text parser in the pinned vLLM registry, each ported 1:1 with its upstream tests), 7 reasoning parsers, google/minja as the template renderer (the full Qwen3.5 template now renders engine-side), per-family structural tags (tool_choice required/named compiles the model's NATIVE syntax where expressible), and template auto-detection for both parser axes. Backend changes: - cModelParams mirrors ABI v5 (tool_parser + reasoning_parser fields, layout-locked by the offset tests; ABI gate now v5). - New model options tool_parser:<name> / reasoning_parser:<name> pass through to the engine; unset means template auto-detection (18-row tool marker table; [THINK]->mistral, <think>->think_auto for reasoning); "none" disables the reasoning split; unknown names fail the first chat call. - Chat chunks parse the `reasoning` field (the pin renamed reasoning_content), flowing into ChatDelta.ReasoningContent which the host already prefers. Live e2e against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU, full suite green: the real chat template renders (no more fallback), reasoning auto-detection picks think_auto so markerless answers stay pure content (the live run caught the deepseek_r1 content-swallow upstream and drove the think_auto fix), required tool_choice returns schema-valid arguments, auto tool_choice engages engine-side and streams parsed deltas, and blocking/streaming stay byte-identical. Turn latency also dropped (proper template EOS behavior). Upstream program landed as mudler/vllm.cpp 86013f3..5fffe7e (ABI v2-v5, minja, parser waves B1/B2/B4, reasoning seam, structural-tag registry, think_auto). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(vllm-cpp): bump the engine pin to the ENG-wave close-out mudler/vllm.cpp@df8909b: the six engine-backed vLLM tool-parser families (qwen3-coder/xml/mimo, kimi_k2, glm45/47, minimax_m2, gemma4, seed_oss) text-reimplemented from their wire formats and held to the upstream test suites - 39 registered dialects; the pinned vLLM registry is now covered except the three Rust/Harmony-backed families, descoped by decision. kimi_k2 also gains a full native structural-tag builder; four new template auto-detection rows land with test-pinned ordering. Full backend e2e re-run green against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): add the vllm-cpp-development gallery meta The gallery grew the twelve latest/development image entries but was missing the separate vllm-cpp-development meta (own capabilities map targeting the -development image names), which every backend ships so the development gallery resolves per-platform. Validated: all capability targets in both metas resolve to existing entries, and every image URI's tag suffix matches a backend-matrix build. Also full-stack verified in this change's context (single-node local-ai from this branch, locally-built backend under --backends-path, Qwen3.5-2B GGUF): /v1/chat/completions non-stream (clean content + usage), streaming (SSE deltas), tool_choice auto engaging get_weather engine-side with schema-valid arguments and finish_reason=tool_calls, and streamed tool-call deltas in the standard name-first cadence. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): repair the CI backend builds - gcc-14 -Werror + fat-arch Triton Two distinct failures took down all five vllm-cpp backend builds on the PR: 1. gcc-14 (ubuntu:24.04 CI images; the local toolchain is gcc-13) fails the engine build with -Werror=maybe-uninitialized in InputBatch::condense - a false positive through a staging std::optional's raw storage. Fixed upstream (mudler/vllm.cpp@61f3e85) by moving slot-to-slot directly; verified BOTH ways under dockerized g++-14.2 (unfixed reproduces CI's two diagnostics exactly, fixed compiles clean) with the engine's behavior suites green. Pin bumped to that sha. 2. The amd64 CUDA builds died at CMake configure: the vendored Triton-AOT cubin trees are per-arch and the engine refuses -DVLLM_CPP_TRITON=ON on a multi-arch (120a;121a) fat build unless pinned to one tree, which would be unsound for the other arch. Triton is now enabled only on the single-arch arm64/GB10 build (where the cubins matter); the fat amd64 binary uses the engine's non-AOT GDN path. Backend e2e re-run green at the new pin (Qwen3.5-2B on CPU, full suite). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): cuda-12 images cannot compile compute_121a - target 120a only The second CI round surfaced a CUDA-version constraint: the cuda-12 (12.8) image's nvcc rejects 'compute_121a' (GB10 arch support landed with CUDA 13), killing the amd64 cuda-12 build at nvcc. Gate the architecture list on CUDA_MAJOR_VERSION (exported by Dockerfile.golang): cuda-12 builds consumer Blackwell 120a only, cuda-13 keeps the 120a;121a fat binary, arm64/l4t (cuda-13) keeps single-arch 121a with the Triton cubins. GB10 is arm64, so the amd64 cuda-12 image never served it - no capability change. Verified by Makefile dry-run variable dumps for all three combinations (cuda12 -> 120a; cuda13 -> 120a;121a; cpu -> CUDA off). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): drop the cuda-12 variant - the engine needs the CUDA 13 toolchain Third CI round, third layer: with the arch list already narrowed to 120a, the cuda-12 (12.8) build still dies in ptxas compiling the sm_120a NVFP4 MMA kernels ("Vector type too large, exceeds 128 bit limit") - the Blackwell fp4 path genuinely requires the CUDA 13 toolchain, and vllm.cpp supports Blackwell-family GPUs only. Shipping a cuda-12 image without the fp4 kernels would be a crippled build of an engine whose whole GPU story is fp4, so the variant is dropped instead: - backend-matrix: cuda-12 vllm-cpp entry removed (cuda-13 amd64, l4t arm64, cpu, vulkan, metal remain). - gallery: cuda12 image entries removed; the nvidia capability now resolves to the cuda13 image in both metas; the nvidia-cuda-12 key is dropped so older-driver hosts fall back to the CPU image instead of an unrunnable one. - backend Makefile: BUILD_TYPE=cublas under CUDA_MAJOR_VERSION=12 now fails fast with a clear message; cuda-13 keeps the 120a;121a fat binary and arm64/l4t keeps 121a with the Triton cubins. Verified: Makefile branch dumps for all four combinations (cuda12 loud error, cuda13 fat, arm64 121a+Triton, cpu off), YAML parses, matrix filter tests green, gallery capability targets all resolve. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): forward multi-turn tool identity and reasoning to the engine chatRequestJSON dropped Message.ToolCallId and Message.Name on role="tool" replies and Message.ReasoningContent on assistant history, so a second turn after tool execution reached the engine's chat template without the fields that bind a tool result to the call it answers. Forward all three (present-only, matching the OpenAI wire shape) and pin vllm.cpp to 6a0bd3e7, where ChatMessage parses/round-trips tool_calls, tool_call_id, name and reasoning and the minja adapter exposes them to the template context. Adds the round-trip request-lowering spec (user -> assistant tool_call -> tool reply -> lowered request) and re-ran the gated e2e suite against the new engine pin with a real Qwen3.5 GGUF: chat, reasoning split, streaming parity, required-tool and auto-tool cases all green. Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): bump vllm.cpp for the darwin arm64 i8mm build fix The darwin-metal CI job was the first build to compile the engine's arm CPU-quant files on macOS and hit their Linux-only <asm/hwcap.h> / <sys/auxv.h> includes. vllm.cpp 9e1c9025 detects i8mm per-OS (auxv on Linux, sysctl on Apple Silicon) with kernels untouched. Gated e2e suite re-run green against the new pin with a real Qwen3.5 GGUF. Assisted-by: Claude Code:claude-fable-5 [Bash] [Read] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): darwin build - bound cmake parallelism when nproc is absent The macOS runners have no nproc, so JOBS evaluated empty and `cmake --build -j$(JOBS)` became bare `-j`: unlimited clang jobs on a 3-core/7GB Mac, which swap-thrashed until the 6h GHA timeout (the log shows "nproc: Command not found" and 7+ concurrent clang processes being reaped at the cutoff). Use the same portable fallback chain as the other darwin backends: nproc, then sysctl hw.ncpu, then 4. Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-26 23:04:48 +02:00
mudler's LocalAI [bot]	2d889e61a6	feat(backend): add magpie-tts-cpp text-to-speech backend (#11115) ... * feat(backend): add magpie-tts-cpp text-to-speech backend Add a Go + purego backend wrapping the magpie-tts.cpp ggml port of NVIDIA's Magpie TTS Multilingual 357M (encoder + autoregressive decoder over NanoCodec tokens), producing 22.05 kHz mono audio in 5 baked voices (Aria, Jason, John, Leo, Sofia; case-insensitive names or indices 0-4) across 9+ languages from a single self-contained GGUF. Mirrors qwen3-tts-cpp / moss-tts-cpp: dlopen the static-ggml shared library, bind the flat magpie_tts_capi_* C-API via purego (no local C shim needed, the upstream .so exports it directly), and serve the gRPC TTS + TTSStream methods behind base.SingleThread (the C context is not reentrant across synthesize calls). The backend CMakeLists translates the Makefile's -DGGML_{CUDA,METAL,VULKAN,HIP} flags into upstream's MAGPIE_GGML_* toggles (upstream FORCE-overwrites the ggml cache entries from those), pinned to magpie-tts.cpp v0.1.1 (e3f3dd1ebe22b64e7405f93b519f2d1930712568), which statically links ggml into libmagpie-tts.so (ldd shows only system libs). Wires the full registration: backend-matrix.yml (CPU amd64/arm64, CUDA 12/13, Intel SYCL f16/f32, Vulkan amd64/arm64, ROCm, NVIDIA L4T + L4T CUDA 13, and Darwin metal), backend/index.yaml metas and image entries, the root Makefile build targets, the changed-backends backend-filter path mapping, the bump_deps auto-bump matrix, a test-extra per-backend smoke job, the /backends/known pref-only importer entry, the backend capabilities map (TTS + TTSStream, no voice cloning), and the README / compatibility-table docs rows. Verified locally: unit + e2e Ginkgo suites pass against the real q8_0 GGUF (22.05 kHz mono WAV, RMS > 0.01), a live gRPC LoadModel + TTS round-trip returns valid non-silent audio, and the pre-commit gates (make lint, make test-coverage-check) pass, run manually with LOCALAI_TEST_HTTP_PORT overriding the locally-occupied 9090. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * gallery: add magpie-tts-cpp model entries (q8_0 + f16) Add the Magpie TTS Multilingual 357M GGUFs from mudler/magpie-tts.cpp-gguf to the model gallery: q8_0 (~624 MB, near-lossless, fastest decode, recommended) with an f16 (~784 MB) variant, both served by the magpie-tts-cpp backend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * magpie-tts-cpp: bump pin to rewritten upstream v0.1.1 SHA Upstream history was rewritten to purge accidentally committed build artifacts; v0.1.1 now resolves to 6f7696cf. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-25 08:38:48 +02:00
mudler's LocalAI [bot]	05e16e0fa8	chore: remove local pre-commit gates (#11116) ... Remove the versioned pre-commit hook and its installer while retaining CI coverage and conformance checks. Assisted-by: Codex:gpt-5 Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-25 01:19:32 +02:00
mudler's LocalAI [bot]	6e52d0c2ef	fix(ci): rebuild backends when shared build inputs change (#10975) ... The backend matrix path filter only matched files under a backend's own directory, so a change to shared build infrastructure rebuilt nothing at all: an empty matrix, every job green, and the change reaching no image. PR #10946 fixed scripts/build/package-gpu-libs.sh shipping a partial 4-of-8 cuDNN library set, which mixed versions with the venv's pip cuDNN and produced CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH at inference time. It merged 1h48m after the weekly full-matrix cron had already run, so no backend image ever received the fix and nothing signalled that it had been un-shipped. Add a SHARED_BUILD_INPUTS table mapping each shared path to the narrowest set of matrix entries it can honestly invalidate, plus a generic rule for backend/Dockerfile.<x> (which each entry already names). A full matrix is 417 Linux + 56 Darwin builds, so package-gpu-libs.sh now rebuilds the 176 Python entries rather than everything. Unclassified files under scripts/build/ fall back to a full rebuild deliberately: over-building is recoverable, silently shipping nothing is not. Extract the filtering logic to scripts/lib/backend-filter.mjs so it can be unit-tested without bun, js-yaml or a GitHub API round-trip, and run those tests from the existing lint workflow via `make test-ci-scripts`. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-20 13:48:12 +02:00
mudler's LocalAI [bot]	963c637130	fix(gpu-libs): bundle cuDNN only where it is used, and complete it when it is (#10946) ... cuDNN 9 is a dispatcher (libcudnn.so.9) plus seven sublibraries the dispatcher dlopen()s by bare soname. Only the dispatcher is ever a DT_NEEDED, so ldd finds it and never the seven. The allowlist force-copied three of them (libcudnn.so*, libcudnn_ops.so*, libcudnn_cnn.so*) into every CUDA backend, which is wrong in both directions at once: too few libraries for a backend that uses cuDNN, and too many for one that does not. On an L4T fleet, ten of the eleven backends carrying cuDNN were in a broken end state; the one that was correct was correct by accident, being BUILD_TYPE=cpu so package_cuda_libs never ran for it. longcat-video bundled 4 of 8 at 9.24.0 over a complete pip set at 9.20.0.48 in its venv. libbackend.sh puts lib/ on LD_LIBRARY_PATH, searched before DT_RUNPATH, so the bundle won and the rest still came from the venv: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH. Nine others bundled 3 of 8 and had no venv cuDNN. None bundled libcudnn_graph, which libcudnn_cnn has a hard DT_NEEDED on, so it resolved out of the runtime image and the process ran bundled 9.22.0 against system 9.23.2. Five of those nine - llama-cpp, whisper, rfdetr-cpp, sam3-cpp, stablediffusion-ggml - do not reference cuDNN at all. ggml goes through cuBLAS. They were carrying ~57 MB of cuDNN with no consumer, and completing the family for them would have taken that to ~576 MB for nothing. Sizes overall: backends with no cuDNN consumer shed ~57 MB each (seven instances on the fleet measured, plus longcat's ~60 MB), while the ones that genuinely use cuDNN grow from ~57 MB to ~576 MB, because the five missing sublibraries are ~517 MB, dominated by libcudnn_engines_precompiled. Net on that fleet is an increase of roughly 570 MB. That growth is the bug being paid off, not a regression: those backends only work today by silently borrowing the missing five from the runtime image. Whether the engines set can be trimmed is an open question, not addressed here. So bundle per backend, by what that backend actually needs: - venv has a complete pip cuDNN -> bundle nothing; $ORIGIN resolves the pip set, which is the one its torch was built against (longcat-video) - venv has no pip cuDNN -> bundle the complete family. Stays conservative rather than detecting consumers: for a Python backend they sit inside the venv (torch, ctranslate2, onnxruntime) where the sweep does not look (vllm) - no venv, nothing references cuDNN -> bundle nothing (llama-cpp, whisper, rfdetr-cpp, sam3-cpp, stablediffusion-ggml) - no venv, something references it -> bundle the complete family (face-detect, voice-detect) The no-venv case needs no new machinery. Go backends stage their own shared object into package/lib, which IS the target dir, so sweep_transitive_deps already pulls the dispatcher when it is a genuine dependency - that is exactly how libcudnn_graph reached longcat. cuDNN simply comes off the force-copy list, and complete_cudnn_family fills in the seven dlopen'd sublibraries around whatever the sweep found. Detection is a string scan rather than ldd, so a consumer that only dlopen()s cuDNN is seen too; over-matching costs an unused library, under-matching costs a backend that cannot load. Keeping bundled and pip versions in agreement instead is not viable: nothing here pins nvidia-cudnn (zero occurrences), torch is unpinned for l4t13 except longcat-video, and the fleet already runs five concurrent cuDNN versions - 9.19.0.56, 9.20.0.48, 9.22.0, 9.23.2, 9.24.0. verify_cudnn_bundle asserts the end state: exactly one complete cuDNN visible to whoever needs one - never both, never partial, and never zero for a backend that references it. Zero is correct and common otherwise. It deliberately does not accept the build image's system cuDNN as completing a partial bundle, which is the shape that had been shipping silently; the build image is not the runtime image. A version check alone would have missed longcat too, whose four bundled libs were all 9.24.0 and mutually consistent. Match per family for the other components for the same dlopen reason: TensorRT (libnvinfer_plugin, libnvinfer_builder_resource), cuBLAS, cuFFT, cuSPARSE, cuSOLVER, nvRTC. Exclusions bind inside copy_lib so they cover the sweep. The packaging scripts' shell tests ran nowhere in CI. Add make test-build-scripts and a lint workflow job so they gate every PR. Fixes #10905 Assisted-by: Claude:claude-opus-4-8 golangci-lint shellcheck Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-19 07:48:51 +00:00
Richard Palethorpe	9c43b2da8f	fix(model): make backend shutdown model-scoped (#10865) ... Avoid holding the global loader lock across backend lifecycle waits and propagate forced shutdown through distributed workers. Track parallel requests with in-flight counters and reserve worker ports until process termination. Add focused race tests and an authoritative FizzBee lifecycle model with a fail-closed conformance target. Assisted-by: Codex:GPT-5 [FizzBee] [Ginkgo] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-07-19 08:43:17 +02:00
LocalAI [bot]	3bb0d1cb49	feat(backend): add moss-tts-cpp text-to-speech backend (#10860) ... * feat(backend): add moss-tts-cpp text-to-speech backend Add a Go + purego backend wrapping the moss-tts.cpp ggml port of the OpenMOSS MOSS-TTS-Local v1.5 text-to-speech model (GPT-J local transformer decoded through MOSS-Audio-Tokenizer-v2), producing 48 kHz stereo audio with optional reference-audio voice cloning. Mirrors the qwen3-tts-cpp backend: dlopen the static-ggml shared library, bind the moss-tts.cpp C-API via purego, and serve the gRPC TTS method. A thin C shim holds the pipeline handle and copies engine PCM into a Go-freeable buffer. Wires the CI registration: backend-matrix.yml (CPU, CUDA 12/13, Intel SYCL f16/f32, Vulkan, ROCm, NVIDIA L4T, plus Darwin metal), backend/index.yaml metas and image entries pointing at mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF, the root Makefile build targets, and the changed-backends.js path mapping. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: list the moss-tts-cpp backend among the LocalAI-maintained engines Add moss-tts.cpp to the README "Backends built by us" table, the Text-to-Speech compatibility table, and the reference-audio voice-cloning backend list, so the new backend is documented alongside its peers. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(moss-tts-cpp): pin moss-tts.cpp to the squashed single-commit release moss-tts.cpp history was collapsed to a single commit; repoint MOSSTTS_CPP_VERSION to ee722b8e9205ee9b1b1c398a4e87e4e393e9be41. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(moss-tts-cpp): add the moss-tts-cpp-development gallery meta The gallery had the -development image entries but no matching -development meta anchor (as locate-anything-cpp and depth-anything-cpp have), so the master build was not installable as a gallery backend. Add moss-tts-cpp-development mirroring the production meta with the -development capability image names. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-17 09:26:12 +02:00
LocalAI [bot]	bbe018c1a0	feat(bonsai): PrismML llama.cpp fork backend + Bonsai/Ternary-Bonsai gallery models (#10834) ... feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp (github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0 (1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants. Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models are served with the same OpenAI-compatible API. No grpc-server allow-list patch is needed (bonsai adds weight quants, transparent to the server, not KV-cache types), and the reused server compiles cleanly against the fork with no skew patches (validated locally via a CPU docker build; patches/ is present but empty for any future re-pin skew). Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh, backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml meta-backend + per-platform images, and a nightly bump_deps entry tracking the `prism` branch. Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64, +pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-16 10:09:14 +02:00
Richard Palethorpe	b9d6d49e31	fix(cloud-proxy): publish backend gallery entries (#10858) ... Add stable and development gallery variants for Linux and Darwin, and wire the backend build matrix so the referenced images are published. Assisted-by: Codex:gpt-5 [yq] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-07-16 08:36:23 +02:00
LocalAI [bot]	b00422e45f	feat(backends): add LongCat video and avatar generation (#10792) ... * feat(backends): add LongCat video and avatar generation Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web] * refactor(config): declare model I/O modalities Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook. Assisted-by: Codex:GPT-5 [web] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-12 23:58:46 +02:00
LocalAI [bot]	7e8542ba32	fix(tests): make e2e backend model downloads resumable and stall-based (#10766) ... The vibevoice transcription e2e hangs until the go test timeout when the HF CDN is slow: the ASR Q4_K model is >10 GB, downloadFile capped every curl attempt at --max-time 600 (needs a sustained ~17 MB/s to fit), and curl's --retry restarts from byte zero, so no attempt ever makes forward progress. This killed the job twice on PR #10764 and previously forced skipping it on release tags (#10567). Replace the wall-clock cap with stall detection (--speed-limit 1 MiB/s over --speed-time 120s) and resume from the bytes already on disk with -C -, retrying from Go because curl does not re-evaluate the resume offset on its internal retries. Resume against the HF Xet CDN was verified by killing a transfer mid-flight and confirming the next invocation appended (114 MB -> 235 MB, GGUF magic intact). Also parameterize the suite timeout (BACKEND_TEST_TIMEOUT, default 30m) and raise it to 120m for the vibevoice transcription wrapper: a 10 GB download plus 25 specs does not fit in 30m even on a good day, and the job-level GHA timeout there is already 150m. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-10 09:19:17 +02:00
LocalAI [bot]	94bdc825dc	feat(backend): add moss-transcribe-cpp backend (MOSS-Transcribe-Diarize) (#10756) ... C++/ggml transcription + speaker diarization + timestamps backend. Purego dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output into segments with nanosecond timestamps. Adds the importer (surfaces in GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml, and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf). Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues) passed against the real libmoss-transcribe.so. The pre-commit coverage gate (full pkg/core + tests/e2e) could not run in the authoring sandbox (no live models, port 9090 held); CI must enforce it before merge. Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-09 23:27:11 +02:00
LocalAI [bot]	dd625921ff	fix(macos): staple the notarization ticket to the .app, not just the dmg (#10606) ... Stapling only the dmg leaves the LocalAI.app bundle with no embedded notarization ticket. Gatekeeper then falls back to an online notarization check on first launch, so the app fails to open on a Mac that is offline or behind a firewall, or once it has been copied out of the dmg — while it keeps working on the (online) build host, which masks the problem. Notarize and staple the .app before packaging it into the dmg so the bundle verifies offline. Adds a `notarize-app` subcommand to contrib/macos/sign-and-notarize.sh (zips the bundle for notarytool, then staples + validates) and invokes it from dmg-launcher-darwin. Stays a no-op when notary secrets are unset, so unsigned local/fork builds are unaffected. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: mudler <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-30 17:38:47 +02:00
Richard Palethorpe	5d0c43ec6e	feat(realtime): Semantic VAD EOU token (#10444) ... * feat(realtime): EOU-driven semantic_vad turn detection Add a `semantic_vad` turn-detection mode to the realtime API that feeds the transcription model live and decides "the user finished speaking" from the `<EOU>` end-of-utterance token rather than from silence alone. When EOU fires the turn commits immediately (~0.3s); otherwise it falls back to an eagerness-scaled silence threshold (low/med/high = 8/4/2s). Plumbing, bottom to top: - proto: `AudioTranscriptionLive` bidirectional RPC (config-first oneof, mono float PCM @16k, ready-ack / Unimplemented degrade signal) plus `TranscriptResult.eou` for the unary retranscribe gate. - pkg/grpc: client/server/base/embed scaffolding for the bidi stream, modeled on AudioTransformStream; release stream conns on terminal Recv. - parakeet-cpp: live transcription RPC with per-C-call engine locking (one live stream per turn, finalize+free at commit); bump parakeet.cpp to ABI v5 — incremental StreamingMel (no more quadratic per-feed mel recompute that delayed EOU on long turns) and the <EOU>/<EOB> split; strip the literal <EOU>/<EOB> from offline text and set Eou. - core/backend: LiveTranscriptionSession wrapper + pipeline `turn_detection:` config block (type/eagerness/retranscribe). - realtime: semantic_vad integration — live input captions streamed as transcription deltas while the user speaks, EOU-immediate commit with eagerness fallback, optional retranscribe gate (batch re-decode must also end in <EOU> to confirm), clause synthesis off the LLM token callback, and per-turn live-transcription / model_load telemetry. - UI: show the realtime pipeline components as a vertical list. Docs and tests included; opt-in via the pipeline YAML or per-session `session.update`. Non-streaming STT backends degrade to silence-only. Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Write] [Bash] Assisted-by: Claude Code:claude-fable-5 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): explicit formally-verified state machines + parakeet streaming driver The realtime API had several implicit state machines whose state was inferred from scattered booleans, channels, and five separate mutexes, leaving illegal/inconsistent states reachable. Make them explicit and keep the implementation in step with a formal design; rework the parakeet streaming backend along the same lines. Realtime state machines (M1-M5). Each is a sealed sum-type State/Event/Effect with a total, pure Next(state,event)->(state,[]effect) behind a single-writer Coordinator: M1 conncoord connection lifecycle: VAD toggle + once-only teardown (replaces vadServerStarted + a `done` channel closed from two sites). M2 turncoord turn detection: collapses speechStarted and the live-stream "turn open" flag into one state, so discardTurn can no longer desync them and suppress the next onset. M3 respcoord response coordination: serializes the dual-writer start/cancel so at most one response is live; one response.done per response.create. M4 compactcoord conversation compaction: single-flight (replaces the `compacting atomic.Bool` CAS). M5 ttscoord TTS pipeline: open->closing->closed, idempotent wait(), rejects enqueue-after-close (was a silent drop). The Coordinator/Sink/Next plumbing — only the sealed types and Next differed per machine — is extracted once into core/http/endpoints/openai/coordinator as a generic Coordinator[S,E,F]; each machine keeps its public API via type aliases, so no sink, call-site, or test moved. Hierarchy. session_lifecycle.fizz models M1 as the parent region with its children (M2/M3/M4) as one statechart and asserts ChildrenDieWithParent (conn torn => all children terminal, none start after teardown). respcoord and compactcoord gain an absorbing Terminated state + Shutdown event; conncoord's teardown drives the children terminal. This closes a compaction teardown gap: a fire-and-forget compaction could outlive a torn session — compactionSink now takes a session-scoped cancellable context + WaitGroup and joins the in-flight summarize+evict on shutdown. Formal verification. formal-verification/ holds one authoritative FizzBee spec per machine plus the composition spec, each with an always-assertion and a documented one-line edit that makes the checker fail (verified non-vacuous). scripts/realtime-conformance.sh is fail-closed: all Go conformance suites under -race AND a model-check of every .fizz spec; a missing FizzBee is a hard error (only the loud REALTIME_CONFORMANCE_SKIP_FIZZBEE=1 bypasses it, never in CI). FizzBee is pinned by sha256 and installed via scripts/install-fizzbee.sh into .tools/ (gitignored). Wired as make test-realtime-conformance, a CI workflow, and a pre-commit path filter. Go conformance tests are Ginkgo/Gomega (per the repo's forbidigo lint): transition tables + fixed-seed property walks + concurrent/-race specs, no rapid dependency. Design map: docs/design/realtime-state-machines.md. Parakeet streaming backend. The same treatment applied to the parakeet-cpp streaming paths: - AudioTranscriptionStream returns codes.Unimplemented for non-streaming models instead of decoding offline and emitting it as one delta + final. A client that asked for streaming learns the model cannot stream rather than receiving a batch result shaped like a stream. New grpcerrors.StreamTranscriptionUnsupported carries that signal; the HTTP /v1/audio/transcriptions stream path surfaces it as an SSE error event. Mirrors AudioTranscriptionLive, which already did this. - utteranceBoundary (boundary.go): a single definition of the end-of-utterance latch, replacing three open-coded finalEou toggles. Modelled as a two-valued type so illegal states are unrepresentable. - Shared decode driver (driver.go): streamFeedResult (one per-feed event) + feedChunk (hides the ABI v4 JSON vs text-only split) + feedSlices + flushTail. The feed loop is written once. - AudioTranscriptionLive becomes a bidi adapter: it streams the per-feed {delta,eou,eob,words} the realtime turn detector consumes and a terminal FinalResult carrying only Text. Segments/duration/eou are offline-only and no longer produced (nor read) on the live path; liveTraceState drops the terminal eou and keeps the per-feed eou_events count. - AudioTranscriptionStream + streamJSON merge into one driver-based function; streamSegmenter is generalized to the unified event with a text-only fallback that preserves the legacy (no-words) library's per-utterance segmentation. Verified: build/vet/gofumpt clean, golangci-lint 0 issues, all coordinator and parakeet packages under -race, the fail-closed conformance gate green, and make test-realtime (12 e2e WS+WebRTC). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-30 09:01:22 +02:00
LocalAI [bot]	f0d0bff232	fix(llama-cpp): stop reinterpreting plain-string message content as JSON (#10524) (#10538) ... The llama-cpp gRPC backend reconstructs OpenAI messages from proto for the tokenizer-template path and blindly json::parse'd each message's content string. LocalAI's Go layer always flattens content to a plain string, so a user prompt that merely looks like JSON (e.g. mealie's ingredient array ["1/4 cup brown sugar", ...]) was reinterpreted as structured content parts and rejected by oaicompat_chat_params_parse with "unsupported content[].type". Normalize content per role instead: user/system/developer content is opaque text and is never JSON-sniffed; assistant/tool content still collapses a literal JSON null/object (tool-call bookkeeping) to a string, but a plain string is never turned into an array/scalar. The array defense is role-independent, so the role gate only governs the benign null/object case. While here, extract the duplicated per-message reconstruction and the pre-template content sanitization into shared, unit-tested helpers (message_content.h) so the streaming (PredictStream) and non-streaming (Predict) paths cannot drift. This removes ~490 lines of copy-pasted defensive code, the dead tool-role parse branches, and the redundant Predict-only tool_calls branch, while preserving the prior #7324 (null content -> "") and #7528 (tool array content -> string) fixes. Tests: - backend/cpp/llama-cpp/message_content_test.cpp: standalone C++ unit tests for all three helpers (#10524, #7324, #7528, multimodal), discovered and run by `make test-backend-cpp` and a new generic tests-backend-cpp CI job. Also wired as an opt-in CMake/ctest target (-DLLAMA_GRPC_BUILD_TESTS=ON). - core/schema/message_test.go: Go regression pinning that ToProto flattens a JSON-array-looking text part to the verbatim string. - prepare.sh now copies message_content.h into the build tree. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-27 01:42:05 +02:00
LocalAI [bot]	5b3572f8b8	feat(macos): sign and notarize the DMG, app, and server binary (#10510) ... Produce a Gatekeeper-clean macOS distribution with no user workaround: - Launcher DMG + the LocalAI.app inside it are built via fyne, codesigned with the Developer ID under the hardened runtime, then the DMG is signed, notarized (notarytool) and stapled. Replaces macos-dmg-creator (which had no signing hook) with fyne package + hdiutil so we control the .app before packaging. - The bare local-ai darwin server binary is signed + notarized via GoReleaser's native notarize block (quill backend, runs on Linux). - All signing is gated on secrets being present, so forks/PRs/local builds stay unsigned and green (contrib/macos/sign-and-notarize.sh no-ops). - Add hardened-runtime entitlements and FyneApp.toml for deterministic packaging; update macOS install docs to drop the quarantine workaround. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-26 12:45:51 +02:00
LocalAI [bot]	d388f874de	feat(backends): darwin/Metal build for the privacy-filter backend (#10513) ... * feat(backends): darwin/Metal build for the privacy-filter backend (timeboxed try) The privacy-filter.cpp engine is already Metal-capable on Apple Silicon: it pulls ggml and never forces GGML_METAL=OFF, and ggml defaults Metal ON on Apple, so a plain Darwin build is Metal-enabled. grpc++/protobuf resolve from Homebrew via find_package(... CONFIG). It just had no darwin build path - the existing package.sh and run.sh are Linux-only and there was no make target / workflow step. Adds the bespoke darwin path, modeled on the ds4 one: - scripts/build/privacy-filter-darwin.sh: native make grpc-server, otool -L dylib bundling, create-oci-image (no Linux package.sh). - Makefile: backends/privacy-filter-darwin target (+ .NOTPARALLEL). - .github/workflows/backend_build_darwin.yml: gated build step for privacy-filter. - scripts/changed-backends.js: inferBackendPathDarwin special-case -> backend/cpp. - .github/backend-matrix.yml: includeDarwin entry (lang go, like ds4/llama-cpp). - backend/index.yaml: metal: capability + metal-privacy-filter(-development) entries. - backend/cpp/privacy-filter/run.sh: DYLD_LIBRARY_PATH branch on Darwin. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] * fix(privacy-filter): macOS proto include + bundle ggml dylibs Validated natively on an M4 (the build/package/load chain now works with Metal): - CMakeLists.txt: hw_grpc_proto compiles the generated proto/grpc sources but only linked the binary dir, so on macOS it could not find protobuf's headers (runtime_version.h) - Homebrew puts them under /opt/homebrew, not /usr/include. Link protobuf::libprotobuf + gRPC::grpc++ so their include dirs propagate. No-op on Linux (apt headers are already on the default search path). - privacy-filter-darwin.sh: bundle the ggml shared libs the binary @rpath-links (libggml{,-base,-cpu,-blas,-metal}); the otool -L walk only catches on-disk absolute deps and missed them. Resolved at runtime by run.sh's DYLD_LIBRARY_PATH. M4 check: arm64 grpc-server links @rpath/libggml-metal.0.dylib; with the 15 ggml dylibs + grpc/protobuf bundled, it loads clean (no dyld errors) and prints usage. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-26 01:18:41 +02:00
Richard Palethorpe	63bcbf6c12	fix(pii): post-merge review fixes + live NER e2e for the privacy-filter tier (#10401) ... * fix(pii): post-merge review fixes + live NER e2e for the privacy-filter tier Follow-up to the NER tier engine (#10360), already on master. This carries only the incremental review fixes and tests that postdate that merge — the feature itself is not re-introduced. Review fixes: - openai_completion.go: remove the dead `elem >= 0` conjunct in applyAnyText (the `elem < 0` guard above already returns). - application.go: collapse ResolvePIIPolicy's inline re-implementation of PIIIsEnabled to a single cfg.PIIIsEnabled() call (sole source of the "explicit pii.enabled wins, else cloud-proxy default" rule) and return true past the !enabled guard where it is provable. - pattern.go: hoist the triple `appConfig != nil && EnableTracing` check in patternDetector.Detect into one local. - grammar.go: MaxQuantifier was 4096, but Go's regexp/syntax rejects repeat bounds above 1000 at Parse time, so walk()'s {n,m} guard could never fire — dead code shadowed by the parser. Lower it to 512 so a bound in (512,1000] is rejected here with an actionable error; >1000 still fails closed via Parse. Specs pin the relationship so the guard can't silently revert. - PatternListEditor.jsx: clamp a directly-typed negative min_len to >=0 and force the DOM value back when clamping (min={0} only constrained the spinner, so a negative reached saved config and silently disabled the length filter). Tests: - piipattern_test.go: MaxQuantifier guard specs (must stay live, not dead). - model-config.spec.js: assert the min_len clamp, and that entity_actions collapses a duplicate group to a single row (map semantics; regression guard against emitting an array that drops a row on save). - tests/e2e-backends: token_classify capability driving the TokenClassify gRPC RPC against the backend image, asserting byte-correct, UTF-8 rune-aligned spans (entity.Text == text[start:end]) at threshold 0. Verified on CPU via `make test-extra-backend-privacy-filter` (3/3 specs). - Makefile: test-extra-backend-privacy-filter wrapper. - tests/e2e: e2e_pii_ner_test.go drives /api/pii/analyze + /api/pii/redact (mask + block) through the full HTTP -> detector -> redactor path; gated on PII_NER_MODEL_GGUF so the default suite is unaffected. - .github/workflows/tests-pii-ner-e2e.yml: path-filtered / nightly CI job running the container harness on CPU. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(gallery): add privacy-filter-nemotron (f16 + q8) GGUF conversions of OpenMed/privacy-filter-nemotron — a fine-grained English PII token-classifier (55 categories / 221 BIOES classes), fine-tuned from openai/privacy-filter on NVIDIA's Nemotron-PII dataset. Sibling to the existing privacy-filter-multilingual entry, trading language breadth for category depth. - privacy-filter-nemotron: F16 reference artifact (~2.8 GB). - privacy-filter-nemotron-q8: Q8_0 quant (~1.64 GB) for RAM-constrained / edge use; description notes the size/speed tradeoff and to validate on your own data (a single dropped span is a PII leak). Both run on the privacy-filter backend with known_usecases [token_classify] and a default mask policy (min_score 0.5); operators add per-category entity_actions as needed. sha256s taken from the HF repo's LFS object ids. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-22 18:26:19 +02:00
Richard Palethorpe	3fa7b2955c	feat(pii): NER tier engine — privacy-filter.cpp backend + NER-centric PII filter (#10360) ... Squashed feat/pii-ner-tier-engine rebased onto master (was 45 commits; see backup/pii-ner-tier-engine-prerebase). Net change: - privacy-filter.cpp: standalone GGML engine for the openai-privacy-filter PII/NER token classifier, wired as a LocalAI gRPC backend (CPU/CUDA/Vulkan). TokenClassify moves off the patched llama.cpp path onto this backend. - PII filter reworked to be NER-centric (encoder/NER detection tier scanning whole conversations as one document), with a recreated bounded restricted- regex secret-matching pattern detector tier alongside it (per-model pii_detection.builtins / .patterns + core/services/routing/piipattern). - Detection labelled by source (ner vs pattern); backend trace / confidence / debug observability; analyze/redact exposed as a synchronous API. - Instance-wide default detector policy + per-usecase default-on; request filtering extended to completions, embeddings, edits & Ollama. - React UI: NER-centric PII editor, detector-models table, pattern/builtins editor, middleware default-policy UI. - Gallery: privacy-filter-multilingual token-classify model + NER install filter; token_classify known_usecase; batch sized to context for NER models. privacy-filter backend registered in the backend gallery (cpu/vulkan/cuda-13 meta + image entries with a capabilities map) matching its CI matrix jobs, and an /import-model auto-detect importer (PrivacyFilterImporter, narrow privacy-filter GGUF detection) replacing the prior pref-only registration. Reconciled against master's independent evolution: - Dropped master's PIIPatternOverrides feature (global-pattern runtime overrides + /api/pii/patterns API + runtime_settings.json persistence). The per-model NER + pattern-detector design supersedes it; it was built on the global redactor pattern set this branch replaced. - Reverted the llama.cpp Score carry-patch (0006-server-task-type-score): removed the patch and restored master's grpc-server.cpp Score RPC (direct llama_decode, slot-loop bypass) and LLAMA_VERSION pin, plus master's model_config validation forbidding score + chat/completion/embeddings on llama-cpp. token_classify is unaffected (it runs on the privacy-filter backend, not llama-cpp). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-18 11:45:22 +01:00
LocalAI [bot]	294170d3ed	feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery (#10352) ... * feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery Mirrors the locate-anything-cpp backend to register a new depth-anything backend that wraps the Depth Anything 3 ggml port (depth-anything.cpp) via purego (cgo-less, no Python at inference). - backend/go/depth-anything-cpp/: gRPC backend (Load + Predict + GenerateImage), purego binding to the da_capi_* C ABI, CMake/Makefile/run/package/test scripts building depth-anything.cpp's DA_SHARED static .so per CPU variant. - backend/index.yaml: depth-anything backend meta + all hardware-variant capability entries (cpu/cuda12/cuda13/intel-sycl-f32+f16/vulkan/nvidia-l4t). - gallery/index.yaml: 8 Depth Anything 3 GGUF models (base q4_k/q8_0/f16/f32, small, large, giant, mono-large). - .github/backend-matrix.yml: one build entry per hardware variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): typed Depth RPC + REST endpoint exposing full DA3 data Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to e0b6814 (ABI 3 dense C-API) The Depth RPC handler calls da_capi_depth_dense / da_capi_points (C-API ABI 3); pin the native build to the commit that exports them. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to v0.1.0 release (b515c31) Repoint the native version from the now-orphaned e0b6814 to the b515c31 release commit, kept alive by the upstream v0.1.0 tag. C-API is unchanged (da_capi_abi_version == 3). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): wire depth-anything-cpp into build, CI bump, and importer The backend dir, gallery index, and CI build-matrix were present but the backend was never wired into the integration points that adding-backends.md requires: - root Makefile: add to .NOTPARALLEL, the test-extra chain, a BACKEND_* definition, the docker-build target eval, and docker-build-backends (mirrors parakeet-cpp; the backend's own Makefile already documented that its `test` target is driven by test-extra). - bump_deps.yaml: register the DEPTHANYTHING_VERSION pin so the daily auto-bump bot tracks mudler/depth-anything.cpp master (it cannot see an unregistered Makefile pin). - import form: add a preference-only KnownBackend entry so depth-anything is selectable at /import-model (mirrors sam3-cpp; no reliable GGUF auto-detect signal, so pref-only per the doc's default). changed-backends.js needs no entry: the generic golang suffix branch already resolves backend/go/depth-anything-cpp/. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): auto-detect importer for depth-anything GGUFs Replace the preference-only entry with a real auto-detect importer (mirrors parakeet-cpp / locate-anything): - DepthAnythingImporter matches a .gguf whose name carries a depth-anything token (depth-anything-<size>-<quant>.gguf), so /import-model recognises mudler/depth-anything.cpp-gguf repos and direct GGUF URLs without an explicit backend preference. preferences.backend= "depth-anything" still forces it. - Registered before LlamaCPPImporter so its GGUF bundles aren't claimed by the generic .gguf importer; the narrow name match means it cannot claim arbitrary llama GGUFs or the upstream safetensors PyTorch repos. - Multi-quant repos pick the smallest quant by default (q4_k -> ... -> f32, depth stays >0.998 corr even at q4_k); quantizations preference overrides. - Drops the now-redundant knownPrefOnlyBackends entry (importer-backed backends are not listed there, matching parakeet-cpp). - Table-driven Ginkgo test covers detection, negative cases (llama GGUF, upstream safetensors), default/override/fallback quant pick, and direct URL import. 10/10 specs pass. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): check conn.Close error in grpc Depth client (errcheck) The new Depth() client method used a bare `defer conn.Close()`. golangci-lint runs with new-from-merge-base, so although the 39 sibling methods use the same bare form (grandfathered), the newly added line trips errcheck. Drop the result explicitly to satisfy the linter. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.1 (embeddable CMake) v0.1.0 (b515c31) used ${CMAKE_SOURCE_DIR} for its include dirs, which points at the parent project when built via add_subdirectory() as this backend does, so the container build failed with missing stb_image.h / da_gguf_keys.h. v0.1.1 (2d42897) switches to project-relative paths. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): resolve gosec findings in the backend wrapper The code-scanning gate flagged three new failure-level alerts in godepthanythingcpp.go (gosec runs with -no-fail; GitHub gates on new alerts): - G301: export dirs were created with 0o755. Tighten to 0o750 (no world access needed for backend-written export output). - G304: writeDepthPNG creates req.GetDst(). That path is chosen by the LocalAI core as the intended output destination (same pattern every image backend uses), not attacker input, so annotate with #nosec G304 and document why. The remaining G103 "audit unsafe" notes on the unsafe.Slice C-buffer copies are warning-level (the same purego interop whisper/parakeet use) and do not gate the check, per the supertonic exclusion precedent in secscan.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.2 (CUDA cross-build arch) v0.1.1 forced CMAKE_CUDA_ARCHITECTURES=native, which breaks the GPU-less l4t/cublas CI builds (nvcc "Unsupported gpu architecture 'compute_'" on CMake 3.22). v0.1.2 (442eea4) drops the override and lets ggml pick its default cross-build arch list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-16 16:28:28 +02:00
LocalAI [bot]	2df2876db2	feat(supertonic): add Supertonic ONNX TTS backend (CPU) (#10342) ... * feat(supertonic): vendor upstream Go TTS pipeline (helper.go) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): add gRPC backend (Load/TTS/TTSStream, CPU) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(supertonic): satisfy unused linter (use onnxProvider; exclude vendored helper.go) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(supertonic): unit tests for resolvers + gated end-to-end synthesis Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(supertonic): gofmt backend.go comment block Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): add Makefile, run.sh, package.sh (CPU build) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(supertonic): wire backend into root Makefile Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(supertonic): check ort.DestroyEnvironment return (errcheck) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(supertonic): resolve voice_styles as sibling of onnx dir; guard trim; test voice Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): add CPU build matrix + gallery index entries Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): expose as pref-only importable backend Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(supertonic): add Supertonic/supertonic-3 TTS model to the gallery 16 files (4 onnx + tts.json + unicode_indexer.json + 10 voice styles) from HF Supertone/supertonic-3, served via the supertonic backend. Defaults to voice F1; onnx/ + sibling voice_styles/ layout matches the backend's resolveVoicesDir. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(meta): register pipeline.max_history_items config field Pre-existing on master: the field was added without a registry entry, failing TestAllFieldsHaveRegistryEntries (core/config/meta). Add the entry so it renders properly in the model-config UI. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(secscan): exclude vendored supertonic backend from gosec helper.go is vendored from supertone-inc/supertonic; its G304/G404/G104 findings are inherent to upstream and the math/rand use is correct for flow-matching noise (crypto/rand would be wrong). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-15 16:54:11 +02:00
LocalAI [bot]	0854932a25	feat(omnivoice-cpp): add OmniVoice TTS backend (file + streaming, voice cloning + voice design) (#10310) ... * feat(omnivoice-cpp): add C wrapper + CMake/Makefile build over OmniVoice ov_* ABI Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): add option/language parsing + WAV framing helpers with tests Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): wire purego binding with TTS + streaming TTSStream Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(omnivoice-cpp): wire backend into root Makefile Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(omnivoice-cpp): add build matrix entries + dep-bump registration Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): register backend meta + image entries Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): expose as preference-only importable backend Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add omnivoice-cpp TTS models (Q8_0 default + BF16 HQ) Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(omnivoice-cpp): document the OmniVoice TTS backend Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(omnivoice-cpp): add env-gated e2e for TTS + streaming Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(omnivoice-cpp): honor tts.audio_path/tts.voice config as default cloning reference The model config tts.audio_path (ModelOptions.AudioPath) and tts.voice now provide a default voice-cloning reference used when a request omits Voice, so a cloned voice can be pinned in the model YAML instead of passed per request. A per-request voice still overrides. Paths resolve relative to the model dir. Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(omnivoice-cpp): add missing omnivoice-cpp-development backend meta Mirrors the whisper/vibevoice convention: a -development meta aggregating the master-tagged image variants (the production meta and per-variant prod+dev image entries already existed; only the development meta aggregator was missing). Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-13 21:28:46 +02:00
LocalAI [bot]	56cc4f63fc	feat(backend): locate-anything-cpp (open-vocabulary object detection via ggml) (#10264) ... * feat(backend): add locate-anything-cpp backend (open-vocab detection via la_capi) A Go/purego backend wrapping locate-anything.cpp's la_capi C ABI, implementing the gRPC Detect RPC: image + open-vocabulary text prompt -> labeled boxes. Mirrors backend/go/rfdetr-cpp; static-links ggml into a per-CPU-variant .so. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(backend): register locate-anything-cpp in build matrix Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): locate-anything gallery entry + model importer Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(backend): locate-anything-cpp Load+Detect wire test Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add locate-anything-3b model to the gallery index Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(backend): register locate-anything.cpp in bump_deps auto-bump Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: mudler <mudler@localai.io> * ci(test): e2e smoke for locate-anything-cpp in test-extra (loads the 3B + image, runs Detect) Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: mudler <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: mudler <mudler@localai.io> Co-authored-by: mudler <mudler@localai.io>	2026-06-12 14:59:07 +02:00
Pete	d2e6b93369	feat(agents): surface KB source citations in RAG responses (#10228) ... * dev knowledge.go structure Signed-off-by: Pete Chen <petechentw@gmail.com> * feat(agents): append KB source citations to responses Render structured KB citations as a Sources block after agent responses, linking each source to the existing raw collection entry endpoint. Keep long-term memory writes on the original model response so citation blocks do not get stored back into the knowledge base. Tested with: go test ./core/services/agents Assisted-by: Codex:gpt-5 Signed-off-by: Pete Chen <petechentw@gmail.com> * Collect KB citations from tool searches Signed-off-by: Pete Chen <petechentw@gmail.com> * fix(agents): append KB sources in local chats Apply the shared KB citation post-processing to standalone LocalAGI chat responses so the React agent chat receives the same clickable Sources block as the native executor path. Also fix the run target to use the current cmd/local-ai entrypoint. Assisted-by: Codex:gpt-5 Signed-off-by: Pete Chen <petechentw@gmail.com> --------- Signed-off-by: Pete Chen <petechentw@gmail.com> Co-authored-by: shihyunhuang <shihyunhuang88@gmail.com> Co-authored-by: TLoE419 <tloemizuchizu@gmail.com> Co-authored-by: Ching Kao <0980124jim@gmail.com>	2026-06-09 16:32:56 +02:00
Richard Palethorpe	3a932a9803	feat(distributed): Add NATS JWT authentication and TLS/mTLS options (#10159) ... * feat(distributed): NATS JWT auth, TLS/mTLS options, and e2e coverage Mint per-node NATS user JWTs at registration when LOCALAI_NATS_ACCOUNT_SEED is set, and connect workers with scoped credentials from the register response. Add optional LOCALAI_NATS_TLS_CA/CERT/KEY for private CA and mTLS alongside tls:// URLs, plus test-e2e-distributed and NatsJWT container e2e specs. Document JWT setup (nats-auth-setup.sh) and TLS env vars in distributed-mode. Assisted-by: Grok:grok grok-build Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(distributed): correct NATS JWT scoping and harden client auth The JWT-auth path added in 46467cc7 had several gaps that fail silently under LOCALAI_NATS_REQUIRE_AUTH: - Agent-worker minted JWTs did not allow the subjects the agent worker actually subscribes to (jobs.mcp-ci.new and nodes.<id>.backend.stop), so MCP-CI jobs and backend-stop session cleanup were silently dropped. Scope the agent permission set to those subjects. - NATS subscription permission violations were swallowed (Subscribe returned a live-but-dead subscription). Confirm subscriptions with a server round-trip so a denial surfaces synchronously, and log async permission errors. - The backend worker connected anonymously when given a JWT without its paired seed; reject the unpaired credential instead. - The documented service-user permissions in nats-auth-setup.sh omitted prefixcache.>, which the frontend publishes and subscribes; add it. Also: add a credential-provider hook to the messaging client (consumed by the follow-up credential-lifecycle change), drop the always-nil error from NatsMessagingOptions, run go mod tidy (jwt/v2 and nkeys are now direct), and gofmt the feature's files. Tests: an agent-JWT e2e spec that connects to the enforcing NATS server and exercises every subscription the agent worker makes, plus permission allow-list coverage unit tests. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(distributed): acquire and auto-refresh worker NATS credentials Workers fetched NATS credentials once at startup, which broke two cases under JWT auth: a worker that registered while still pending admin approval never received a minted JWT (it connected unauthenticated and gave up), and a long-running worker's 24h JWT expired with no way to renew it. Introduce workerregistry.NATSCredentialManager, built on idempotent re-registration (the frontend preserves the node row and mints a fresh JWT each call): - Acquire re-registers through admin approval until the node is approved and credentials are minted (or returns the first success when auth is not required, preserving anonymous-NATS behavior). - RefreshLoop re-registers before the JWT expires (~75% of its lifetime), updating the credentials served to the connection. - Both are bounded (default 100 attempts / consecutive failures) and return an error on exhaustion, so an unapprovable or unrenewable worker exits non-zero and surfaces the problem instead of hanging or drifting toward an expired credential. The messaging client gains WithUserJWTProvider, fetching credentials on each (re)connect so the connection transparently adopts a refreshed JWT when the server expires the old one. RegisterFull exposes the approval status and full response; Register delegates to it. Both the backend worker and the agent worker are wired to this: explicit env credentials are used as-is, minted credentials are acquired-with-wait and refreshed, and a permanent refresh failure shuts the worker down so it restarts and re-acquires. Tests cover Acquire (wait-through-pending, bounded give-up, context cancel), RefreshLoop (refresh-before-expiry, bounded failure, no-expiry exit) and jwtExpiry decoding. Docs updated in distributed-mode.md. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-03 19:43:56 +02:00
LocalAI [bot]	76fe0bb929	feat(crispasr): add CrispASR backend — multi-architecture ASR + TTS (#10099) ... * feat(crispasr): backend source files (Go gRPC server, C-ABI shim, build files) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * polish(crispasr): brand error strings + fix stale shim comment Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(crispasr): register backend in root Makefile Mirror the whisper Go backend registration for the new crispasr backend: NOTPARALLEL entry, prepare-test-extra/test-extra hooks, BACKEND_CRISPASR definition, docker-build target generation, and the docker-build-backends aggregate target. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(crispasr): add backend build matrix entries Mirror the 11 whisper golang Dockerfile matrix entries (CPU amd64/arm64, CUDA 12/13, L4T CUDA 13, Intel SYCL f32/f16, Vulkan amd64/arm64, L4T arm64, ROCm hipblas) with backend and tag-suffix substituted to crispasr. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add crispasr backend gallery entries Add the crispasr meta anchor and its full set of image gallery entries (cpu, metal, cuda12/13, rocm, intel-sycl f32/f16, vulkan, L4T arm64, L4T cuda13 arm64, plus -development variants), mirroring the whisper backend gallery block. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(crispasr): bump CRISPASR_VERSION via bump_deps workflow Track CrispStrobe/CrispASR main branch and bump CRISPASR_VERSION in backend/go/crispasr/Makefile. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(crispasr): don't wire fixture-gated test into test-extra Mirror the whisper Go backend: its AudioTranscription test is gated on model/audio fixtures and skips in CI, so building crispasr (the heaviest ggml compile in the tree) inside the unit-test lane adds a long compile for zero coverage. The backend image build in backend-matrix.yml remains the authoritative compile check. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(crispasr): add darwin metal build entry (mirror whisper) The metal-crispasr gallery entries and capabilities.metal mapping reference -metal-darwin-arm64-crispasr, which is only produced by an includeDarwin entry. Mirror whisper's darwin metal entry so the tag actually gets built. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(crispasr): place hipblas matrix entry next to whisper twin Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(crispasr): register crispasr as pref-only ASR backend + test Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(crispasr): port whisper behavioral suite (cancellation + streaming) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(crispasr): fix skip message env var names to CRISPASR_* Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(crispasr): switch shim to crispasr_session_* multi-architecture API The shim used whisper_full(), which in CrispASR is the whisper-only path: libcrispasr only transcribes Whisper GGUFs through it. Multi-architecture transcription (Parakeet, Voxtral, Qwen3-ASR, Canary, Granite, FunASR, Paraformer, SenseVoice, ...) goes through the crispasr_session_* C-ABI, which auto-detects the architecture from the GGUF and dispatches to the matching backend. Rewrite the C shim around crispasr_session_open / _transcribe_lang / _result_* and add get_backend() so the selected backend is logged. load_model now takes a threads param (session_open binds n_threads at open). The session result is segment+word based with no token IDs and no per-decode callback, so drop n_tokens / get_token_id / get_segment_speaker_turn_next / set_new_segment_callback. set_abort is kept for API parity but is best-effort: the session transcribe is blocking with no abort hook. Update the purego bindings and gocrispasr.go to match: tokens are left empty, speaker-turn handling is removed, and AudioTranscriptionStream emits one delta per non-empty segment after the blocking decode returns (no progressive streaming via the session API), preserving the concat(deltas) == final.Text invariant. crispasr_session_set_translate is exported by libcrispasr but not declared in crispasr.h, so it is forward-declared in the shim alongside the open/transcribe/result functions. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(crispasr): link full CrispASR backend set for multi-arch support The shim's crispasr_session_* dispatch calls into the per-architecture backend libs (parakeet, voxtral, qwen3_asr, canary, funasr, paraformer, sensevoice, ...), which CrispASR builds as static archives. Linking only crispasr + ggml dead-stripped every backend object from the final module (nm backend-symbol count: 0), leaving a whisper-only .so. Link the same backend set as crispasr-cli so the static archives are pulled in. After this the module carries the backend symbols (nm count 407, .so grows from ~2.1MB to ~6.7MB) and the session API can dispatch to every compiled-in architecture. Also rewrite ${CMAKE_SOURCE_DIR}/examples/talk-llama to ${PROJECT_SOURCE_DIR}/... in the vendored src/CMakeLists.txt: CrispASR locates its vendored llama.cpp via ${CMAKE_SOURCE_DIR}, which is wrong when CrispASR is add_subdirectory'd (CMAKE_SOURCE_DIR points at this backend dir, not the CrispASR root). PROJECT_SOURCE_DIR is correct both standalone and as a subproject; the sed is idempotent. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(crispasr): adapt suite to session API (blocking, no decode callback) Register the new symbol set (drop the removed token/speaker/callback funcs, add get_backend; load_model now takes 2 args). The session transcribe is blocking with no abort hook, so a mid-decode cancel can't interrupt it: change the cancellation spec to cancel the context before the call and assert codes.Canceled from the pre-call ctx.Err() check, dropping the <5s mid-decode timing assertion. The streaming spec still holds with per-segment post-decode emission (>=2 deltas, concat(deltas) == final.Text). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add CrispASR ASR model entries (-crispasr) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): keep only session-auto-detectable CrispASR ASR models The crispasr backend loads models via crispasr_session_open, which auto-detects the backend from the GGUF general.architecture using crispasr_detect_backend_from_gguf. Architectures not in that detect map cannot be opened, so those gallery entries fail to load. Removed entries whose architecture is not wired into CrispASR v0.6.11's session auto-detect router (they can be re-added when upstream maps them): - Not in the detect map: data2vec, firered-asr, funasr, fun-asr-mlt-nano, glm-asr, hubert, kyutai-stt, mega-asr, mimo-asr, moonshine{,-de,-streaming,-tiny-de}, omniasr{,-llm,-llm-1b}, paraformer, sensevoice. - Pending verification (filename-heuristic routed, not arch-detected): parakeet-ctc-0.6b, parakeet-ctc-1.1b. Their GGUFs are routed to the fastconformer-ctc backend by a filename heuristic in the model registry, which implies general.architecture is not a mapped string. Kept the parakeet rnnt/tdt_ctc variants: convert-parakeet-to-gguf.py writes general.architecture="parakeet" unconditionally and encodes the rnnt/ctc distinction in metadata fields, so they session-auto-detect. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(crispasr): TTS synthesis via crispasr_session_synthesize (24kHz) Add tts_synthesize/tts_free/tts_set_voice to the C-ABI shim. They reuse the already-open g_session (crispasr_session_open auto-detects a TTS model) and dispatch to the upstream synthesis call, which returns malloc'd 24 kHz mono float PCM. Orpheus needs a SNAC codec path that we do not set, so it returns NULL here and surfaces as an error Go-side. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(crispasr): implement TTS/TTSStream gRPC methods Bind the new shim functions via purego and implement TTS, TTSStream and a writeWAV24k helper. synthesize copies the C-owned PCM out before freeing it; TTS writes a 24 kHz mono 16-bit WAV to req.Dst via go-audio/wav. CrispASR has no progressive synth, so TTSStream synthesizes fully, encodes to WAV, and emits the bytes as a single chunk; it owns the results-channel close (the gRPC server wrapper ranges until close), mirroring vibevoice-cpp's TTSStream. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(crispasr): log when a TTS voice override is not honored Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add CrispASR vibevoice-tts model entry Only vibevoice-tts works through the current shim: qwen3-tts, chatterbox, and orpheus require companion codec/s3gen/SNAC paths (set_codec_path / set_s3gen_path) that the shim doesn't wire yet, and kokoro/indextts/voxcpm2 aren't in the session auto-detect map. Those are follow-ups. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(crispasr): gated TTS synthesis spec Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(crispasr): satisfy golangci-lint (errcheck defers + unsafeptr nolint) The crispasr Go file is entirely new, so new-from-merge-base lints every line (unlike the grandfathered whisper backend it was forked from): - handle os.RemoveAll / fh.Close return values in AudioTranscription - annotate the two intentional C-pointer unsafe.Slice sites with //nolint:govet Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(crispasr): backend: and codec: model options (explicit arch + companion files) Add two model-config options to the CrispASR backend via opts.Options: - backend:<name> selects an explicit CrispASR backend (bypassing auto-detect) by routing load_model through crispasr_session_open_explicit, unlocking architectures the detector won't pick on its own (qwen3, cohere, granite, voxtral, moonshine, mimo-asr, orpheus, kokoro, chatterbox, etc.). - codec:<path> loads a companion file (qwen3-tts codec, orpheus SNAC, chatterbox s3gen, or mimo-asr tokenizer) via the universal crispasr_session_set_codec_path setter after the session opens. A relative path resolves against the model directory. rc==0 means success or not-applicable; only a negative rc is fatal. The C shim load_model gains a backend_name argument and a new set_codec_path entry point; the Go bridge parses the prefix:value options and registers the new symbol. The vad_only path is unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): expand CrispASR models via backend:/codec: options (explicit arch + companions) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(gallery): use virtual.yaml base for crispasr models The crispasr entries are just backend + model + a couple options, fully expressed inline via overrides:/files: in gallery/index.yaml. Point each url: at the shared gallery/virtual.yaml (the established 'virtual' model trick) and drop the 36 redundant per-model gallery/*-crispasr.yaml files. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): drop voice-requiring TTS entries (keep vibevoice-tts) Real e2e showed qwen3-tts/orpheus/chatterbox don't synthesize through the current shim: the codec: companion loads fine, but these engines additionally need a voice pack / voice prompt / reference clip (qwen3-tts base errors 'no voice'; chatterbox is zero-shot cloning; orpheus uses named voices) that the backend doesn't wire. (qwen3-tts also can't auto-detect: its GGUF arch is 'qwen3tts', unmapped by the detector — would need backend:qwen3-tts.) Removed to avoid shipping non-working gallery entries; vibevoice-tts (built-in voice, e2e-verified) remains the working TTS. Voice-pack wiring is a follow-up. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(crispasr): speaker: and voice: TTS options (baked speakers + voice packs/prompts) speaker:<name> -> crispasr_session_set_speaker_name (baked speakers: qwen3-tts CustomVoice, orpheus). voice:<path>(+voice_text:<ref>) -> crispasr_session_set_voice (voice-pack GGUF, or WAV zero-shot clone with ref text). Applied at Load as the default voice; req.Voice still overrides the speaker per request. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): re-add e2e-verified TTS engines (chatterbox, qwen3-tts-customvoice, orpheus) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-31 12:11:03 +02:00
LocalAI [bot]	4912c9b73a	feat(parakeet-cpp): add NVIDIA NeMo Parakeet ASR backend (parakeet.cpp) (#10084) ... * feat(parakeet-cpp): L0 backend scaffold, LoadModel + AudioTranscription (text) Add a Go gRPC backend that bridges LocalAI to parakeet.cpp via the flat C-API (parakeet_capi.h), loaded with purego (cgo-less, mirrors the whisper / vibevoice-cpp backends). L0 scope: - main.go: dlopen libparakeet.so (override via PARAKEET_LIBRARY), register the C-API entry points, start the gRPC server. - goparakeetcpp.go: Load (parakeet_capi_load), AudioTranscription (parakeet_capi_transcribe_path, decoder=0 = per-arch default head), Free, serialized through base.SingleThread since the C engine is a thread-unsafe singleton. char* returns are bound as uintptr so the malloc'd buffer is freed via parakeet_capi_free_string after copy. - AudioTranscriptionStream returns a clear "not implemented in L0" error (closes the channel so the server doesn't hang), wired in L2. - Makefile: clone-at-pin + cmake (PARAKEET_VERSION for bump_deps.sh), with a local-symlink dev shortcut; run.sh / package.sh mirror whisper. - Test auto-skips without PARAKEET_BACKEND_TEST_MODEL/_WAV fixtures. Builds clean (CGO_ENABLED=0), gofmt clean, test passes. The single unsafeptr vet note in goStringFromCPtr is documented and matches the whisper backend's tolerated pattern. Word/segment timestamps (L1) and cache-aware streaming (L2) follow. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(parakeet-cpp): L1 word/segment timestamps via transcribe_path_json AudioTranscription now calls parakeet_capi_transcribe_path_json and shapes the per-word / per-token timestamps into the TranscriptResult: - Bind parakeet_capi_transcribe_path_json (purego, char* as uintptr like the other returns) and register it in main.go + the test loader. - Parse the JSON document ({"text","words":[{w,start,end,conf}], "tokens":[{id,t,conf}]}) into typed structs. - Synthesise a single whole-clip segment (parakeet emits no native segment boundaries) spanning the first word start to the last word end; token ids populate Segment.Tokens. - Attach word-level timings only when timestamp_granularities=["word"], matching the OpenAI API (segment-level default). secondsToNanos mirrors the whisper backend's nanosecond convention. Verified end-to-end against tdt_ctc-110m (f16): both the default and word-granularity specs pass; builds clean, gofmt clean, vet shows only the one documented unsafeptr note shared with the whisper backend. Cache-aware streaming (L2) follows. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(parakeet-cpp): L2 cache-aware streaming with EOU segmentation Wire AudioTranscriptionStream to the streaming RNN-T C-API: - Bind parakeet_capi_stream_{begin,feed,finalize,free}; feed takes 16 kHz mono float PCM ([]float32 via purego) and writes *eou_out on <EOU>/<EOB>. - Decode opts.Dst to 16 kHz mono PCM (utils.AudioToWav + go-audio, same as the whisper backend), feed it in 1 s chunks, and emit each newly-finalized text run as a TranscriptStreamResponse delta. - <EOU>/<EOB> events close the current segment; a closing FinalResult carries the full transcript plus the per-utterance segments (with a whole-clip fallback segment when no EOU fired). - stream_begin returns 0 for non-streaming models, surfaced as a clear error instead of an empty stream. Honours context cancellation between chunks. Frees every malloc'd delta and the session. Verified end-to-end against realtime_eou_120m-v1 (f16): the streamed transcript matches the offline 110m reference word-for-word, deltas reconstruct the final text, and the spec passes alongside the offline specs. Builds clean, gofmt clean, vet shows only the shared documented unsafeptr note. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(parakeet-cpp): L3 register backend in build/CI/gallery (whisper parity) Wire the new Go gRPC parakeet-cpp backend (parakeet.cpp ggml port of NVIDIA NeMo Parakeet ASR) into LocalAI's build/CI/gallery surfaces, matching the existing ggml whisper Go backend 1:1. - .github/backend-matrix.yml: add 11 linux entries + 1 darwin entry mirroring every whisper build (cpu amd64/arm64, intel sycl f32/f16, vulkan amd64/arm64, nvidia cuda-12, nvidia cuda-13, nvidia-l4t-arm64, nvidia-l4t-cuda-13-arm64, rocm hipblas, metal-darwin-arm64), all on ./backend/Dockerfile.golang with backend: "parakeet-cpp" and -*-parakeet-cpp tag-suffixes. - scripts/changed-backends.js: explicit inferBackendPath branch resolving parakeet-cpp to backend/go/parakeet-cpp/ before the generic golang branch. - .github/workflows/bump_deps.yaml: track the PARAKEET_VERSION pin in backend/go/parakeet-cpp/Makefile (repo mudler/parakeet.cpp, branch master). - backend/index.yaml: add &parakeetcpp meta + latest/development image entries for every matrix tag-suffix. - Makefile: add backends/parakeet-cpp to .NOTPARALLEL, BACKEND_PARAKEET_CPP definition, docker-build target eval, and test-extra-backend-parakeet-cpp- transcription target (mirrors test-extra-backend-whisper-transcription). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(parakeet-cpp): L4 gallery importer for parakeet GGUFs Add ParakeetCppImporter so parakeet.cpp GGUFs auto-detect on /import-model and route to the parakeet-cpp backend (it also surfaces in /backends/known, which drives the import dropdown). - Match is narrow: a .gguf whose name carries a parakeet architecture token (<arch>-<size>-<quant>.gguf, e.g. tdt_ctc-110m-f16.gguf, rnnt-0.6b-q4_k.gguf, realtime_eou_120m-v1-q8_0.gguf), a direct URL to one, or preferences.backend="parakeet-cpp". It deliberately does NOT claim arbitrary llama-style GGUFs, nor the upstream nvidia/parakeet-* NeMo repos (.nemo, not runnable here). - Registered in the ASR batch BEFORE LlamaCPPImporter so its GGUFs aren't swallowed by the generic .gguf importer. - Import nests files under parakeet-cpp/models/<name>/, defaults to the smallest quant (q4_k, near-lossless on parakeet) with a size-ladder fallback, and honours preferences.quantizations / name / description. Tested with synthetic HF details (no network): metadata, positive matches (HF repo, direct URL, preference), narrowness negatives (llama GGUF, NeMo repo), and import (default quant, override, direct URL), 9 specs pass, build/vet/gofmt clean. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(parakeet-cpp): document the parakeet-cpp transcription backend Add parakeet-cpp to the audio-to-text backend list and a dedicated usage section: direct GGUF import (auto-detects to the backend), model YAML, word-level timestamps via timestamp_granularities[]=word, and cache-aware streaming with the realtime_eou model. Points at the mudler/parakeet-cpp-gguf collection repo. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(parakeet-cpp): wire transcription gRPC e2e test into test-extra The L3 commit added the test-extra-backend-parakeet-cpp-transcription Makefile target but never invoked it in CI. Mirror the whisper job: - Add a parakeet-cpp output to detect-changes (emitted by changed-backends.js from the matrix entry). - Add tests-parakeet-cpp-grpc-transcription, gated on the parakeet-cpp path filter / run-all, building the backend image and running the transcription e2e against tdt_ctc-110m + the JFK clip. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(parakeet-cpp): drop em dashes from comments and docs Replace em dashes with plain punctuation in the backend comments, the importer, package.sh, and the audio-to-text docs section (and use "and" instead of the multiplication sign). No behaviour change. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add parakeet-cpp f16 models to the model gallery Add the 10 NVIDIA Parakeet models (f16, the recommended quality/speed default) as gallery entries that install on the parakeet-cpp backend from mudler/parakeet-cpp-gguf: tdt_ctc-110m/1.1b, tdt-0.6b-v2/v3, tdt-1.1b, ctc-0.6b/1.1b, rnnt-0.6b/1.1b, and the cache-aware streaming realtime_eou_120m-v1. Each pins the file sha256 and routes transcript usecases to the backend. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): satisfy govet lint + bump PARAKEET_VERSION - goparakeetcpp.go: //nolint:govet on the C-owned-pointer unsafe.Pointer conversion (golangci-lint reports new-only issues, so unlike the whisper backend's identical line this one is flagged). - Makefile: bump PARAKEET_VERSION to the current parakeet.cpp master commit (the previous pin's commit no longer exists after upstream history was squashed), so the backend image clone/build resolves again. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): pin PARAKEET_VERSION to a tag-stable commit The previous SHA pin was orphaned when parakeet.cpp's single-commit master was amended/force-pushed, so the backend image clone (git fetch <sha>) failed across every build variant. Repoint to 845c29e, which upstream now keeps permanently fetchable via the `localai-backend-pin` tag, so future upstream amends no longer break the backend build. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): init the ggml submodule in the backend image clone The backend Dockerfile clones parakeet.cpp at PARAKEET_VERSION with a shallow fetch + checkout but never initialised submodules, so third_party/ggml was empty and the parakeet.cpp cmake build failed at `add_subdirectory(third_party/ggml)` (CMakeLists.txt:53) on every build variant. Add `git submodule update --init --recursive --depth 1 --single-branch` after checkout, mirroring the whisper backend. Verified locally: clone + submodule + cmake configure now succeeds. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): statically link ggml into libparakeet.so The shared libparakeet.so linked ggml's shared libs (libggml*.so), but the package only ships libparakeet.so, so at runtime dlopen failed with "libggml.so.0: cannot open shared object file" (the e2e transcription test panicked on load). Build ggml static + PIC (BUILD_SHARED_LIBS=OFF, CMAKE_POSITION_INDEPENDENT_CODE=ON) so libparakeet.so embeds ggml and depends only on system libs already present in the runtime image. Verified locally: ldd shows no libggml dependency. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(parakeet-cpp): non-streaming fallback in AudioTranscriptionStream The e2e streaming test ran AudioTranscriptionStream against tdt_ctc-110m (not a cache-aware streaming model), so stream_begin returned 0 and the call errored. Per LocalAI's streaming contract (and the whisper backend), a non-streaming model should fall back to a single offline transcription emitted as one delta plus a closing FinalResult. Do that instead of erroring, so the streaming endpoint works for every parakeet model. Verified locally: the streaming spec passes against the non-streaming 110m model via fallback. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-30 14:46:10 +02:00
Richard Palethorpe	b81a6d01b3	perf(react-ui): code-split bundle, speed up coverage suite (#10042) ... * Curate the highlight.js build to ~29 languages (lib/core + the common set) instead of the full ~190-grammar default: -787 KB raw / -230 KB gz on the base bundle. * Code-split every route via React.lazy with a per-layout <Suspense> in App.jsx so the sidebar stays mounted on navigation. Initial entry chunk drops from 3194 KB raw / 887 KB gz to 397 KB / 122 KB (-87%). Warm chunks on sidebar hover/focus/touch via a preload registry so the click finds the chunk already in flight or cached. * Migrate Playwright coverage from istanbul (build-time counters) to native Chromium V8 coverage, with per-worker accumulation + conversion. Suite drops from 71s to 30s at 20 workers (~58%) at the non-instrumented floor. * Keep the coverage gate bundling-invariant: the coverage build inlines dynamic imports so every shipped source file lands in the denominator (otherwise untested page chunks silently drop out and inflate the percentage). Production builds stay code-split. * Add UI_TEST_WORKERS=N Makefile knob; tighten coverage tolerance to 0.8pp now that jitter sits near istanbul's ~0.5pp again. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-28 13:43:15 +02:00
LocalAI [bot]	7a4ca8f60d	feat(backend): rfdetr-cpp native object detection + segmentation backend (#10028) ... Adds a Go native gRPC backend that dlopens librfdetrcpp.so (built from mudler/rf-detr.cpp at the pinned RFDETR_VERSION) via purego and exposes the rfdetr.cpp inference pipeline through LocalAI's existing Detect RPC. Supports all 5 RF-DETR detection variants (Nano/Small/Base/Medium/Large) and 6 segmentation variants (SegNano/SegSmall/SegMedium/SegLarge/ SegXLarge/Seg2XLarge) with F32/F16/Q8_0/Q4_K quantizations. Pre-built GGUFs ship at mudler/rfdetr-cpp-* on HuggingFace. Detection returns Bbox + class_name + confidence; segmentation also returns PNG-encoded per-detection masks via the rfdetr_capi accessor functions (rfdetr_capi_get_detection_{class_id,box,score,class_name, mask_png}). End-to-end verified through POST /v1/detection: HTTP -> gRPC -> purego dlopen -> rfdetr.cpp -> ggml -> response (9 detections on the detection model, 21 detections + valid PNG masks on the seg-nano model against the kitchen fixture). Wiring: - backend/go/rfdetr-cpp/{main.go,gorfdetrcpp.go,CMakeLists.txt, Makefile,run.sh,package.sh,test.sh,.gitignore} - Top-level Makefile: BACKEND_RFDETR_CPP, docker-build target, .NOTPARALLEL, prepare-test-extra, test-extra - backend/go/rfdetr-cpp/Makefile: `test` target invoked by test-extra - .github/backend-matrix.yml: CPU + CUDA-12/13 + L4T CUDA-12/13 (arm64) + HIP + Vulkan (amd64 + arm64) + SYCL f32/f16 - backend/index.yaml: rfdetr-cpp meta anchor + latest/development image entries for every matrix tag-suffix - .github/workflows/bump_deps.yaml: RFDETR_VERSION pin tracking (mudler/rf-detr.cpp branch main) - gallery/index.yaml: 11 rfdetr-cpp-* entries (nano + 4 detection variants + 6 seg variants), all backed by mudler/rfdetr-cpp-* on HuggingFace with sha256 pinning on the F16 default - core/gallery/importers/rfdetr.go: GGUF auto-routing for HF imports (mudler/rfdetr-cpp-* repos route to rfdetr-cpp, Transformer-format repos stay on the Python rfdetr backend; explicit preferences.backend overrides both heuristics) - core/gallery/importers/rfdetr_test.go: table-driven coverage of the auto-routing + a live mudler/rfdetr-cpp-nano cross-check scripts/changed-backends.js needs no change: the existing Dockerfile.golang -> backend/go/${item.backend}/ branch already routes the 9 rfdetr-cpp matrix entries to the correct backend path. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-27 18:43:57 +02:00
Richard Palethorpe	8d70855ea6	test: add Go + React UI coverage gates and fill test gaps (#9989) ... - Strict monotonic Go coverage gate (make test-coverage-check, 45% baseline) run in CI; fixes ginkgo dropping all-but-one coverprofile across multiple recursive roots, builds with -tags auth, and folds in the in-process tests/e2e suite via --coverpkg. - React UI e2e coverage (make test-ui-coverage: vite-plugin-istanbul + nyc, nix-provided Chromium) plus e2e specs for 6 previously-untested pages, and a UI coverage gate (make test-ui-coverage-check) with a small tolerance since e2e line coverage jitters ~0.5pp run-to-run. - pre-commit hook: lint + coverage on Go changes, Playwright e2e + UI coverage gate on react-ui changes; install with make install-hooks. - New Go handler tests (settings, branding), hermetic base64 download test. - fix(ui): model editor reads vram_display (snake_case), so the VRAM estimate renders again; covered by a regression test. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-26 22:06:10 +02:00
Richard Palethorpe	6a80e23733	feat(middleware): Model routing, PII filtering, Cloud model proxies (#9802) ... Add a routing middleware stack and a cloud-proxy backend. * cloud-proxy: a Go gRPC backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers, with an optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response) and full tool-calling support. * routing: admission control, content-aware model routing (embedding cache + classifier + rerank + Arch-Router score), PII detection/redaction (regex + NER) with streaming filter and OpenAI/Anthropic adapters, and a per-user/per-key billing recorder backed by GORM or in-memory storage. * middleware: UsageMiddleware records usage via the billing recorder, plus admission, route-model, usage-stamp and trace middlewares. * observability: BackendTrace ring buffer stores full request bodies (capped), MITM proxy emits structured trace events, and router classifier decisions surface at /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0). * UI: cloud-proxy model-editor fields, classifier system-prompt and score-normalization config, and a Traces page rendering request bodies. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-25 09:28:27 +02:00
Richard Palethorpe	0245b33eab	feat(realtime): Add Liquid Audio s2s model and assistant mode on talk page (#9801) ... * feat(liquid-audio): add LFM2.5-Audio any-to-any backend + realtime_audio usecase Wires LiquidAI's LFM2.5-Audio-1.5B as a self-contained Realtime API model: single engine handles VAD, transcription, LLM, and TTS in one bidirectional stream — drop-in alternative to a VAD+STT+LLM+TTS pipeline. Backend - backend/python/liquid-audio/ — new Python gRPC backend wrapping the `liquid-audio` package. Modes: chat / asr / tts / s2s, voice presets, Load/Predict/PredictStream/AudioTranscription/TTS/VAD/AudioToAudioStream/ Free and StartFineTune/FineTuneProgress/StopFineTune. Runtime monkey-patch on `liquid_audio.utils.snapshot_download` so absolute local paths from LocalAI's gallery resolve without a HF round-trip. soundfile in place of torchaudio.load/save (torchcodec drags NVIDIA NPP we don't bundle). - backend/backend.proto + pkg/grpc/{backend,client,server,base,embed, interface}.go — new AudioToAudioStream RPC mirroring AudioTransformStream (config/frame/control oneof in; typed event+pcm+meta out). - core/services/nodes/{health_mock,inflight}_test.go — add stubs for the new RPC to the test fakes. Config + capabilities - core/config/backend_capabilities.go — UsecaseRealtimeAudio, MethodAudio ToAudioStream, UsecaseInfoMap entry, liquid-audio BackendCapability row. - core/config/model_config.go — FLAG_REALTIME_AUDIO bitmask, ModalityGroups membership in both speech-input and audio-output groups so a lone flag still reads as multimodal, GetAllModelConfigUsecases entry, GuessUsecases branch. Realtime endpoint - core/http/endpoints/openai/realtime.go — extract prepareRealtimeConfig() so the gate is unit-testable; accept realtime_audio models and self-fill empty pipeline slots with the model's own name (user-pinned slots win). - core/http/endpoints/openai/realtime_gate_test.go — six specs covering nil cfg, empty pipeline, legacy pipeline, self-contained realtime_audio, user-pinned VAD slot, and partial legacy pipeline. UI + endpoints - core/http/routes/ui.go — /api/pipeline-models accepts either a legacy VAD+STT+LLM+TTS pipeline or a realtime_audio model; surfaces a self_contained flag so the Talk page can collapse the four cards. - core/http/routes/ui_api.go — realtime_audio in usecaseFilters. - core/http/routes/ui_pipeline_models_test.go — covers both code paths. - core/http/react-ui/src/pages/Talk.jsx — self-contained badge instead of the four-slot grid; rename Edit Pipeline → Edit Model Config; less pipeline-specific wording. - core/http/react-ui/src/pages/Models.jsx + locales/en/models.json — new realtime_audio filter button + i18n. - core/http/react-ui/src/utils/capabilities.js — CAP_REALTIME_AUDIO. - core/http/react-ui/src/pages/FineTune.jsx — voice + validation-dataset fields, surfaced when backend === liquid-audio, plumbed via extra_options on submit/export/import. Gallery + importer - gallery/liquid-audio.yaml — config template with known_usecases: [realtime_audio, chat, tts, transcript, vad]. - gallery/index.yaml — four model entries (realtime/chat/asr/tts) keyed by mode option. Fixed pre-existing `transcribe` typo on the asr entry (loader silently dropped the unknown string → entry never surfaced as a transcript model). - gallery/lfm.yaml — function block for the LFM2 Pythonic tool-call format `<|tool_call_start|>[name(k="v")]<|tool_call_end|>` matching common_chat_params_init_lfm2 in vendored llama.cpp. - core/gallery/importers/{liquid-audio,liquid-audio_test}.go — detector matches LFM2-Audio HF repos (excludes -gguf mirrors); mode/voice preferences plumbed through to options. - core/gallery/importers/importers.go — register LiquidAudioImporter before LlamaCPPImporter. - pkg/functions/parse_lfm2_test.go — seven specs for the response/argument regex pair on the LFM2 pythonic format. Build matrix - .github/backend-matrix.yml — seven liquid-audio targets (cuda12, cuda13, l4t-cuda-13, hipblas, intel, cpu amd64, cpu arm64). Jetpack r36 cuda-12 is skipped (Ubuntu 22.04 / Python 3.10 incompatible with liquid-audio's 3.12 floor). - backend/index.yaml — anchor + 13 image entries. - Makefile — .NOTPARALLEL, prepare-test-extra, test-extra, docker-build-liquid-audio. Docs - .agents/plans/liquid-audio-integration.md — phased plan; PR-D (real any-to-any wiring via AudioToAudioStream), PR-E (mid-audio tool-call detector), PR-G (GGUF entries once upstream llama.cpp PR #18641 lands) remain. - .agents/api-endpoints-and-auth.md — expand the capability-surface checklist with every place a new FLAG_* needs to be registered. Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): function calling + history cap for any-to-any models Three pieces, all on the realtime_audio path that just landed: 1. liquid-audio backend (backend/python/liquid-audio/backend.py): - _build_chat_state grows a `tools_prelude` arg. - new _render_tools_prelude parses request.Tools (the OpenAI Chat Completions function array realtime.go already serialises) and emits an LFM2 `<|tool_list_start|>…<|tool_list_end|>` system turn ahead of the user history. Mirrors gallery/lfm.yaml's `function:` template so the model sees the same prompt shape whether served via llama-cpp or here. Without this the backend silently dropped tools — function calling was wired end-to-end on the Go side but the model never saw a tool list. 2. Realtime history cap (core/http/endpoints/openai/realtime.go): - Session grows MaxHistoryItems int; default picked by new defaultMaxHistoryItems(cfg) — 6 for realtime_audio models (LFM2.5 1.5B degrades quickly past a handful of turns), 0/unlimited for legacy pipelines composing larger LLMs. - triggerResponse runs conv.Items through trimRealtimeItems before building conversationHistory. Helper walks the cut left if it would orphan a function_call_output, so tool result + call pairs stay intact. - realtime_gate_test.go: specs for defaultMaxHistoryItems and trimRealtimeItems (zero cap, under cap, over cap, tool-call pair preservation). 3. Talk page (core/http/react-ui/src/pages/Talk.jsx): - Reuses the chat page's MCP plumbing — useMCPClient hook, ClientMCPDropdown component, same auto-connect/disconnect effect pattern. No bespoke tool registry, no new REST endpoints; tools come from whichever MCP servers the user toggles on, exactly as on the chat page. - sendSessionUpdate now passes session.tools=getToolsForLLM(); the update re-fires when the active server set changes mid-session. - New response.function_call_arguments.done handler executes via the hook's executeTool (which round-trips through the MCP client SDK), then replies with conversation.item.create {type:function_call_output} + response.create so the model completes its turn with the tool output. Mirrors chat's client-side agentic loop, translated to the realtime wire shape. UI changes require a LocalAI image rebuild (Dockerfile:308-313 bakes react-ui/dist into the runtime image). Backend.py changes can be swapped live in /backends/<id>/backend.py + /backend/shutdown. Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): LocalAI Assistant ("Manage Mode") for the Talk page Mirrors the chat-page metadata.localai_assistant flow so users can ask the realtime model what's loaded / installed / configured. Tools are run server-side via the same in-process MCP holder that powers the chat modality — no transport switch, no proxy, no new wire protocol. Wire: - core/http/endpoints/openai/realtime.go: - RealtimeSessionOptions{LocalAIAssistant,IsAdmin}; isCurrentUserAdmin helper mirrors chat.go's requireAssistantAccess (no-op when auth disabled, else requires auth.RoleAdmin). - Session grows AssistantExecutor mcpTools.ToolExecutor. - runRealtimeSession, when opts.LocalAIAssistant is set: gate on admin, fail closed if DisableLocalAIAssistant or the holder has no tools, DiscoverTools and inject into session.Tools, prepend holder.SystemPrompt() to instructions. - Tool-call dispatch loop: when AssistantExecutor.IsTool(name), run ExecuteTool inproc, append a FunctionCallOutput to conv.Items, skip the function_call_arguments client emit (the client can't execute these — it doesn't know about them). After the loop, if any assistant tool ran, trigger another response so the model speaks the result. Mirrors chat's agentic loop, driven server-side rather than via client round-trip. - core/http/endpoints/openai/realtime_webrtc.go: RealtimeCallRequest gains `localai_assistant` (JSON omitempty). Handshake calls isCurrentUserAdmin and builds RealtimeSessionOptions. - core/http/react-ui/src/pages/Talk.jsx: admin-only "Manage Mode" checkbox under the Tools dropdown; passes localai_assistant: true to realtimeApi.call's body, captured in the connect callback's deps. Mirroring chat's pattern means the in-process MCP tools surface "just works" for the Talk page without exposing a Streamable-HTTP MCP endpoint (which was the alternative). Clients with their own MCP servers can still use the existing ClientMCPDropdown path in parallel; the realtime handler distinguishes them by AssistantExecutor.IsTool() at dispatch time. Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): render Manage Mode tool calls in the Talk transcript Previously the realtime endpoint only emitted response.output_item.added for the FunctionCall item, and Talk.jsx's switch ignored the event — so server-side tool runs were invisible in the UI. The model would speak the result but the user had no way to see what tool was actually called. realtime.go: after executing an assistant tool inproc, emit a second output_item.added/.done pair for the FunctionCallOutput item. Mirrors the way the chat page displays tool_call + tool_result blocks. Talk.jsx: handle both response.output_item.added and .done. Render FunctionCall (with arguments) and FunctionCallOutput (pretty-printed JSON when possible) as two transcript entries — `tool_call` with the wrench icon, `tool_result` with the clipboard icon, both in mono-space secondary-colour. Resets streamingRef after the result so the next assistant text delta starts a fresh transcript entry instead of appending to the previous turn. Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * refactor(realtime): bound the Manage Mode tool-loop + preserve assistant tools Fallout from a review pass on the Manage Mode patches: - Bound the server-side agentic loop. triggerResponse used to recurse on executedAssistantTool with no cap — a model that kept calling tools would blow the goroutine stack. New maxAssistantToolTurns = 10 (mirrors useChat.js's maxToolTurns). Public triggerResponse is now a thin shim over triggerResponseAtTurn(toolTurn int); recursion increments the counter and stops at the cap with an xlog.Warn. - Preserve Manage Mode tools across client session.update. The handler used to blindly overwrite session.Tools, so toggling a client MCP server mid-session silently wiped the in-process admin tools. Session now caches the original AssistantTools slice at session creation and the session.update handler merges them back in (client names win on collision — the client is explicit). - strconv.ParseBool for the localai_assistant query param instead of hand-rolled "1" || "true". Mirrors LocalAIAssistantFromMetadata. - Talk.jsx: render both tool_call and tool_result on response.output_item.done instead of splitting them across .added and .done. The server's event pairing (added → done) stays correct; the UI just doesn't need to inspect both phases of the same item. One switch case instead of two, no behavioural change. Out of scope (noted for follow-ups): extract a shared assistant-tools helper between chat.go and realtime.go (duplication is small enough that two parallel implementations stay readable for now), and an i18n key for the Manage Mode helper text (Talk.jsx doesn't use i18n anywhere else yet). Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci(test-extra): wire liquid-audio backend smoke test The backend ships test.py + a `make test` target and is listed in backend-matrix.yml, so scripts/changed-backends.js already writes a `liquid-audio=true|false` output when files under backend/python/liquid-audio/ change. The workflow just wasn't reading it. - Expose the `liquid-audio` output on the detect-changes job - Add a tests-liquid-audio job that runs `make` + `make test` in backend/python/liquid-audio, gated on the per-backend detect flag The smoke covers Health() and LoadModel(mode:finetune); fine-tune mode short-circuits before any HuggingFace download (backend.py:192), so the job needs neither weights nor a GPU. The full-inference path remains gated on LIQUID_AUDIO_MODEL_ID, which CI doesn't set. The four new Go test files (core/gallery/importers/liquid-audio_test.go, core/http/endpoints/openai/realtime_gate_test.go, core/http/routes/ui_pipeline_models_test.go, pkg/functions/parse_lfm2_test.go) are already picked up by the existing test.yml workflow via `make test` → `ginkgo -r ./pkg/... ./core/...`; their packages all carry RunSpecs entries. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-13 21:57:27 +02:00
LocalAI [bot]	d892e4af80	feat: add ds4 backend (DeepSeek V4 Flash) with tool calls, thinking, KV cache (#9758) ... * test(e2e-backends): allow BACKEND_BINARY for native-built backends Adds an escape hatch for hardware-gated backends (e.g. ds4) where the model is too large for Docker build context. When BACKEND_BINARY points at a run.sh produced by 'make -C backend/cpp/<name> package', the suite skips docker image extraction and drives the binary directly. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e-backends): validate BACKEND_BINARY basename + log actual source Two follow-ups from the cbcf5148 code review: - BACKEND_BINARY now requires a path whose basename is `run.sh`. Without this check, `filepath.Dir(binary)` silently discarded the filename, so pointing the env var at an arbitrary binary failed later with a confusing assertion that named a path the user never typed. - The "Testing image=..." debug line printed an empty string when the binary path was used, hiding the actual source in CI logs. The line now reports whichever of BACKEND_IMAGE / BACKEND_BINARY is in effect as `src=...`. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): scaffold ds4 backend dir Adds prepare.sh, run.sh, and a .gitignore. CMakeLists, Makefile, and the implementation arrive in follow-up commits. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): add backend Makefile Drives ds4's upstream Makefile to produce engine .o files (CUDA on Linux when BUILD_TYPE=cublas, Metal on Darwin, otherwise CPU debug path), then invokes CMake on our wrapper. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): add CMakeLists for grpc-server Generates protoc stubs from backend.proto, links grpc-server.cpp + dsml_parser.cpp + dsml_renderer.cpp + kv_cache.cpp against pre-built ds4 engine .o files. DS4_GPU=cuda|metal|cpu selects the backend. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): grpc-server skeleton + module stubs The minimum that links: Backend service with Health + Free; other RPCs default to UNIMPLEMENTED. Stub headers/sources for dsml_parser, dsml_renderer, and kv_cache are in place so CMake links cleanly even before those modules ship. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement LoadModel Opens engine + creates session sized to ContextSize (default 32768). Backend is compile-time: CPU when DS4_NO_GPU, Metal on __APPLE__, else CUDA. MTP/speculative options are accepted via ModelOptions.Options[] (mtp_path, mtp_draft, mtp_margin). kv_cache_dir option is captured into g_kv_cache_dir for the cache module (Task 19 wires it in). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement TokenizeString Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement Predict (plain text) Tool calls + thinking-mode split arrive in Task 13 once dsml_parser is in. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement PredictStream (plain text) ChatDelta + reasoning/tool_calls split arrives in Task 14. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement Status RPC Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): add DSML streaming parser Classifies raw model-emitted token text into CONTENT / REASONING / TOOL_START / TOOL_ARGS / TOOL_END events. Markers it watches for are the literal DSML strings rendered by ds4_server.c's prompt template (<｜DSML｜tool_calls>, <｜DSML｜invoke name=...>, <think>, etc.) - these are plain text the model emits, not special tokens. Partial markers split across token chunks are buffered until a full marker or a definitively-not-a-marker '<' is observed. RandomToolId() generates the API-side tool call id (call_xxx) that exact-replay would key on. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): split hex escapes in DSML markers + add cstring/cstdio includes C++ \x hex escapes have no length cap. '\x9cD' was read as a single escape producing byte 0xCD, eating the 'D'. The markers were never actually matching the DSML text the model emits. Split each escape with adjacent string literal concatenation so the byte sequence is exactly EF BD 9C 44 (｜D) at runtime. Also adds <cstring> and <cstdio> includes (libstdc++ 13 does not transitively expose std::strlen / std::snprintf via <string>). The local plan file (uncommitted) was also updated with the same fixes so Task 16's dsml_renderer.cpp does not re-introduce the bug. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): wire DsmlParser into Predict (ChatDelta) Non-streaming Predict now emits one ChatDelta carrying content, reasoning_content, and tool_calls[] parsed from the model's DSML output. Reply.message still carries the raw model bytes for backends that prefer the regex fallback path. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): wire DsmlParser into PredictStream Per-token ChatDelta writes: content/reasoning_content go incrementally, tool_calls emit TOOL_START as one delta (id + name) followed by TOOL_ARGS deltas with incremental JSON. The Go-side aggregator (pkg/functions/chat_deltas.go) reassembles them. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): chat template + reasoning_effort mapping UseTokenizerTemplate=true + Messages -> ds4_chat_begin / append / assistant_prefix. PredictOptions.Metadata['enable_thinking'] and ['reasoning_effort'] map to ds4_think_mode (DS4_THINK_HIGH default; 'max'/'xhigh' -> DS4_THINK_MAX; disabled -> DS4_THINK_NONE). Tool-call rendering for assistant turns with tool_calls JSON arrives in the next commit (dsml_renderer). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): render assistant tool_calls + tool results to DSML Closes the round-trip: when an OpenAI client sends a multi-turn chat where prior turns contain tool_calls or role=tool messages, build_prompt serializes them back to the DSML shape the model was trained on. Mirrors ds4_server.c's prompt renderer; uses nlohmann::json for parsing the OpenAI tool_calls payload. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): disk KV cache module Dir-based cache keyed by SHA1(rendered prompt prefix). File format: 'DS4G' magic + version + ctx_size + prefix_len + prefix + payload_bytes + ds4_session_save_payload output. NOT bit-compatible with ds4-server's KVC files - that interop is a follow-up plan. LoadLongestPrefix walks the dir picking the longest stored prefix that prefixes the incoming prompt. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): wire KvCache into Predict/PredictStream LoadModel reads 'kv_cache_dir' from ModelOptions.Options[], passes it to g_kv_cache.SetDir. Each Predict/PredictStream computes a render text for the request, tries LoadLongestPrefix to recover state, then Saves the new state after generation. ds4_session_sync handles the live-cache fast path internally, so the disk cache only matters for cold-starts and cross-session reuse. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): add package.sh Linux: bundles libc + ld + libstdc++ + libgomp + GPU runtime libs into package/lib so the FROM scratch image boots without a host libc. Darwin is handled by scripts/build/ds4-darwin.sh which uses otool -L. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): rename namespace ds4_backend -> ds4cpp ds4.h defines 'typedef enum {...} ds4_backend' which collides with our C++ 'namespace ds4_backend' anywhere a TU includes both. kv_cache.h includes ds4.h directly and surfaces the conflict immediately; other TUs would hit it once gRPC dev headers are available. Renames the C++ namespace to ds4cpp across all wrapper files and the plan, leaving the upstream ds4 typedef untouched. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend): add Dockerfile.ds4 Single-stage builder (CUDA devel image for cublas, ubuntu:24.04 for cpu) -> FROM scratch with packaged grpc-server + bundled runtime libs. nlohmann-json3-dev is required for dsml_renderer's JSON handling. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(make): wire backend/cpp/ds4 + ds4-darwin into root Makefile BACKEND_DS4 entry + generate-docker-build-target eval + docker-build-ds4 in docker-build-backends + .NOTPARALLEL guards. Also adds the backends/ds4-darwin target which delegates to scripts/build/ds4-darwin.sh (landed in Task 24). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci: add backend-matrix entries for ds4 (cpu + cuda13, per-arch) Two entries per build (amd64 + arm64) so backend-merge-jobs assembles a multi-arch manifest. Skipping cuda12 - ds4 was validated against CUDA 13. Darwin Metal is handled outside this matrix by backend_build_darwin.yml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/index): add ds4 meta + image entries cpu + cuda13 x latest + master. Darwin Metal builds publish under ds4-darwin via the existing llama-cpp-darwin OCI pipeline. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(scripts/build): add ds4-darwin.sh Native macOS/Metal build for the ds4 backend. Mirrors llama-cpp-darwin.sh: make grpc-server -> otool -L for dylib bundling -> OCI tar that 'local-ai backends install' consumes via the backends/ds4-darwin Makefile target. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(darwin): build ds4-darwin in backend_build_darwin Adds a 'Build ds4 backend (Darwin Metal)' step that runs the backends/ds4-darwin Makefile target on the macOS runner. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(import): auto-detect ds4 weights via DS4Importer Adds core/gallery/importers/ds4.go which matches on the antirez/deepseek-v4-gguf repo URI and the DeepSeek-V4-Flash-*.gguf filename pattern. Registered before LlamaCPPImporter so ds4 weights route to backend: ds4 instead of falling through to llama-cpp. Also lists ds4 in /backends/known so the /import-model UI surfaces it as a manual choice for users who want to force the backend on a non-canonical URI. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add deepseek-v4-flash-q2 (ds4 backend) One-click install of the q2 weights with backend: ds4. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(.agents): add ds4-backend.md Documents the backend shape, DSML state machine, thinking-mode mapping, disk KV cache, build matrix (cpu/cuda13/Darwin), and the BACKEND_BINARY hardware-validation path. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): pass UBUNTU_VERSION + arch env vars to install-base-deps The .docker/install-base-deps.sh script needs UBUNTU_VERSION (defaults to 2404), TARGETARCH, SKIP_DRIVERS, and APT_MIRROR/APT_PORTS_MIRROR exported into the environment so it can pick the right cuda-keyring / cudss / nvpl debs and apt mirrors. Dockerfile.ds4 was declaring some of the ARGs but not re-exporting them via ENV. Mirrors Dockerfile.llama-cpp's pattern. Without this fix 'make docker-build-ds4 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13' failed at: /usr/local/sbin/install-base-deps: line 120: UBUNTU_VERSION: unbound variable Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/index): add Metal image entries for ds4 Adds metal-ds4 + metal-ds4-development image entries pointing at quay.io/go-skynet/local-ai-backends:{latest,master}-metal-darwin-arm64-ds4 (built by scripts/build/ds4-darwin.sh on macOS arm64 runners), plus the 'metal' and 'metal-darwin-arm64' capability mappings on the ds4 meta and ds4-development variant. Closes a gap from the initial Task 23 landing - the Darwin Metal build script and CI workflow step were already wired (Tasks 24-25), but the gallery had no image entry for users to install the Metal variant. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): use ubuntu:24.04 base for ds4 cuda13 matrix entries The initial Task 22 matrix landing used base-image: 'nvidia/cuda:13.0.0-devel-ubuntu24.04' which clashes with install-base-deps.sh's cuda-keyring step: E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/ The canonical pattern (llama-cpp, ik-llama-cpp, turboquant) uses plain 'ubuntu:24.04' + 'skip-drivers: false' so install-base-deps installs CUDA from scratch via its own keyring setup. Adopting that here. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): drop install-base-deps.sh dependency The .docker/install-base-deps.sh pipeline is built around the llama-cpp needs: NVIDIA keyring + cuda-toolkit apt + gRPC-from-source build at /opt/grpc. For ds4 we don't need any of that: - CUDA: nvidia/cuda:13.0.0-devel-ubuntu24.04 ships /usr/local/cuda ready to go; install-base-deps's keyring step then conflicts with the pre-installed Signed-By. - gRPC: ds4's grpc-server.cpp only links against grpc++; system libgrpc++-dev (apt) is sufficient, no source build needed. Replaced the install-base-deps invocation in Dockerfile.ds4 with a direct 'apt-get install libgrpc++-dev libprotobuf-dev protobuf-compiler-grpc nlohmann-json3-dev cmake build-essential pkg-config git'. Matrix entries back to nvidia/cuda base + skip-drivers=true so install-base-deps would no-op even if some downstream tooling calls it. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): correct proto accessors + alias grpc::Status as GStatus Two compile bugs caught by the docker build: 1. proto::Message uses snake_case accessors. The build_prompt loop called m.toolcalls() / m.toolcallid() - the protoc-generated names are m.tool_calls() / m.tool_call_id(). Plan-text bug propagated to the wrapper. 2. The Status RPC method shadowed the 'using grpc::Status' alias, so any later method declaration using Status as a return type failed to parse ('Status does not name a type' starting at LoadModel). Solution: alias grpc::Status as GStatus instead, with no 'using' clause that would conflict. All RPC method declarations and return-statement constructions now use GStatus. Pre-existing code reviewer flagged the Status-shadow concern as 'minor' in the original Task 10 commit; it turned out to be a real compile blocker under libstdc++ 13 once the surrounding methods were filled in. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): preserve TOOL_ARGS content in dsml_parser Flush When the model emitted a parameter value that arrived in the same buffer as the surrounding tool_call markers (e.g. the buffered tail after a literal '</think>' opened the model output), the parser deferred all buffered bytes to Flush() because looks_like_prefix() always returns true while buf starts with '<'. Flush() then drained the buffer as plain CONTENT/REASONING regardless of parser state, so the bytes between the parameter open and close markers were classified as CONTENT instead of TOOL_ARGS. Symptom: the model emitted <|DSML|parameter name="location" string="true">Paris, France</|DSML|parameter> and the assembled tool_call arguments came out as {"location":""} - the opener and closer were emitted into the args stream but the "Paris, France" content went to the assistant message instead. Fix: 1. Flush() now uses the same state-aware emit logic as DrainPlain: PARAM_VALUE bytes become TOOL_ARGS (json-escaped when string), THINK bytes become REASONING, TEXT bytes become CONTENT, and INVOKE / TOOL_CALLS structural whitespace is discarded. 2. looks_like_prefix() restricts its leading-'<' fallback to buffers that have not yet seen a '>'. Without that change, char-by-char feeds would discard the '<' of '<|DSML|invoke name="..."' once the marker prefix length was reached but the closing quote/'>' were still in flight. Verified with a standalone harness that runs the failing input three ways (single Feed, split-after-'>', and char-by-char) and aggregates TOOL_ARGS for tool index 0: all three now produce {"location":"Paris, France"}. Assisted-by: Claude:opus-4.7 [Read,Edit,Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): use ds4_session_sync + manual generation loop for KV persistence ds4_engine_generate_argmax() is a self-contained helper that doesn't take or update a ds4_session - it manages its own internal state. Our Predict and PredictStream methods created g_session via ds4_session_create() but then called ds4_engine_generate_argmax(), so g_session's KV state never advanced. ds4_session_payload_bytes(g_session) returned 0 and the disk KV cache save correctly rejected with 'session has no valid checkpoint to save'. Switch both RPCs to the proper session API: ds4_session_sync(g_session, &prompt, ...) loop: int token = ds4_session_argmax(g_session) if token == eos: break emit(token) ds4_session_eval(g_session, token, ...) After the loop the session has a real checkpoint and ds4_session_save_payload writes the KV state to disk. Verified end-to-end on a DGX Spark GB10: three .kv files (15-30 MB each) are written when BACKEND_TEST_OPTIONS sets kv_cache_dir, and the e2e tool-call assertion still passes. Also added stderr diagnostics to KvCache (enabled/disabled at SetDir; per-save path + payload_bytes + result) so future failures are visible instead of silent. The 'wrote ok' lines are low-volume - one per Predict/PredictStream when the cache is enabled - and skipped entirely when the option is unset. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): use ds4_session_eval_speculative_argmax when MTP loaded Wires MTP (Multi-Token Prediction) speculative decoding into the manual generation loop in both Predict and PredictStream. When the upstream MTP weights are loaded via 'mtp_path:' option AND we're on CUDA / Metal, ds4_engine_mtp_draft_tokens() returns >0 and we switch the inner loop to ds4_session_eval_speculative_argmax(), which can accept N>1 tokens per verifier step. When MTP is not loaded (no option, CPU backend, or weights absent), we fall through to the simple ds4_session_argmax + ds4_session_eval path with no behavior change. Validated on a DGX Spark GB10 with the optional MTP GGUF (DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf, ~3.6 GB). LoadModel logs 'ds4: MTP support model loaded ... (draft=2)' on stderr. Caveat per upstream README: 'currently provides at most a slight speedup, not a meaningful generation-speed win'. Wired now mainly to track the upstream API; bigger speedups arrive when ds4 improves the speculative path. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): honor PredictOptions sampling with DSML-aware override Mirrors ds4_server.c:7102-7115 sampling-policy semantics on the LocalAI gRPC side. The generation loop now consults compute_sample_params() per token to pick the effective (temperature, top_k, top_p, min_p), based on: 1. Request defaults: PredictOptions.temperature / .topk / .topp / .minp 2. Thinking-mode override: when enable_thinking != false, force T=1.0, top_k=0, top_p=1.0, min_p=0.0 (creativity for the reasoning pass and the trailing content) 3. DSML structural override: when DsmlParser::IsInDsmlStructural() returns true (we are between tool-call markers but NOT in a param value payload), force T=0.0 so protocol bytes parse cleanly When the effective temperature is 0, we keep using ds4_session_argmax + MTP speculative path (matches ds4-server's gate that only enables MTP for greedy positions). When > 0, we call ds4_session_sample(s, T, ...) with a per-thread RNG seeded from system_clock and fall back to single-token ds4_session_eval. New public method on DsmlParser: IsInDsmlStructural() encodes which states need protocol-byte determinism. PARAM_VALUE is excluded (payload uses user sampling); TEXT and THINK are excluded (no tool-call context to protect). Verified on the DGX Spark GB10: the e2e suite still passes with all 5 specs including tools, and the Predict output now varies between runs (creative sampling active) while the tool-call args remain a clean '{"location":"Paris, France"}' because the parser-state check forces greedy on the structural bytes. UX note: thinking mode is ON by default (matching ds4-server). Users who want deterministic output should set Metadata.enable_thinking = false. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add sha256 to deepseek-v4-flash-q2 entry Per HF LFS metadata for antirez/deepseek-v4-gguf: size: 86720111200 bytes (~80.76 GiB) sha256: 31598c67c8b8744d3bcebcd19aa62253c6dc43cef3b8adf9f593656c9e86fd8c LocalAI's downloader verifies sha256 when present, so users who install deepseek-v4-flash-q2 from the gallery get integrity-checked weights and the partial-download issue (an 81 GB file is easy to truncate) becomes recoverable instead of silently producing a broken backend. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-11 22:15:47 +02:00
LocalAI [bot]	19d59102d5	feat(whisper-cpp): implement streaming transcription (#9751) ... * test(whisper): wire e2e streaming transcription target Adds test-extra-backend-whisper-transcription, mirroring the existing llama-cpp / sherpa-onnx / vibevoice-cpp targets. The generic AudioTranscriptionStream spec at tests/e2e-backends/backend_test.go:644 fails today because backend/go/whisper has no streaming impl - this target is the failing TDD gate that the next phase makes pass. Confirmed RED locally: 3 Passed (health, load, offline transcription), 1 Failed (streaming spec hits its 300s context deadline because the base implementation returns 'unimplemented' but doesn't close the result channel, leaving the gRPC stream open until the client times out). Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(whisper-cpp): expose new_segment_callback to the Go side Adds set_new_segment_callback() and a C-side trampoline that whisper.cpp invokes once per new text segment during whisper_full(). The trampoline dispatches (idx_first, n_new, user_data) to a Go function pointer registered via purego.NewCallback - text and timings are pulled by Go through the existing get_segment_text/get_segment_t0/get_segment_t1 getters. Wires the hook only when streaming is actually requested, to avoid a per-segment function-pointer dispatch on the offline path. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(whisper-cpp): implement AudioTranscriptionStream Wires whisper.cpp's new_segment_callback through purego back to Go so the streaming transcription RPC produces real, time-correlated deltas while whisper_full() is still decoding. Each segment becomes one TranscriptStreamResponse{Delta}; whisper_full's return is the TranscriptStreamResponse{FinalResult} carrying the full segment list, language, and duration. Per-call state is tracked in a sync.Map keyed by an atomic counter; the Go callback registered via purego.NewCallback is a singleton, dispatched through user_data. SingleThread today means only one entry is ever live, but the map shape matches the sherpa-onnx TTS callback pattern. The streaming path's final.Text is the literal concat of every emitted delta (a strings.Builder accumulated by onNewSegment) so the e2e invariant `final.Text == concat(deltas)` holds exactly. The first delta has no leading space; subsequent deltas are space-prefixed. The offline AudioTranscription path is unchanged. Closes the gap with sherpa-onnx, vibevoice-cpp, llama-cpp, and tinygrad, which already implement AudioTranscriptionStream. Verified GREEN locally: make test-extra-backend-whisper-transcription passes 4/4 specs (3 Passed initially under RED, +1 streaming spec now). Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(whisper-cpp): assert progressive multi-segment streaming Drives AudioTranscriptionStream against a real long-audio fixture and asserts len(deltas) >= 2. The generic e2e spec at tests/e2e-backends/backend_test.go:644 only checks len(deltas) >= 1 which is satisfied by both real and faked streaming - this spec is the guardrail that a future "fake" impl can't sneak past. Skipped by default (env-gated, like the cancellation spec); set WHISPER_LIBRARY, WHISPER_MODEL_PATH, and WHISPER_AUDIO_PATH to a 30+ second clip to run. Verified locally with a 55s 5x-JFK concat against ggml-base.en.bin: 1 Passed in 7.3s, deltas >= 2, finalSegmentCount >= 2, concat(deltas) == final.Text. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(whisper-cpp): add transcription gRPC e2e job Mirrors tests-sherpa-onnx-grpc-transcription / tests-llama-cpp-grpc-transcription. Runs make test-extra-backend-whisper-transcription whenever the whisper backend or the run-all switch fires, so a pin-bump or refactor that breaks streaming transcription gets caught before merge. The whisper output on detect-changes is already emitted by scripts/changed-backends.js (it iterates allBackendPaths); this PR just exposes it as a workflow output and consumes it. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(whisper-cpp): silence errcheck on AudioTranscriptionStream defers golangci-lint runs with new-from-merge-base=origin/master, so the identical defer patterns in the existing offline AudioTranscription path are grandfathered while the new ones in AudioTranscriptionStream trip errcheck. Wrap both defers in `func() { _ = ... }()` to match what errcheck wants without altering behavior. The errors from os.RemoveAll and *os.File.Close are not actionable inside a defer here (we're already returning), matching the offline path's contract. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-10 23:11:46 +02:00
Ettore Di Giacinto	3bc5ae8da6	fix(tests/e2e-backends): bump ctx_size for llama-cpp transcription ... Qwen3-ASR-0.6B encodes the jfk.wav fixture into 777 audio tokens via its mmproj, but the test harness defaulted BACKEND_TEST_CTX_SIZE to 512, so llama.cpp server rejected every transcription request with "request (777 tokens) exceeds the available context size (512 tokens)". Set BACKEND_TEST_CTX_SIZE=2048 on the llama-cpp transcription target only — sherpa-onnx and vibevoice transcription targets don't go through llama.cpp's slot/n_ctx and weren't failing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code]	2026-05-07 22:31:08 +00:00
Richard Palethorpe	8e43842175	feat(vllm, distributed): tensor parallel distributed workers (#9612) ... * feat(vllm): build vllm from source for Intel XPU Upstream publishes no XPU wheels for vllm. The Intel profile was silently picking up a non-XPU wheel that imported but errored at engine init, and several runtime deps (pillow, charset-normalizer, chardet) were missing on Intel -- backend.py crashed at import time before the gRPC server came up. Switch the Intel profile to upstream's documented from-source procedure (docs/getting_started/installation/gpu.xpu.inc.md in vllm-project/vllm): - Bump portable Python to 3.12 -- vllm-xpu-kernels ships only a cp312 wheel. - Source /opt/intel/oneapi/setvars.sh so vllm's CMake build sees the dpcpp/sycl compiler from the oneapi-basekit base image. - Hide requirements-intel-after.txt during installRequirements (it used to 'pip install vllm'); install vllm's deps from a fresh git clone of vllm via 'uv pip install -r requirements/xpu.txt', swap stock triton for triton-xpu==3.7.0, then 'VLLM_TARGET_DEVICE=xpu uv pip install --no-deps .'. - requirements-intel.txt trimmed to LocalAI's direct deps (accelerate / transformers / bitsandbytes); torch-xpu, vllm, vllm_xpu_kernels and the rest come from upstream's xpu.txt during the source build. - requirements.txt: add pillow + charset-normalizer + chardet -- used by backend.py and missing on the Intel install profile. - run.sh: 'set -x' so backend startup is visible in container logs (the gRPC startup error path was previously opaque). Also adds a one-line docs example for engine_args.attention_backend under the vLLM section, since older XE-HPG GPUs (e.g. Arc A770) need TRITON_ATTN to bypass the cutlass path in vllm_xpu_kernels. Tested end-to-end on an Intel Arc A770 with Qwen2.5-0.5B-Instruct via LocalAI's /v1/chat/completions. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(vllm): add multi-node data-parallel follower worker vLLM v1's multi-node story is one process per node sharing a DP coordinator over ZMQ -- the head runs the API server with data_parallel_size > 1 and followers run `vllm serve --headless ...` with matching topology. Today LocalAI can already configure DP on the head via the engine_args YAML map, but there's no way to bring up the follower nodes -- so the head sits waiting for ranks that never handshake. Add `local-ai p2p-worker vllm`, mirroring MLXDistributed's structural precedent (operator-launched, static config, no NATS placement). The worker: - Optionally self-registers with the frontend as an agent-type node tagged `node.role=vllm-follower` so it's visible in the admin UI and operators can scope ordinary models away via inverse selectors. - Resolves the platform-specific vllm backend via the gallery's "vllm" meta-entry (cuda*, intel-vllm, rocm-vllm, ...). - Runs vLLM as a child process so the heartbeat goroutine survives until vLLM exits; forwards SIGINT/SIGTERM so vLLM can clean up its ZMQ sockets before we tear down. - Validates --headless + --start-rank 0 is rejected (rank 0 is the head and must serve the API). Backend run.sh dispatches `serve` as the first arg to vllm's own CLI instead of LocalAI's backend.py gRPC server -- the follower speaks ZMQ directly to the head, there is no LocalAI gRPC on the follower side. Single-node usage is unchanged. Generalises the gallery resolution helper into findBackendPath() shared by MLX and vLLM workers; extracts ParseNodeLabels for the comma-separated label parsing both use. Ships with two compose recipes (`docker-compose.vllm-multinode.yaml` for NVIDIA, `docker-compose.vllm-multinode.intel.yaml` for Intel XPU/xccl) plus `tests/e2e/vllm-multinode/smoke.sh`. Both vendors are supported (NCCL for CUDA/ROCm, xccl for XPU) but mixed-vendor DP is not -- PyTorch's process group requires every rank to use the same collective backend, and NCCL/xccl/gloo don't interoperate. Out of scope (deferred): SmartRouter-driven placement of follower ranks via NATS backend.install events, follower log streaming through /api/backend-logs, tensor-parallel across nodes, disaggregated prefill via KVTransferConfig. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * test(vllm): CPU-only end-to-end test for multi-node DP Adds tests/e2e/vllm-multinode/, a Ginkgo + testcontainers-go suite that brings up a head + headless follower from the locally-built local-ai:tests image, bind-mounts the cpu-vllm backend extracted by make extract-backend-vllm so it's seen as a system backend (no gallery fetch, no registry server), and asserts a chat completion across both DP ranks. New `make test-e2e-vllm-multinode` target wires the docker build, backend extract, and ginkgo run together; BuildKit caches both images so re-runs only rebuild what changed. Tagged Label("VLLMMultinode") so the existing distributed suite isn't pulled along. Two pre-existing bugs surfaced by the test: 1. extract-backend-% (Makefile) failed for every backend, because all backend images end with `FROM scratch` and `docker create` rejects an image with no CMD/ENTRYPOINT. Fixed by passing --entrypoint=/run.sh -- the container is never started, only docker-cp'd, so the path doesn't have to exist; we just need anything that satisfies the daemon's create-time validation. 2. backend/python/vllm/run.sh's `serve` shortcut for the multi-node DP follower exec'd ${EDIR}/venv/bin/vllm directly, but uv bakes an absolute build-time shebang (`#!/vllm/venv/bin/python3`) that no longer resolves once the backend is relocated to BackendsPath. _makeVenvPortable's shebang rewriter only matches paths that already point at ${EDIR}, so the original shebang slips through unchanged. Fixed by exec-ing ${EDIR}/venv/bin/python with the script as an argument -- Python ignores the script's shebang in that case. The test fixture caps memory aggressively (max_model_len=512, VLLM_CPU_KVCACHE_SPACE=1, TORCH_COMPILE_DISABLE=1) so two CPU engines fit on a 32 GB box. TORCH_COMPILE_DISABLE is currently mandatory for cpu-vllm: torch._inductor's CPU-ISA probe runs even with enforce_eager=True and needs g++ on PATH, which the LocalAI runtime image doesn't ship -- to be addressed in a follow-up that bundles a toolchain in the cpu-vllm backend. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(vllm): bundle a g++ toolchain in the cpu-vllm backend image torch._inductor's CPU-ISA probe (`cpu_model_runner.py:65 "Warming up model for the compilation"`) shells out to `g++` at vllm engine startup, regardless of `enforce_eager=True` -- the eager flag only disables CUDA graphs, not inductor's first-batch warmup. The LocalAI CPU runtime image (Dockerfile, unconditional apt list) does not ship build-essential, and the cpu-vllm backend image is `FROM scratch`, so any non-trivial inference on cpu-vllm crashes with: torch._inductor.exc.InductorError: InvalidCxxCompiler: No working C++ compiler found in torch._inductor.config.cpp.cxx: (None, 'g++') Bundling the toolchain in the CPU runtime image would bloat every non-vllm-CPU deployment and force a single GCC version on backends that may want clang or a different version. So this lives in the backend, gated to BUILD_TYPE=='' (the CPU profile). `package.sh` snapshots g++ + binutils + cc1plus + libstdc++ + libc6 (runtime + dev) + the math libs cc1plus links (libisl/libmpc/libmpfr/ libjansson) into ${BACKEND}/toolchain/, mirroring /usr/... layout. The unversioned binaries on Debian/Ubuntu are symlink chains pointing into multiarch packages (`g++` -> `g++-13` -> `x86_64-linux-gnu-g++-13`, the latter in `g++-13-x86-64-linux-gnu`), so the package list resolves both the version and the arch-triplet variant. Symlinks /lib -> usr/lib and /lib64 -> usr/lib64 are recreated under the toolchain root because Ubuntu's UsrMerge keeps them at /, and ld scripts (`libc.so`, `libm.so`) hardcode `/lib/...` paths that --sysroot re-roots into the toolchain. The unversioned `g++`/`gcc`/`cpp` symlinks are replaced with wrapper shell scripts that resolve their own location at runtime and pass `--sysroot=<toolchain>` and `-B <toolchain>/usr/lib/gcc/<triplet>/<ver>/` to the underlying versioned binary. That's how torch's bare `g++ foo.cpp -o foo` invocation finds cc1plus (-B), system headers (--sysroot), and the bundled libstdc++ (--sysroot, --sysroot is recursive into linker). `run.sh` adds the toolchain bin dir to PATH and the toolchain's shared-lib dir to LD_LIBRARY_PATH -- everything else (header search, linker search, executable search) is encapsulated in the wrappers. No-op for non-CPU builds, the dir doesn't exist there. The cpu-vllm image grows by ~217 MB. Tradeoff is acceptable -- cpu-vllm is already a niche profile (few users compared to GPU vllm) and the alternative is a backend that crashes at first inference unless the operator manually sets TORCH_COMPILE_DISABLE=1, which silently disables all torch.compile optimizations. Drops `TORCH_COMPILE_DISABLE=1` from tests/e2e/vllm-multinode -- the smoke now exercises the real compile path through the bundled toolchain. Test runtime is +20s for the warmup compile, still <90s end to end. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(vllm): scope jetson-ai-lab index to L4T-specific wheels via pyproject.toml The L4T arm64 build resolves dependencies through pypi.jetson-ai-lab.io, which hosts the L4T-specific torch / vllm / flash-attn wheels but also transparently proxies the rest of PyPI through `/+f/<sha>/<filename>` URLs. With `--extra-index-url` + `--index-strategy=unsafe-best-match` uv would pick those proxy URLs for ordinary PyPI packages — anthropic/openai/propcache/annotated-types — and fail when the proxy 503s. Master is hitting the same bug on its own l4t-vllm matrix entry. Switch the l4t13 install path to a pyproject.toml that marks the jetson-ai-lab index `explicit = true` and pins only torch, torchvision, torchaudio, flash-attn, and vllm to it via [tool.uv.sources]. uv won't consult the L4T mirror for anything else, so transitive deps fall back to PyPI as the default index — no exposure to the proxy 503s. `uv pip install -r requirements.txt` ignores [tool.uv.sources], so the l4t13 branch in install.sh now invokes `uv pip install --requirement pyproject.toml` directly, replacing the old requirements-l4t13*.txt files. Other BUILD_PROFILEs continue using libbackend.sh's installRequirements and never read pyproject.toml. Local resolution test (x86_64, dry-run) confirms uv hits the L4T index for torch and falls through to PyPI for everything else. Assisted-by: claude-code:claude-opus-4-7-1m [Read] [Edit] [Bash] [Write] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-06 00:22:50 +02:00
Richard Palethorpe	bb033b16a9	feat: add LocalVQE backend and audio transformations UI (#9640) ... feat(audio-transform): add LocalVQE backend, bidi gRPC RPC, Studio UI Introduce a generic "audio transform" capability for any audio-in / audio-out operation (echo cancellation, noise suppression, dereverberation, voice conversion, etc.) and ship LocalVQE as the first backend implementation. Backend protocol: - Two new gRPC RPCs in backend.proto: unary AudioTransform for batch and bidirectional AudioTransformStream for low-latency frame-by-frame use. This is the first bidi stream in the proto; per-frame unary at LocalVQE's 16 ms hop would be RTT-bound. Wire it through pkg/grpc/{client,server, embed,interface,base} with paired-channel ergonomics. LocalVQE backend (backend/go/localvqe/): - Go-Purego wrapper around upstream liblocalvqe.so. CMake builds the upstream shared lib + its libggml-cpu-*.so runtime variants directly — no MODULE wrapper needed because LocalVQE handles CPU feature selection internally via GGML_BACKEND_DL. - Sets GGML_NTHREADS from opts.Threads (or runtime.NumCPU()-1) — without it LocalVQE runs single-threaded at ~1× realtime instead of the documented ~9.6×. - Reference-length policy: zero-pad short refs, truncate long ones (the trailing portion can't have leaked into a mic that wasn't recording). - Ginkgo test suite (9 always-on specs + 2 model-gated). HTTP layer: - POST /audio/transformations (alias /audio/transform): multipart batch endpoint, accepts audio + optional reference + params[*]=v form fields. Persists inputs alongside the output in GeneratedContentDir/audio so the React UI history can replay past (audio, reference, output) triples. - GET /audio/transformations/stream: WebSocket bidi, 16 ms PCM frames (interleaved stereo mic+ref in, mono out). JSON session.update envelope for config; constants hoisted in core/schema/audio_transform.go. - ffmpeg-based input normalisation to 16 kHz mono s16 WAV via the existing utils.AudioToWav (with passthrough fast-path), so the user can upload any format / rate without seeing the model's strict 16 kHz constraint. - BackendTraceAudioTransform integration so /api/backend-traces and the Traces UI light up with audio_snippet base64 and timing. - Routes registered under routes/localai.go (LocalAI extension; OpenAI has no /audio/transformations endpoint), traced via TraceMiddleware. Auth + capability + importer: - FLAG_AUDIO_TRANSFORM (model_config.go), FeatureAudioTransform (default-on, in APIFeatures), three RouteFeatureRegistry rows. - localvqe added to knownPrefOnlyBackends with modality "audio-transform". - Gallery entry localvqe-v1-1.3m (sha256-pinned, hosted on huggingface.co/LocalAI-io/LocalVQE). React UI: - New /app/transform page surfaced via a dedicated "Enhance" sidebar section (sibling of Tools / Biometrics) — the page is enhancement, not generation, so it lives outside Studio. Two AudioInput components (Upload + Record tabs, drag-drop, mic capture). - Echo-test button: records mic while playing the loaded reference through the speakers — the mic naturally picks up speaker bleed, giving a real (mic, ref) pair for AEC testing without leaving the UI. - Reusable WaveformPlayer (canvas peaks + click-to-seek + audio controls) and useAudioPeaks hook (shared module-scoped AudioContext to avoid hitting browser context limits with three players on one page); migrated TTS, Sound, Traces audio blocks to use it. - Past runs saved in localStorage via useMediaHistory('audio-transform') — the history entry stores all three URLs so clicking re-renders the full triple, not just the output. Build + e2e: - 11 matrix entries removed from .github/workflows/backend.yml (CUDA, ROCm, SYCL, Metal, L4T): upstream supports only CPU + Vulkan, so we ship those two and let GPU-class hardware route through Vulkan in the gallery capabilities map. - tests-localvqe-grpc-transform job in test-extra.yml (gated on detect-changes.outputs.localvqe). - New audio_transform capability + 4 specs in tests/e2e-backends. - Playwright spec suite in core/http/react-ui/e2e/audio-transform.spec.js (8 specs covering tabs, file upload, multipart shape, history, errors). Docs: - New docs/content/features/audio-transform.md covering the (audio, reference) mental model, batch + WebSocket wire formats, LocalVQE param keys, and a YAML config example. Cross-links from text-to-audio and audio-to-text feature pages. Assisted-by: Claude:claude-opus-4-7 [Bash Read Edit Write Agent TaskCreate] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-04 22:07:11 +02:00
Ettore Di Giacinto	8edac61e57	feat(ci): allow routing apt traffic through an alternate Ubuntu mirror (#9650) ... * feat(ci): allow routing apt traffic through an alternate Ubuntu mirror Adds opt-in APT_MIRROR / APT_PORTS_MIRROR knobs to all Dockerfiles, the Makefile, and CI workflows so we can fail over to a non-canonical Ubuntu mirror when archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com are degraded (recently observed: multi-day DDoS against the default pool). Defaults are empty everywhere — behavior is unchanged unless a mirror is configured. To enable in CI, set the repo-level GitHub Actions variables APT_MIRROR (and APT_PORTS_MIRROR for arm64 builds). Locally: make docker APT_MIRROR=http://azure.archive.ubuntu.com A small POSIX-sh helper in .docker/apt-mirror.sh rewrites both DEB822 (/etc/apt/sources.list.d/ubuntu.sources, Ubuntu 24.04+) and the legacy /etc/apt/sources.list before the first apt-get update. Dockerfile stages load it via RUN --mount=type=bind, so there is no extra layer and no cache invalidation when the script is unchanged. Reusable workflows also rewrite the runner's own /etc/apt sources before any sudo apt-get call. Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apt-mirror): default to the Azure mirror, visible in the workflow source Bakes Azure (http://azure.archive.ubuntu.com / http://azure.ports.ubuntu.com) in as the default for both Docker builds and runner-side apt — rather than hiding the URL behind a GitHub Actions repo variable that's not visible from the source tree. A new composite action at .github/actions/configure-apt-mirror is the single source of truth for runner-side rewrites. Five standalone workflows (build-test, release, tests-e2e, tests-ui-e2e, update_swagger) just `uses: ./.github/actions/configure-apt-mirror`. Three workflows (image_build, backend_build, checksum_checker) keep an inline bash rewrite, because they install/upgrade git via apt *before* the checkout step (so the local composite action isn't loadable yet). The Azure URL is visible in those files too. The `apt-mirror` / `apt-ports-mirror` inputs of the reusable workflows keep their now-Azure defaults — they still feed the Docker build-args block in addition to the inline runner-side rewrite. Callers (image.yml, image-pr.yml, backend.yml, backend_pr.yml) drop the previous `vars.APT_MIRROR` plumbing and rely on those defaults. Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apt-mirror): drop Force Install GIT, consolidate on the composite action The PPA git upgrade ran add-apt-repository ppa:git-core/ppa, which talks to api.launchpad.net — also part of Canonical's infrastructure and currently returning HTTP 504. The Azure mirror only covers archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com, not PPAs. The system git that ubuntu-latest already ships is sufficient for actions/checkout and the build pipeline, so just drop the upgrade. With that gone, the apt-before-checkout constraint disappears too — all three holdouts (image_build, backend_build, checksum_checker) can now switch to ./.github/actions/configure-apt-mirror like the other five. Net: 0 inline apt-mirror blocks, all 8 workflows route through the composite action. Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-03 23:50:13 +02:00
Russell Sim	18e039f305	fix(ci): fix AMDGPU_TARGETS empty-string bypass in hipblas builds (#9626) ... * fix(ci): fix AMDGPU_TARGETS empty-string bypass in hipblas builds 399c1dec wired amdgpu-targets through the backend_build workflow_call interface, intending the input's default value to cover matrix entries that don't specify targets. However, GitHub Actions only applies a workflow_call input default when the caller omits the input entirely. When backend.yml passes `amdgpu-targets: ${{ matrix.amdgpu-targets }}` and the matrix entry has no amdgpu-targets key, the expression evaluates to an empty string, which is treated as an explicit value — bypassing the default. The result is Docker receiving AMDGPU_TARGETS="" which in turn causes Make's ?= default to be skipped (since the variable is already set in the environment, even to empty), and cmake gets -DAMDGPU_TARGETS= with no targets, so the HIP backend compiles for an indeterminate target rather than the intended GPU list. Fix this at two levels: 1. backend.yml: use a || fallback in the expression so that an undefined matrix.amdgpu-targets never reaches the reusable workflow as an empty string. The target list is the canonical default and lives here. 2. backend_build.yml: remove the now-misleading default value from the input declaration. The default never fired due to the above bug, so keeping it implied a guarantee that didn't exist. 3. backend/cpp/llama-cpp/Makefile: add an explicit $(error ...) guard after the ?= assignment so that if AMDGPU_TARGETS is empty (whether from environment or any future CI wiring mistake) the build fails immediately with a clear message rather than silently producing a binary compiled for an unknown GPU target. Assisted-by: Claude Code:claude-sonnet-4-6 Signed-off-by: Russell Sim <rsl@simopolis.xyz> * fix(build): plumb AMDGPU_TARGETS through to Docker builds The docker-build-backend Makefile macro and Dockerfile.golang did not pass AMDGPU_TARGETS to the inner make invocation, so hipblas builds always used the backend Makefile's hardcoded default GPU targets regardless of what was specified via environment or CI inputs. Signed-off-by: Russell Sim <rsl@simopolis.xyz> --------- Signed-off-by: Russell Sim <rsl@simopolis.xyz>	2026-05-02 15:53:14 +02:00
Ettore Di Giacinto	fe6eb57082	feat(vibevoice-cpp): add purego TTS+ASR backend (#9610) ... * feat(vibevoice-cpp): add purego TTS+ASR backend Wire up Microsoft VibeVoice via the vibevoice.cpp C ABI as a new purego-based Go backend that serves both Backend.TTS and Backend.AudioTranscription from a single gRPC binary. Mirrors the qwen3-tts-cpp / sherpa-onnx pattern so the variant matrix (cpu/cuda12/cuda13/metal/rocm/sycl-f16/f32/vulkan/l4t) and the e2e-backends gRPC harness reuse existing infrastructure. - backend/go/vibevoice-cpp/ - Makefile, CMakeLists, purego shim, gRPC Backend with model-dir auto-detection, closed-loop TTS->ASR smoke test - backend/index.yaml - &vibevoicecpp meta + 18 image entries - Makefile - .NOTPARALLEL, BACKEND_VIBEVOICE_CPP, docker-build wiring, test-extra-backend-vibevoice-cpp-{tts,transcription} e2e wrappers - .github/workflows/backend.yml - matrix entries for all variants - .github/workflows/test-extra.yml - per-backend smoke + 2 gRPC e2e jobs * feat(vibevoice-cpp): drop hardcoded glob detection, add gallery entries Refactor backend Load() to follow the standard Options[] convention used by sherpa-onnx and the rest of the multi-role backends: ModelFile is the primary gguf, supplementary paths come through opts.Options[] as key=value (or key:value for Make-target compat), resolved against opts.ModelPath. type=asr/tts decides the role of ModelFile when neither tts_model nor asr_model is set explicitly. Add gallery/index.yaml entries: - vibevoice-cpp - realtime 0.5B Q8_0 TTS + tokenizer + Carter voice - vibevoice-cpp-asr - long-form ASR Q8_0 + tokenizer Both pull from huggingface://mudler/vibevoice.cpp-models with sha256 verification. parameters.model + Options[] paths are siblings under {models_dir} per the qwen3-tts-cpp convention. Update Makefile e2e wrappers to pass BACKEND_TEST_OPTIONS comma+colon style, and tighten the per-backend Go closed-loop test to use the explicit Options API. * fix(vibevoice-cpp): force whole-archive link so vv_capi_* exports survive libvibevoice is a STATIC archive linked into the MODULE library. Without --whole-archive (or -force_load on Apple, /WHOLEARCHIVE on MSVC), the linker garbage-collects symbols not referenced from this translation unit - which means dlopen+RegisterLibFunc panics with 'undefined symbol: vv_capi_load' at backend startup, since purego looks them up by name and our cpp/govibevoicecpp.cpp doesn't call them directly. * test(vibevoice-cpp): rewrite suite with Ginkgo v2 Match the convention used by backend/go/sherpa-onnx/backend_test.go. The suite now covers backend semantics that don't need purego (Locking, empty-ModelFile rejection, TTS/ASR-without-loaded-model errors) on top of the gRPC lifecycle specs (Health, Load, closed-loop TTS->ASR). Model-dependent specs Skip() when VIBEVOICE_MODEL_DIR is unset, so `go test ./backend/go/vibevoice-cpp/` is green on a clean checkout and runs the heavyweight closed-loop spec when test.sh has staged the bundle. * fix(vibevoice-cpp): implement TTSStream + AudioTranscriptionStream The gRPC server's stream handlers (pkg/grpc/server.go) spawn a goroutine that ranges over a chan; the only thing closing that chan is the backend's own *Stream method. With the default Base stub returning 'unimplemented' and never touching the chan, the server goroutine hangs forever and the client hits DeadlineExceeded - which is exactly what the e2e harness saw in the test-extra-backend-vibevoice-cpp-tts matrix run. TTSStream synthesizes via vv_capi_tts to a tempfile, then emits a streaming WAV header (chunk sizes 0xFFFFFFFF so HTTP clients can start playback before the full PCM lands) followed by the PCM body in 64 KB slices. The header + >=2 PCM frames satisfy the harness's 'expected >=2 chunks' assertion and give a real progressive stream. AudioTranscriptionStream runs the offline transcription, emits each segment as a delta, and closes with a final_result whose Text equals the concatenated deltas (the harness asserts those match). Two new Ginkgo specs guard the close-channel-on-error path so the deadline-exceeded regression can't come back silently. * fix(vibevoice-cpp): silence errcheck on cleanup paths Lint flagged six unchecked Close()/Remove()/RemoveAll() calls along purely-cleanup deferred paths. Wrap each in '_ = ...' (or a closure for defers that take args) - matches what the rest of the LocalAI backend/go/* tree already does for these callsites. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vibevoice-cpp): closed-loop slot fill + modelRoot-relative path resolution Two bugs the test-extra-backend-vibevoice-cpp-* CI matrix surfaced: 1. Closed-loop Load with ModelFile=tts.gguf + Options[asr_model=...] left v.ttsModel empty, because the default-fill block only ran when BOTH slots were empty. vv_capi_load then got tts="" + a voice and the C side rejected it with rc=-3 'TTS model required to load a voice'. Fix: ModelFile fills the *primary* role-slot (decided by 'type=' in Options, defaulting to tts) independently of the secondary, so ModelFile + asr_model resolves to both. 2. resolvePath stat'd CWD before falling back to relTo. With LocalAI launched from a directory that happens to contain a same-named file, supplementary Options[] paths could leak away from the models dir. Drop the CWD probe entirely - relative paths now *always* join onto opts.ModelPath (the gallery convention). New Ginkgo coverage: * 'ModelFile slot resolution' (4 specs) - asr_model+ModelFile, type=asr, explicit tts_model override, key:value variant. * 'resolvePath (relative-to-modelRoot)' (5 specs) - join, abs passthrough, empty input, empty relTo, and the CWD-trap regression test. * 'Load resolves relative Options paths against opts.ModelPath' - end- to-end gallery layout round-trip. Verified locally: 19/19 specs pass (with model bundle, including the closed-loop TTS->ASR; without bundle, 17 pass + 2 model-dependent skip). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(vibevoice-cpp): use gallery convention in closed-loop spec The 'loads the realtime TTS model' / closed-loop specs were passing already-prefixed paths into Options[]: Options: ['tokenizer=' + filepath.Join(modelDir, 'tokenizer.gguf')] Combined with no ModelPath set on the request, the backend's modelRoot fell back to filepath.Dir(ModelFile) = modelDir, then resolvePath joined the prefixed Options path on top of it - producing 'vibevoice-models/vibevoice-models/tokenizer.gguf' when the CI's VIBEVOICE_MODEL_DIR is the relative './vibevoice-models'. The fix is to mirror the gallery contract LocalAI core actually sends in production: ModelPath is the models root (absolute), ModelFile is a name *under* it, every Options[] path is relative to ModelPath. Uses filepath.Base() to get bare filenames. Verified locally with both VIBEVOICE_MODEL_DIR=/tmp/vv-bundle (abs) and VIBEVOICE_MODEL_DIR=vibevoice-models (the relative shape that broke CI). Both: 19/19 specs pass, ~55-60s. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(vibevoice-cpp): switch ASR to Q4_K + bump transcription timeout The Q8_0 ASR gguf is ~14 GB - too big to fit alongside the runner image, the docker build cache, and the test artifacts on a free ubuntu-latest GHA runner; 'test-extra-backend-vibevoice-cpp-transcription' was getting SIGTERM'd at 90 min before the model could finish loading. Switch to Q4_K (~10 GB on disk, slightly faster CPU decode) for: * the e2e harness Make target * the gallery 'vibevoice-cpp-asr' entry (parameters + files block) * the per-backend test.sh auto-download list Bump tests-vibevoice-cpp-grpc-transcription's timeout-minutes from 90 to 150 - even with Q4_K, the 30 s JFK clip on a CPU runner needs runway above the previous 90 min cap. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(vibevoice-cpp): drop transcription gRPC e2e job - too heavy for free runners The vibevoice ASR is a 7B-parameter model. Even on Q4_K (~10 GB on disk) a single 30 s transcription saturates the per-test 30 min timeout in the e2e-backends harness on a 4-core ubuntu-latest, and the 10 GB download + Docker layer + working space leaves no headroom on the runner's free disk. Two attempts in CI got SIGTERM'd at the LoadModel boundary - the bottleneck isn't tunable from the workflow side without a paid-tier runner. The per-backend tests-vibevoice-cpp job already runs the same AudioTranscription path via a closed-loop TTS->ASR Ginkgo spec - same gRPC contract, same model, single process - so the standalone tests-vibevoice-cpp-grpc-transcription job was redundant on top of the disk/CPU pressure. The Makefile target test-extra-backend-vibevoice-cpp-transcription stays for local invocation on workstations that can afford it - useful when developing the streaming codepaths. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(vibevoice-cpp): restore transcription gRPC e2e on bigger-runner Switch tests-vibevoice-cpp-grpc-transcription from ubuntu-latest to the self-hosted 'bigger-runner' label that GPU image builds in backend.yml use, plus the documented Free-disk-space prep step (purge dotnet / ghc / android / CodeQL caches) the disabled vllm/sglang entries in this file describe. That gives the 7B-param Q4_K ASR model the disk + CPU runway it needs. Keep timeout-minutes: 150 - even on a beefier runner the 30 s JFK decode plus 10 GB download has to fit comfortably. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(vibevoice-cpp): apt-get install make on bigger-runner before transcription e2e bigger-runner is a self-hosted bare runner without the standard ubuntu image's preinstalled build tools, so the previous job died at the very first command with 'make: command not found' (exit 127). Add the Dependencies step that the disabled vllm/sglang entries in this file already document - apt-get installs make + build-essential + curl + unzip + ca-certificates + git + tar before the make target runs. Mirrors how every other 'runs-on: bigger-runner' entry in backend.yml prepares the runner. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-29 22:22:14 +02:00
Richard Palethorpe	4443250756	chore: add golangci-lint with new-from-merge-base baseline (#9603) ... * chore: add golangci-lint with new-from-merge-base baseline Configure golangci-lint v2 with the standard linter set (errcheck, govet, ineffassign, unused) plus forbidigo, which enforces the Ginkgo/Gomega-only test convention from .agents/coding-style.md by rejecting stdlib testing calls (t.Errorf, t.Fatalf, t.Run, ...). staticcheck is disabled — the codebase has many pre-existing QF-style suggestions not worth gating on. issues.new-from-merge-base = master makes the lint job a gate for new issues only; the ~1300 pre-existing baseline stays visible via 'make lint-all' for incremental cleanup. CI runs 'make lint'. Backends needing C/C++ headers we don't install in the lint runner are excluded via a deny list in the Makefile (backend/go/{piper,silero-vad, llm}, cmd/launcher). Discovery still flows through 'go list ./...', so new packages are scanned automatically. To make backend/go/{sam3-cpp,stablediffusion-ggml,whisper} typecheckable, move their .cpp/.h sources into cpp/ subdirs (matching qwen3-tts-cpp / acestep-cpp). Without this 'go list' rejects the package because Go does not allow .cpp alongside .go without cgo. Fix two real bugs found by lint in tests/integration/ (run only via 'make test-stores', not default CI): a stale zerolog reference left over from the slog migration (c37785b7) and an unused 'os' import. Assisted-by: Claude Code:Opus 4.7 (1M) [Bash] [Read] [Edit] [Write] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci(lint): generate proto sources and fetch full history The lint job was failing for two reasons: - pkg/grpc/proto/*.go is generated, not checked in. Several packages import it, so without 'make protogen-go' typecheck fails project-wide with "no required module provides package github.com/mudler/LocalAI/ pkg/grpc/proto". - golangci-lint's new-from-merge-base needs to git-merge-base the PR against master, but actions/checkout's default shallow clone doesn't fetch master. fetch-depth: 0 brings full history; the config now references origin/master (the remote-tracking branch that survives the shallow checkout) instead of bare master (which doesn't exist locally after checkout). Assisted-by: Claude Code:Opus 4.7 (1M) [Bash] [Read] [Edit] [Write] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci(lint): stub react-ui/dist for go:embed glob core/http/app.go has //go:embed react-ui/dist/*. The glob must match at least one non-hidden entry or typecheck fails the whole core/http package. We don't need the real React bundle to lint Go code, so just touch an empty index.html to satisfy the embed. Assisted-by: Claude Code:Opus 4.7 (1M) [Bash] [Read] [Edit] [Write] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-04-28 22:07:44 +02:00
Ettore Di Giacinto	a0317d9926	refactor(tests): split app_test.go, move real-backend coverage to e2e-backends ... core/http/app_test.go had grown to 1495 lines exercising three concerns at once: HTTP-layer integration, real-backend inference (llama-gguf, tts, stablediffusion, transformers embeddings, whisper), and service logic that already has unit-level coverage. Each PR paid for 6 backend builds plus real-model downloads to satisfy a single suite. Reorg per layer: - app_test.go (1495 -> 1003 lines) drives the mock-backend binary only. Kept: auth, routing, gallery API, file:// import, /system, agent-jobs HTTP plumbing, config-file model loading. Deleted real-inference specs (llama-gguf chat, ggml completions/streaming, logprobs, logit_bias, transcription, embeddings, External-gRPC, Stores duplicate, Model gallery Context). Lifted Agent Jobs out of the deleted Stores Context. - tests/e2e-backends/backend_test.go gains logprobs, logit_bias, and no-first-token-dup specs (the latter folded into PredictStream). Two new caps gate them so non-LLM backends opt out. - tests/e2e-aio/e2e_test.go gains a streaming smoke under Context("text") to catch container-level streaming regressions. - tests/models_fixtures/ removed; all fixtures referenced testmodel.ggml. app_test.go now writes per-Context inline mock-model YAMLs. CI: - test.yml + tests-e2e.yml gain paths-ignore (docs/, examples/, *.md, backend/) so docs and backend-only PRs skip them. test.yml drops the 6-backend Build step plus TRANSFORMER_BACKEND/GO_TAGS=tts; tests-apple drops the llama-cpp-darwin build. - New tests-aio.yml runs the AIO container nightly + on workflow_dispatch + master/tags. The tests-e2e-container job moved out of test.yml so PRs no longer pay AIO cost. - New tests-llama-cpp-smoke job in test-extra.yml runs on every PR with no detect-changes gate; pulls quay.io/go-skynet/local-ai-backends: master-cpu-llama-cpp (no build on PR) and exercises predict/stream/ logprobs/logit_bias against Qwen3-0.6B. This is the PR-acceptance real-backend gate after AIO moved to nightly. The path-gated heavy test-extra-backend-llama-cpp wrapper appends the same caps so it exercises the moved specs when the backend actually changes. Makefile: - Deleted test-models/testmodel.ggml (the wget chain), test-llama-gguf, test-tts, test-stablediffusion, test-realtime-models. test target drops --label-filter, HUGGINGFACE_GRPC, TRANSFORMER_BACKEND, TEST_DIR, FIXTURES, CONFIG_FILE, MODELS_PATH, BACKENDS_PATH; depends on build-mock-backend. test-stores keeps a focused entry point and depends on backends/local-store. clean-tests also clears the mock-backend binary. Net per typical Go-side PR: ~25min (6 backend builds + tests + AIO) + ~8min e2e drops to ~5min mock-backend test + ~8min e2e + ~5-10min llama-cpp-smoke (image pulled). Docs and backend-only PRs skip the always-on workflows entirely. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: claude-code:claude-opus-4-7 [Edit] [Write] [Bash]	2026-04-27 23:09:20 +00:00
Alex Brick	e5337039b0	[intel GPU support] Use latest oneapi-basekit image for Intel images to support b70 (#9543) ... * Use latest oneapi-basekit image for Intel images The current `localai/localai:master-gpu-intel` images don't work with the intel arc pro b70. Updating the base_image to 2025.3.2 fixes it. Signed-off-by: Alex Brick <3220905+arbrick@users.noreply.github.com> * Update github workflow base image --------- Signed-off-by: Alex Brick <3220905+arbrick@users.noreply.github.com>	2026-04-24 18:29:10 +02:00
Richard Palethorpe	13734ae9fa	feat: Add Sherpa ONNX backend for ASR and TTS (#8523) ... feat(backend): Add Sherpa ONNX backend and Omnilingual ASR Adds a new Go backend wrapping sherpa-onnx via purego (no cgo). Same approach as opus/stablediffusion-ggml/whisper — a thin C shim (csrc/shim.c + shim.h → libsherpa-shim.so) wraps the bits purego can't reach directly: nested struct config writes, result-struct field reads, and the streaming TTS callback trampoline. The Go side uses opaque uintptr handles and purego.NewCallback for the TTS callback. Supports: - VAD via sherpa-onnx's Silero VAD - Offline ASR: Whisper, Paraformer, SenseVoice, Omnilingual CTC - Online/streaming ASR: zipformer transducer with endpoint detection (AudioTranscriptionStream emits delta events during decode) - Offline TTS: VITS (LJS, etc.) - Streaming TTS: sherpa-onnx's callback API → PCM chunks on a channel, prefixed by a streaming WAV header Gallery entries: omnilingual-0.3b-ctc-q8-sherpa (1600-language offline ASR), streaming-zipformer-en-sherpa (low-latency streaming ASR), silero-vad-sherpa, vits-ljs-sherpa. E2E coverage: tests/e2e-backends for offline + streaming ASR, tests/e2e for the full realtime pipeline (VAD + STT + TTS). Assisted-by: claude-opus-4-7-1M [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-04-24 14:40:06 +02:00