119 Commits

Author	SHA1	Message	Date
mudler's LocalAI [bot]	9c85cacfe3	feat(audio-cpp): add the audio.cpp native backend (#11141) ... * backend(audio-cpp): add the native build scaffold Links 0xShug0/audio.cpp engine_runtime through its public framework headers and serves Health/Status. Model loading and the audio RPCs follow. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): keep the build-tree rpath at $ORIGIN Upstream sets CMAKE_BUILD_WITH_INSTALL_RPATH in its own directory scope, so CMake was appending its build-tree library dir to our target and baking an absolute build-host path into the shipped binary. Set BUILD_WITH_INSTALL_RPATH on the target so a package that forgets to bundle libggml*.so fails on the build machine too, instead of only on a user's box. Also document why EXCLUDE_FROM_ALL must stay on the add_subdirectory call, correct the claim that Ubuntu ships no gRPC CMake config, stop the pin comment from repeating the assignment token that bump_deps.sh rewrites, and make test-engine fail rather than pass when no test is registered. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): parse namespaced model options Splits option entries on the first colon so path values survive, and routes load./session. prefixes to the upstream load and session option maps. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): reject out-of-range numeric model options std::atoi is undefined once the digits exceed long and in practice wraps, so device:2147483648 was accepted and handed the ggml backend selector a device index of -2147483648 from a function whose error text promises a non-negative integer. Parse with strtol and reject on ERANGE, on a value above INT_MAX, and on any unconsumed trailing input. The error strings are unchanged. Name the whole entry in the unknown-key error too: an entry like ':value' has an empty key and left the user nothing to grep for in their YAML. Tests look keys up through a helper instead of map::at, so a prefix off-by-one fails one named check rather than aborting the binary and skipping the rest of the suite, and cover the overflow, negative and non-numeric paths. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): route LocalAI RPCs onto audio.cpp tasks Task-major resolution over the family's advertised capability set, with the voice-reference and instructions signals selecting cloning and voice design, and a streaming-to-offline fallback for server-streaming transcription only. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): use upstream's 'spk' task name and pin the preference order The SpeakerRecognition short name was 'spkrec', which audio.cpp neither prints nor parses; a name copied out of audio.cpp was rejected and a pinned 'spkrec' would not survive the engine boundary. Emit 'spk', keep 'spkrec' as an input-only alias, and correct the known-tasks lists. Three assertions were vacuous because their fixtures advertised a single task, so reversing a preference order or dropping the RPC name and the attempted pairs from the capability error all passed. Give them fixtures that can tell the orderings apart. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): convert sample, time and PCM units Integer nanosecond conversion so 44.1 kHz stays exact, float seconds for the VAD and diarization messages, and saturating s16le encode so an overshooting sample cannot wrap to the opposite sign. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): harden seconds_to_samples against NaN and overflow seconds_to_samples is the one entry point fed by untrusted-shaped input: a float-seconds timestamp off the wire, or a boundary from a model that diverged. Its guard covered only the low side, so NaN and out-of-range values fell through to an undefined double-to-int64 cast and came back as INT64_MIN. A hugely negative sample index used later as an offset or a length is a wild pointer rather than merely a wrong timestamp. Reject NaN with the !(x > 0) form and saturate before the cast. Also round instead of truncating there. These functions exist to cross the float seconds boundary the VAD and diarize messages use, and truncation lost a sample about half the time on the samples-to-seconds-and-back round trip, starting at n=1. Pin the decode scale at INT16_MIN, pin nanosecond truncation on a nonzero fraction, and record why the clamp argument order in f32_to_s16le is load-bearing for NaN. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): map NaN PCM samples to silence explicitly f32_to_s16le relied on std::min argument order to keep a NaN sample away from std::lround, whose result is unspecified for NaN. That was too subtle to rest on a comment, and the comment was itself wrong: it warned against a spelling that the outer std::max already catches, while three real spellings leak, including std::clamp, which is the idiomatic C++17 way to write the same clamp and so the likeliest future edit. Divert NaN before the clamp and encode it as 0. A NaN sample rendered as a full-scale click is worse audio than a dropped one, and this unit converts audio that may have originated off the wire. Pin it with an exact-value check rather than a range check, since all three outcomes the plausible spellings produce are finite and inside full scale, plus an invalid-operation check that fails unless the NaN is diverted before any ordered comparison. That second check is what catches modernizing the clamp and dropping the guard together. Also bound the seconds round-trip comment, which claimed unconditionally what holds only below roughly 2^23 samples, and document NaN, saturation and that bound in the header. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): assemble transcripts from runtime spans The top-level transcript text is TaskResult.text_output verbatim. audio.cpp carries text nowhere else: speech_segments, speaker_turns and word_timestamps hold spans and labels only, so deriving the text from them empties the transcript for any producer that omits word timing, VibeVoice diarized ASR included. Fixtures cover every observed producer shape. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): keep a nested speaker turn's own label A segment sourced from speaker_turns re-derived its speaker by greatest overlap. A turn's overlap with its own span is the largest possible, so a turn nested inside another speaker's turn could only tie with the container, and the tie went to whichever came first. sortformer_diar binarizes each speaker independently and sorts by start sample, so the container always comes first and the interjecting speaker was silently erased from DiarizeSegment.speaker. choose_segment_spans now carries the label out with the span. Also pins the nearest-segment fallback against measuring from either endpoint or from segment position, which a trailing-only stray word could not do, and exercises the empty-word guard in join_words. Two fixtures that pin a rule but do not mirror any pinned family are relabelled defensive. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serialize runs with a wedge-aware guard audio.cpp sessions are not reentrant and a wedged CUDA call cannot be cancelled, so a plain mutex would pile every worker thread behind a stuck GPU. Callers waiting past the configured bound, or arriving while the holder has already overrun it, fail fast instead. A caller that queues behind a healthy run deliberately does not stamp the clock: only the thread that takes the lock does. Stamping on arrival would restart the wedge clock on every request and hide a stuck run from everyone behind it, which is the pile-up this guard exists to prevent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serialize inference through an InferenceLane One audio.cpp model is loaded per backend process and its sessions are not reentrant, so concurrent gRPC handlers have to take turns. Serialization alone is not enough: a wedged GPU call cannot be cancelled from userspace, so an unbounded queue behind one stuck run would swallow every gRPC worker thread until the process is useless. InferenceLane gives handlers a lane with room for one runner. LaneEntry occupies it for a scope and gives it back on every exit, including an exception, and is the only way to take the lane at all: occupy/vacate are private with LaneEntry as the sole friend, so a caller cannot acquire without holding something that releases. LaneEntry is immovable on purpose, because a moved-from entry would have to stop releasing while the lane still recorded it as occupied. A caller either waits indefinitely or brings a millisecond budget. A bounded caller that cannot get in fails instead of waiting on, and a bounded caller whose budget is already shorter than the age of the run in the lane fails immediately, which is what stops a queue forming behind a wedged run. The two failures carry different text: one names the wait it exhausted, the other states the measured age of the run without claiming to know why it is long, since a short budget meeting a legitimately long run lands there too. The run's age is stamped only after acquisition. A waiter that published itself as holder would restart the measurement and hide a genuinely stuck holder from every caller behind it. Budget negotiation and the overrun decision are pure functions taking their inputs explicitly, so both are covered without threads or sleeping. The per-model ceiling arrives as an int of milliseconds; a request may tighten it and may never loosen it. Replaces the previous run_guard unit, which was a derivative of an Apache-2.0 file upstream and could not stay in an MIT tree. Written from a behaviour contract with no reference to the removed code. Tests: 65 checks, standard library only, single translation unit, clean under -Wall -Wextra. Mutation tested at 23/23 killed; two of those mutants exposed missing coverage and the tests were extended until they died. ThreadSanitizer clean. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the B10 test able to fail, and document LaneEntry Review of the previous commit found the B10 test could not fail for the reason it was named. It aged the in-flight run to about 120 ms and then tried two budgets, 30 ms and 60 ms, both under that age, so both callers took the fail-fast path. "The two failure modes do not share one message" was comparing two fail-fast messages that differ only in the budget they print, and the timeout path was never reached. The second budget is now 400 ms, well over the run's age, so that caller queues and times out, and a new check asserts which path each caller took instead of inferring it from inequality. A mutant that makes the fail-fast path emit the timeout message previously died only on B4 and B8 checks; it now also dies on B10. Comment-only changes elsewhere. LaneEntry now says it is not reentrant and does not detect reentrancy: a second entry on a thread that already holds the lane surfaces as LaneUnavailable with a positive budget, but parks silently in unbounded mode, which matters because a handler may hold one across a whole stream. The immovability note now names the shapes that work, an optional emplaced in place or a unique_ptr, rather than saying to hold the entry indirectly without saying how; all three documented forms were compiled before being written down, which is how the note came to say that an optional of an immovable type cannot itself be returned. The header's explanation of why fail-fast exists is reworded. Two clauses traced back to a specification written after reading the Apache-2.0 upstream header, and while that was judged de minimis, this unit was rewritten precisely to carry no upstream expression at all. The margin table in the report was also wrong about which wall-clock margins are load-sensitive: there are four, not one, and the tightest is the B3 arrival check, which is now flagged at the call site. No margin value changed and none moved across 65 runs. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): gate model loading on the audio.cpp family Refuses any GGUF without an audiocpp.model_spec.family key and any non-GGUF path without an explicit family option, so the model loader's greedy backend probe cannot bind an unrelated llama.cpp GGUF to this backend (#9287). Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): load models and cache sessions per task Loads one ILoadedVoiceModel and creates an IVoiceTaskSession lazily per (task, mode), so the same model serves both the unary and streaming RPCs. LoadModel derives the family from GGUF metadata or an explicit option and fails with INVALID_ARGUMENT otherwise, so a failed load is a gRPC error the backend probe can see. audiocpp_backend::Task mirrors engine::runtime::VoiceTaskKind positionally, and drift there is silent: every unit still compiles and every test still passes while the backend runs a different task. Two mechanisms pin it. The static_asserts in loaded_model.cpp catch an insertion or a reorder, and -Werror=switch on that one file turns an appended upstream enumerator into a build failure rather than a warning in a 600 file log. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop aborting the process on SIGTERM The signal handler called grpc::Server::Shutdown directly. Shutdown takes an absl::Mutex, which is not async-signal-safe: the handler can interrupt a thread already holding that mutex, and abseil's deadlock detector responds by aborting. Every SIGTERM therefore ended in exit 134 and a 'dying due to potential deadlock' stack rather than a drained shutdown. The handler now sets a lock-free atomic and returns. Server::Wait moves to a helper thread so the main thread can poll that flag and call Shutdown itself, outside any signal context. A condition variable would not have helped, because notifying one from a handler is not async-signal-safe either. SIGTERM and SIGINT both exit 0 with no stack trace, where both previously exited 134. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): correct the status, lifetime and state contracts of LoadedModel An environment fault during session creation was reported as UNIMPLEMENTED. A missing libggml-cpu-*.so surfaced to the client as 'family silero_vad advertises vad/offline but refused to create the session: Failed to initialize CPU backend', which tells LocalAI the model cannot do this and must never be retried, and sends an operator hunting a capability bug instead of a packaging one. A throw from create_task_session is now a plain runtime_error, so it maps to INTERNAL. Only a null return, where the family genuinely declined, stays a CapabilityError. The model.'s task: option was parsed and then dropped: it lived in a local that died at the end of LoadModel and had no route to RequestShape::pinned_task. LoadedModel now keeps it and exposes pinned_task(). The global model becomes a shared_ptr reached through snapshot(). An audio RPC runs for seconds and cannot hold g_model_mu for its duration, so under a unique_ptr a Free arriving mid-request would destroy the model underneath it. Handlers now take a counted reference and whichever finishes last does the teardown, outside the lock. session_for documents the streaming state contract rather than resetting the session itself. Resetting on a cache hit was tried first and is not possible: silero_vad throws 'session prepare() must be called before Silero VAD reset()', so it would turn an ordinary second fetch into a hard error. start_stream's base implementation is already a reset, so a caller that runs prepare then start_stream per stream gets a clean session; a probe against the bundled silero_vad confirms an identical replay when it does and a carried-over stream when it does not. Also: an unknown backend: name is rejected before the model loads rather than after; MainGPU is parsed instead of passed through std::atoi, which turned 'gpu1' into device 0 silently; and device carries a device_set flag, because 0 is both the default and a real device index, so MainGPU was overriding an explicit device:0 that the neighbouring threads: handling promises will win. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the VAD and Diarize RPCs Both emit float seconds, converted from the runtime's sample-index spans, and both take a counted reference to the loaded model through snapshot() and hold it for the whole call: a Free arriving mid-request drops only the global's reference, so whichever request finishes last destroys the model instead of one of them running on freed weights. An AddressSanitizer build reproduces exactly that heap-use-after-free inside ggml_vec_dot_f32 when the handler keeps a raw pointer instead, which is why the shape is what it is. The inference lane is taken before session_for, not after. session_for reads and writes an unsynchronised session cache and the offline run calls prepare(), which mutates the session, so both belong inside the lane. Diarize routes before it reads the input file, so a family that cannot diarize at all says so rather than complaining about the audio first. Its per-segment text stays empty because audio.cpp's SpeakerTurn carries a span and a speaker label only, and nested or overlapping turns are passed through untouched: a sortformer turn inside another speaker's turn is correct output for overlapped speech, and LocalAI is overlap-tolerant downstream. Duration counts frames rather than floats, so a stereo input does not report twice its length. Verified end to end against upstream's bundled silero_vad, which needs no download, using the bundled 16 kHz speech asset: a synthetic tone returns nothing, correctly, because silero detects speech and a sine is not speech. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): enforce ModelIdentity on VAD and Diarize audio-cpp was the only C++ backend without the model-identity guard, and no later task in the plan added it. pkg/grpc/server.go enforces checkModelIdentity on exactly these two RPCs, for the reason #10952 records: in distributed mode a worker can recycle a stopped backend's gRPC port for another model's backend, and the controller's liveness-only probe cannot tell a stale cached route from a live one. Without this guard a stale route gets a different model's VAD or diarization answer back with a 200. The loaded identity lives on LoadedModel rather than in a separate global, which is where this differs from llama-cpp. A handler holding the model through snapshot() then necessarily judges against the identity that model was loaded with, and a concurrent reload cannot swap one without the other. The refusal is NOT_FOUND carrying the verbatim grpcerrors.ModelMismatchSentinel substring. session_for and run_offline now take a const LaneEntry & proof-of-holding parameter. The rule that both must run under the inference lane was prose, which is exactly how the plan came to specify the inverted order; it is now a compile error. Restoring the inverted order fails to build rather than racing on an unsynchronised session map with a mutating prepare(). Diarize's speaker-hint comment claimed the dropped hints were "not a silent failure". From the caller's side that is what they are, and backend.proto documents num_speakers as forcing, so the comment now says plainly that the forwarding is dead for sortformer and that the family which lands must either honour num_speakers or refuse it. read_audio_file inspects the error_code from exists(), so an unsearchable parent directory no longer reports as a missing file. The VAD handler records the stimulus that actually works, since silero correctly ignores synthetic tones and the next task would otherwise rediscover that. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the lane and identity guards structural Two hardenings ahead of the eleven handlers still to be written, both of which get harder to retrofit later. The lane proof-of-holding parameter was a const reference, which binds to a temporary, so session_for(rpc, shape, model->acquire(0)) compiled. Each such temporary dies at the end of its own full-expression, releasing the lane between two calls that must share one: precisely the split the parameter exists to prevent, and the form a future author is most likely to reach for because it reads as tidy. A non-const reference requires an lvalue, so the temporary form now fails to compile while the named-local handlers build unchanged. The header comment no longer implies the check is total either: it proves a lane was taken, not that it is this model's lane. The identity check was two lines each handler had to remember, with nothing failing if a new one forgot them and no C++ equivalent of model_identity_modalities_test.go to notice. snapshot() becomes snapshot_unchecked(), whose only legitimate caller is Status, since HealthMessage carries no ModelIdentity. Handlers go through snapshot_for(), which takes the counted reference, refuses when nothing is loaded, and runs the identity check before anything can route. Every handler already has to call something to obtain the model, so the guarded call is now the shortest path and skipping it means deliberately typing snapshot_unchecked. A convention that has to be remembered can rot; this cannot. Verified: the temporary-argument and inverted-order forms each fail to compile with the expected diagnostic, the real handlers build, and bypassing the guard in Diarize alone turns the identity test red on that RPC while VAD stays green. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the AudioTranscription RPC Adds result_map, the engine-to-proto boundary, and wires the offline transcription RPC. The handler branches on the ROUTED task: for Asr the request's prompt is whisper-style decoding context and becomes a request option, for Alignment the same field IS the transcript to align and becomes the text input. Routing has already decided which. The result text is TaskResult.text_output verbatim and is never derived from the segments. audio.cpp carries transcript text in text_output and nowhere else, so deriving it returns an empty transcript for every producer that reports segments without word timing. transcript_assembly already enforces that; this commit's job is not to undo it at the proto boundary, and result_map_ctest pins it there. read_audio_file now takes the sample rate the caller needs. Both file-fed speech handlers ask for 16 kHz mono, for two reasons: silero_vad and sortformer_diar refuse anything else outright, which turned an ordinary 44.1 kHz upload into INTERNAL, and nemotron_asr emits word timestamps in its own 16 kHz feature domain whatever the input was, so only a 16 kHz buffer makes the emitted nanoseconds right. Zero keeps the file's native rate and channels, which is what source separation will need. LoadedModel::check_can_serve answers a capability refusal before the lane is taken and before the input file is read. Routing is a pure read of the immutable capabilities, so a model that cannot serve an RPC no longer waits out somebody else's run to say so. VAD and Diarize use it too. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop linking sentencepiece's vendored protobuf engine_runtime links sentencepiece, whose default SPM_PROTOBUF_PROVIDER builds the protobuf-lite 3.14.0 sources it vendors. The generated backend.pb.cc is built against the toolchain's protobuf 3.21.12. Both ended up in the binary: 476 google::protobuf:: symbols came from the archive, 278 of them also defined by libprotobuf.so, and the archive won, because once ld pulls a member in for sentencepiece's own code every reference binds to the definitions that member carries. The visible symptom is one function. ParseContext::ParseMessage(MessageLite*, const char*) is what a generated _InternalParse calls for a submessage field and for nothing else, so flat messages parsed and nested ones did not: a TranscriptResult carrying segments serialized to correct bytes that the same process could not read back, and TranscriptLiveRequest, a oneof of submessages, could not have been parsed at all. Underneath that, 3.21 generated code was running 3.14 arena, ArenaStringPtr and ExtensionSet code. -Wl,--exclude-libs does not fix it. It makes those symbols LOCAL in .dynsym and the parse still fails, because the binding was decided at static link time and no visibility flag revisits it. Setting SPM_PROTOBUF_PROVIDER to "package" before add_subdirectory points sentencepiece at the protobuf the generated code was already built against. Zero google::protobuf:: definitions remain in the executable afterwards, every nested message round trips, and citrinet_asr, which parses a SentencePiece ModelProto at load time and would break first if this were wrong, still tokenizes and transcribes correctly. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): fix the segment text a transcription response is built from Segment text is not decoration. core/http/endpoints/openai/transcription.go routes response_format text, srt, vtt and lrc through schema.TranscriptionResponse, which builds the entire body out of Segments[].Text and never reads the top-level text. So for those four formats the segment text IS the response. nemotron_asr emits one word_timestamp per SentencePiece token, and the word boundary is carried as a LEADING SPACE on the piece ("So", "me", " call"). join_words inserted a space unconditionally, so response_format=text returned "So me call me na ture ," while the correct sentence sat unread in the top-level field. The separator is now chosen from the words themselves: whole words are space-joined, subword pieces are concatenated, and one leading space anywhere selects the latter. Concatenating the real nemotron pieces reproduces text_output exactly, verified end to end. This does not touch the top-level text, which is still text_output verbatim. The rule that forbids deriving the transcript from the segments is about the direction segments -> text; segment text has no source other than its words. Two smaller corrections in the same area: timestamp_granularities ["word"] set only "word_timestamps", a key no family in the pinned upstream reads. It now sets "return_timestamps", which qwen3_asr does read and which both runs its forced aligner and shortens its chunk window, so asking for word granularity no longer silently returns nothing. The request-option comment claimed more than it delivered. prompt, translate and temperature are read by no ASR family, and are forwarded only so a family adopting them works unchanged; the comment now says so per key, and gives TranscriptRequest.diarize the same explicit treatment threads already had. Also: the shipping target now carries -Wall -Wextra -Wpedantic, which it never did, so "the build is clean" starts meaning something; and fill_transcript_result no longer swallows a null response pointer, since answering OK with an empty transcript is the one failure mode this unit exists to prevent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the AudioTransform RPC Covers voice conversion, singing voice conversion, speech to speech and source separation, the four tasks LocalAI's AudioTransform can represent. AudioTransformResult carries one dst while htdemucs and mel_band_roformer produce several named stems from a single run, so inference runs ONCE, every stem is written as a sibling file <dst-stem>.<name>.<ext>, and params[stem] selects which one dst receives, defaulting to vocals and falling back to the first output. An unknown stem name is INVALID_ARGUMENT listing the real stem names rather than a silent substitution, and the selection happens before the first write so a refused request leaves no files behind. params[stem] is consumed here and is not forwarded into the engine's request options. The stem decision lives in stem_selection, which is stdlib only and therefore tested by backend/cpp/run-unit-tests.sh. It also validates the names, because they come from the model (htdemucs reads them from the GGUF's config.sources) and each becomes a component of a path this backend writes: a name carrying a path separator would escape the caller's output directory, and two stems sharing a name would silently overwrite one another. Both files are read at their native rate and channel count. Separation forces it, since demucs and roformer refuse any rate but 44.1 kHz and lose the stereo image that separates a centred vocal from a wide mix. The conversion families all resample internally (seed_vc, vevo2, miocodec, chatterbox were each checked), so passing the file through unchanged is also strictly better than band limiting it to 16 kHz first. Verified end to end against htdemucs f16 on a 44.1 kHz stereo mix: four stems plus dst, dst byte identical to the selected stem, params[stem] selecting a different one, an unknown stem refused with no files written, and mono input preserved as mono output. Also against miocodec for the single output path, where params[stem] is refused rather than ignored. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): refuse an impossible stem early, and stop blaming the caller for a failed write Four fixes from the first review of the AudioTransform RPC. check_can_serve now returns the resolved route, so params[stem] on a route that is not source separation is refused from the route instead of after a full inference: 11 ms rather than the 4.5 s a miocodec conversion costs, and far worse on seed_vc or vevo2. The post-run refusal stays as the backstop for a separation-routed family that returns no stems anyway. The typo'd-stem-name case still needs the run, since no framework header publishes the stem names before one. Stem names carrying control bytes are refused. GGUF strings are length prefixed and demucs reads its sources from JSON, so an embedded NUL survives to here: two names differing only after the NUL are distinct std::strings, so the duplicate check passes them, and then path::c_str() truncates both and they open the same file. That is exactly the silent overwrite the duplicate check exists to prevent, with the .wav lost as well. A failed write is now INTERNAL rather than INVALID_ARGUMENT. The destination is LocalAI's own generated-content directory, not anything the caller named, so a full disk or a permission fault there is a server fault and is worth retrying, which is the opposite of what INVALID_ARGUMENT tells a client. An empty output path stays INVALID_ARGUMENT. Two comment corrections and one clarification: the separators' required rate is their checkpoint's declared samplerate rather than a hardcoded 44100, seed_vc resamples with soxr and falls back to sinc-hann, and the "no files left behind" guarantee covers a refused request, not a write that fails partway through the loop. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(audio-transform): stop folding every upload to 16 kHz mono, and name the separation stems Two defects that made source separation unusable through LocalAI's own API, even though the backend served it correctly over gRPC. /audio/transform normalized every upload to 16 kHz mono s16 through utils.AudioToWav, with no way past it. htdemucs and mel_band_roformer refuse any rate but their checkpoint's own and separate a centred vocal from a wide mix using the stereo image, so every separation request through the HTTP API died with "HTDemucs prepare() sample rate mismatch: expected 44100, got 16000" while the same call over gRPC worked. The fold is not wrong, it is backend-specific: LocalVQE's echo cancellation genuinely wants 16 kHz mono and needs the reference in the same shape. So it becomes a declaration, BackendCapability.AudioTransformInputMono16k, set for localvqe and for nothing else. A backend that declares nothing gets its upload unchanged, which means no backend has to opt in to work. utils.AudioToWavPreservingShape is the non-folding conversion: a 16-bit PCM WAV passes through byte for byte at any rate and channel count, anything else is transcoded to WAV with its rate and channel layout kept. The other defect is that the run-once stem design bought nothing. A separation backend writes every stem beside dst from one inference, but AudioTransformResult carried only dst, so the other three were files no caller could find and a caller wanting all four had to run four separations. AudioTransformResult grows a repeated AudioTransformStem, the backend fills it, core/backend validates that each path really is inside the generated-content directory it handed over, and the endpoint publishes them as an X-Audio-Stems JSON header beside the existing X-Audio-Input-Url. JSON because a stem name is the model's own string and could contain any separator a hand-rolled format would use. Verified end to end through the HTTP endpoint with htdemucs f16 on a 44.1 kHz stereo file: 200 with a 44.1 kHz stereo body, all four stems named and fetchable through /generated-audio/, body byte identical to the selected stem, and params[stem]=drums returning a different one. The same upload sent to a model whose backend is localvqe still reaches the backend as 16 kHz mono, confirmed both by the engine's own rate refusal and by the persisted input file. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(audio-transform): reject extensible WAV from the passthrough, escape stem URLs, convert stems with dst Four fixes from the second review, plus one bug they made visible. isPCM16Wav tested only the bit depth, and go-audio's IsValidFile never looks at the format tag, so a 16-bit WAVE_FORMAT_EXTENSIBLE (0xFFFE) upload was passed through untouched where the old fold would have transcoded it. audio.cpp's WAV reader accepts 16-bit only when the tag is 1, so such a file died with "unsupported WAV encoding". Extensible is what many DAWs and Windows tools write and music files are this endpoint's new headline input, so it is a first-contact failure rather than a corner. The check now requires tag 1, with a spec that fails against the old implementation. Stem URLs are percent-escaped. A stem name is the model's own string and legally contains a space, a '#', a '?' or a '%'; an unescaped '#' truncates the URL before the request is even sent. The name field keeps the raw name. sample_rate and response_format are applied to the stems as well as to dst. Applying beat documenting: dst IS one of those stems, so leaving them alone broke the "dst duplicates the selected stem" invariant the whole design rests on, and both conversions are no-ops when unset. A stem whose conversion fails is dropped from the header rather than advertised in the wrong shape. Verifying that turned up why it had never been noticed: the two fields were never bound at all. The request arrives as multipart/form-data and echo's binder falls back to the FIELD NAME without a form tag, matching only case-insensitively, so "SampleRate" never matched "sample_rate" and "Format" never matched "response_format". Both were documented in the endpoint table and silently ignored. Two form tags fix it, and with them the conversion is observable end to end. Docs: audio-transform.md now documents what LocalAI does to an upload before the backend sees it, which backend gets the 16 kHz mono fold and why, params[stem], and the X-Audio-Stems header with a worked example. Also records the known limitation that the fold lookup is on the bare backend name, so pinned variants (vulkan-localvqe) do not match, and points at IsLlamaCppBackend as the suffix-tolerant precedent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the TTS and SoundGeneration RPCs TTSRequest.voice is treated as a speaker reference clip when it names an existing regular file, which makes routing prefer VoiceCloning, and as a named preset otherwise, in which case it travels as VoiceReference::cached_voice_id. Both the clip and SoundGenerationRequest.src are read at the file's own rate and channel count: upstream's own CLI and server do exactly that, every consuming family resamples internally and mostly with a better resampler than ours, and ace_step and stable_audio resample their input per channel, so a downmix here would delete the stereo image they are built to consume. The request builders live in their own unit rather than in grpc-server.cpp's anonymous namespace so they can be tested; grpc-server.cpp has a main() and cannot be linked into a test binary. The option keys are the whole point of these functions, so each one was grepped against the pinned upstream and the accounting is written down beside it. instructions maps to "instruct", which is what upstream's own server maps the OpenAI field to and what qwen3_tts and omnivoice read, and to "caption" for irodori_tts; the style tag is spelled "instruct" too, because "instructions" is looked up nowhere. duration maps to "duration_seconds", read by all three generation families, with the proto's own name kept only as a forward-tolerant alias. Keys that no family reads say so. Both handlers answer a capability refusal before taking the lane and before any file read, so a model that cannot synthesise does not queue behind somebody else's run to be told no. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop emitting an empty style language, and name the missing clip StyleCondition::language was set whenever has_language() was true, with no !empty() guard, while the language option twelve lines below had one. core/backend/tts.go sets Language unconditionally, so has_language() is true on every request LocalAI sends and carries "" when the caller named none. An engaged-but-empty style language is worse than an absent one: supertonic reads text_input->language behind its own !empty() guard and then overrides it from style->language with no guard at all, so "" replaced its "en" default and its tokenizer threw "invalid Supertonic language: ". Every /v1/audio/speech request that set instructions and no language would have been an INTERNAL against a supertonic model. A plain request never saw it, because the style condition only exists when instructions are non-empty, which is why the chatterbox end to end run did not catch it. TTS also stops discarding the Route that check_can_serve already returns. A family routed to voice cloning without a reference clip used to be refused from inside its own prepare(), which meant an INTERNAL naming neither the RPC nor the field to set; chatterbox advertises clon and no tts, so that was every preset-only request to it. It is now an INVALID_ARGUMENT naming TTSRequest.voice, answered in about 4 ms, and it cannot misfire because has_voice_reference is what selected cloning in the first place. Reading CapabilitySet::supports_speaker_reference to generalise this stays a follow-up. The src read carries a written caveat rather than a family blocklist, because ace_step's editing routes legitimately need src: setting src on a stable_audio model corrupts the heap and aborts the process in the pinned upstream, and the only thing keeping that off the network is that schema.ElevenLabsSoundGenerationRequest has no field for it. Nobody reading that Go schema would know why, so the reason is recorded where the field is read. build_tts_shape is extracted so TTSStream cannot describe the same request differently, and it arrived untested: two mutations of it survived until a test_tts_shape case was added. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the TTSStream and AudioTranscriptionStream RPCs TTSStream leads with a streaming WAV header carrying 0xFFFFFFFF sizes, matching the convention backend/go/vibevoice-cpp established, so an HTTP client can start playback before the full PCM exists. Its chunks are read from StreamEvent::named_audio_outputs and not audio_output: supertonic, omnivoice and voxcpm2 all put their streamed audio there and leave audio_output empty until the very end, so reading the obvious field yields a stream with no audio in it. The finish_stream result is the family's own merged whole rather than a tail, so it is emitted only when nothing was streamed. Streaming transcription sends incremental deltas and degrades to a single delta plus the final result on families that offer no streaming ASR, which is the same message sequence with fewer deltas. The four streaming ASR families disagree on what partial_text means: nemotron_asr, vibevoice_asr and higgs_audio_stt report incremental fragments while voxtral_realtime reports the whole hypothesis and reports it twice, so the reconciliation lives in one tested unit rather than in the handler. nemotron_asr reports only through the stream event sink, and only from inside finalize, so the audio driver installs one and clears it again before returning: the session is cached and a sink left holding the caller's frame is a use after free waiting for the next stream. begin_stream is now the only implementation of the streaming state obligation, prepare then start_stream. Streaming sessions are cached, and what clears the previous stream is start_stream's reset; a family override that dropped it would break every call site with no compile error, so there is one call site. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): keep streaming deltas on UTF-8 boundaries, refuse dtypes that abort TranscriptStreamResponse.delta is a proto3 string, whose wire format requires valid UTF-8. voxtral_realtime reports its hypothesis as a concatenation of raw token BYTES (tokenizer_text.cpp:171-183), so the cumulative difference between two consecutive reports is eventually a lone continuation byte, and the C++ runtime serializes that with only a logged warning while the Go runtime refuses to unmarshal it: the client loses the remaining deltas AND the final_result. Measured on a trace of a non-ASCII sentence, 11 of 31 messages failed to unmarshal and every accented character was lost. TranscriptDeltaTracker now holds back an incomplete trailing sequence and merges it into the next fragment; reconcile flushes it, which it always can because the final text is complete. The same trace now unmarshals in full with zero failures. A streaming buffer whose float count is not a whole number of frames is refused rather than truncated. The integer division dropped the tail floats from the fed audio and therefore from the transcript, with no diagnostic; vibevoice_asr refuses the same thing from the other side of the call. A supertonic GGUF whose weights are not f32 is refused at load. It reaches ggml_concat with mismatched operand types and ggml_abort takes the whole backend process down on the first request, so nothing downstream can report it: the model loads, then every request kills the process. Attributed rather than assumed, the unary TTS path aborts identically, and upstream records that package as untested. The refusal names the orig package and says what to run before deleting the guard. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop a repeated lead byte from orphaning the next delta The first UTF-8 fix closed the cumulative half only. Rule 2 discards a fragment the known text already starts with, and when that fragment is the LEAD BYTE of a new character it looks exactly like a repeat of an older character beginning with the same byte. It was discarded rather than held, its continuation bytes then arrived alone and began the next delta, and utf8_complete_prefix_length only ever inspected the trailing sequence, so a delta invalid at the FRONT went out whole. Through a real Go proto.Unmarshal the review's four-character repro gave 3 deltas, 2 unmarshal failures and a lost transcript. Reachable from the incremental families, not only from voxtral: nemotron_asr's decoder cuts at a byte offset and vibevoice_asr's common_prefix_size compares bytes, so both split characters. Measured over 30,000 randomized incremental traces, 53.28% of Japanese traces and 9.52% of French ones carried at least one delta the Go runtime refuses. Two changes. Rule 2 no longer judges a fragment that ends mid-character, so the lead byte is held instead of swallowed and the character survives intact; the cost is a few duplicated bytes in a shrinking cumulative report, which no pinned family produces. release() additionally drops leading orphan continuation bytes, so no delta can begin mid-character whatever the rules above it decide. Losing a byte keeps the stream alive; emitting one ends the RPC and takes the final_result with it. Post-fix all 60,000 traces produce zero unmarshal failures, and the cumulative streams plus both pure-ASCII incremental streams are byte-identical to the previous commit, so nothing changed for the families already working. The weight-dtype allow list moves to family_gate, where it is stdlib-only and pinned by a test rather than only by a comment. Two comment citations corrected. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): read only an exact repeat as a repeat, not any prefix Rule 2 discarded any partial the known text merely started with. For a cumulative family that is a duplicate; for an incremental family it is an ordinary short fragment that happens to coincide with the start of the transcript, and it was dropped, silently corrupting the text. Pure ASCII, no multi-byte character anywhere: the fragments "pure ", "ascii ", "trans", "c", "ri", "p", "t" left the client holding "pure ascii transcrit". Over 5,000 randomized traces per transcript, 9.50% of pure-ASCII and 29.12% of French traces ended with the client holding something other than final_result.text, with a 200 and no diagnostic. Both incremental families emit fragments that small routinely, since nemotron_asr cuts at a byte offset and vibevoice_asr at a common prefix. Narrowing rule 2 to an exact repeat drives that to zero on all six transcripts and changes no cumulative stream at all: 30,000 randomized cumulative traces are byte-identical to the previous commit. What rule 2 guarded was established from upstream rather than from its own comment. The only duplicate any pinned family produces is voxtral_realtime's, where process_available_stream_chunks feeds each event to the sink from inside its loop and returns the last of the batch, so that event arrives twice with byte-equal text. A duplicate is an exact repeat, so equality still covers it. The case given up is a cumulative report that SHRINKS, which no pinned family can produce: voxtral decodes a token vector that is only push_back'ed and cleared by reset(), so within a stream it can only grow. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the AudioTranscriptionLive RPC The one bidirectional stream this backend serves. The client sends a TranscriptLiveConfig, then TranscriptLiveAudio frames; the server acknowledges with ready, emits deltas as the audio arrives, and sends final_result once the read side closes. There is no offline fallback: live transcription has to consume audio incrementally, so a family with no streaming ASR is refused rather than served a batch run, which is what this RPC's Streaming-only mode_candidates list already says. The driver is a new sibling of run_streaming_audio, run_streaming_live, because the audio does not exist yet: instead of slicing a buffer it pulls frames from the caller until the read side closes. It installs the same ScopedStreamSink in the same order, which is not optional, since nemotron_asr returns a bare event from process_audio_chunk and reports every partial through the sink from inside finalize(). It buffers the wire's frames up to the family's own preferred window rather than feeding whatever size the client's audio callback produced, and it does not call finish_stream at all when no audio arrived, because nemotron_asr throws "finalize requires streamed audio" and an empty transcript is the truthful answer to transcribing nothing. Three things the handler had to get right and one it cannot: - The audio contract. A live request carries no samples, but nemotron_asr's streaming prepare() throws without an audio contract, and build_preparation_request derives it from TaskRequest::audio_input, so that field is an EMPTY buffer holding only the rate and the channel count. - 16 kHz or a refusal. The families express their spans in their own 16 kHz feature domain whatever the input was, and live frames cannot be resampled on the way in the way a file can, so an 8 kHz session would return timestamps 2x off with a 200. core/backend hardcodes 16000 anyway. - A mid-stream Config is refused. backend.proto calls it a decoder reset, but deltas already on the wire cannot be retracted, so a reset would leave the final text contradicting the transcript the client assembled. Ignoring the message would hand a client that believes it reset the decoder a transcript that silently continues the audio it thought it discarded. - The stale-route identity check cannot run here: TranscriptLiveRequest carries no ModelIdentity in either arm of its oneof, so snapshot_for does not instantiate for it. snapshot_unchecked's comment now names that as a second legitimate class of caller and says the fix is a proto change. eou and eob stay false. They exist for cache-aware models that emit end-of-utterance and end-of-backchannel tokens; audio.cpp's StreamEvent has no equivalent signal, and a client uses eou to decide the speaker yielded the turn, so a guess inferred from silence cuts people off mid-sentence. The lane is held for the whole stream, which is as long as the user keeps talking: the streaming session is stateful and cached, so a concurrent run would interleave two callers' audio and corrupt both transcripts. Verified against nemotron_asr over a real connection with a 14 s WAV in 512-sample frames: ready first, 59 incremental deltas with no repeated prefix, concat(deltas) equal to final_result.text, word timestamps in nanoseconds, eou and eob false. citrinet_asr answers UNIMPLEMENTED naming the family and listing asr/offline. A config followed by a close returns an empty final_result rather than hanging, and a first message that is not a config is INVALID_ARGUMENT. Two concurrent streams both return the complete transcript. Two cleanups on lines Task 12 touched, folded in. The DtypeAllowList terminator is now asserted at compile time: the reported out-of-bounds read did not exist, the single entry does terminate, but the loops have no other bound and any edit that widened an entry would walk off the end. And the dtype guard now short-circuits on "is there a table entry" through a new predicate rather than on the emptiness of the description string, which would have skipped the check on an entry with an empty allow list, i.e. on precisely the entry that refuses every dtype. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): bound the lane a live stream can hold AudioTranscriptionLive holds the model's inference lane for the whole stream, which is correct (the streaming session is stateful and a concurrent run would interleave two callers' audio) and newly dangerous. Every other RPC holds the lane across compute, or across a write to a slow reader, and both of those terminate on their own. A live stream instead blocks in a client-driven read, and a peer that goes silent WITHOUT closing the stream never terminates anything: the lane stays taken and every other request against that model queues behind a client that stopped speaking. live_watchdog is a one-shot idle timer that ends the stream when no frame has arrived inside a window. It is standard library only, so it is unit tested without an engine. gRPC's synchronous Read has no timeout and cannot be given one, so the only way to unblock it is ServerContext::TryCancel, which decides the wire status itself: the client sees CANCELLED rather than the DEADLINE_EXCEEDED the handler returns, the reason is logged, and the lane coming back is the point. When it fires the read loop throws rather than reporting end-of-input, so the driver does not go on to finalize a decode nobody is waiting for. It is armed only after the lane is taken and disarmed as soon as the read side closes, and both ends matter. Arming earlier would cover acquire(), which legitimately blocks while another live stream runs, so a queued caller would be cancelled for waiting its turn. Disarming later would cover our own decode, where a window overrun is not a peer going quiet and cancelling would throw away the transcript the client is waiting for. The window is the new live_idle_timeout_ms option, 30 s by default, 0 meaning no limit. core/http/endpoints/openai/realtime.go drives a 300 ms ticker and feeds every tick that produced new audio while a turn is open, so 30 s of silence is a hundred ticks that delivered nothing. It is also longer than any pause a speaker takes mid-utterance, which is the case that must never be cut off, and backend.proto lets one stream span many utterances, so a client that pauses longer between them raises the option rather than discovering it. Two smaller corrections in the same handler: - check_can_serve now runs BEFORE the sample rate check. pkg/grpc/grpcerrors/errors.go degrades to the file path on UNIMPLEMENTED and on nothing else, so a live-incapable model asked at a wrong rate was answering INVALID_ARGUMENT and costing the caller its fallback. - a negative sample rate is refused instead of silently becoming 16000. Zero still means 16000, which is what the proto documents; -1 is malformed rather than absent and gets the same refusal every other bad rate gets. And one thing recorded rather than changed, at the handler: "live" here means incremental INPUT, not low latency, and with the pinned families it does not yet mean incremental OUTPUT either. nemotron_asr's process_audio_chunk only appends to its buffer, so its whole decode and every delta happen inside finalize(), after the client closes its send side. The policy-window buffering is inert for that family and matters only for vibevoice_asr and higgs_audio_stt. Verified on the wire with live_idle_timeout_ms:3000. A silent client acked at 371 ms and was cancelled at 3.371 s; a second live stream opened one second later received its ack 2.37 s in, i.e. at the instant the first was cancelled, and then transcribed successfully on the same cached session. Without the watchdog it would still be waiting. Re-ran the live transcription (ready first, 59 incremental deltas, concat equal to the final text, word timestamps in nanoseconds, eou and eob false), the citrinet refusal at both a right and a wrong rate (UNIMPLEMENTED either way now), and Task 12's AudioTranscriptionStream on nemotron_asr, which is unchanged. Mutation testing the watchdog found a weakness in its own test: the destructor test slept past the window inside the watched scope, so a destructor that DETACHED the thread instead of joining it passed unnoticed. The test now uses a window longer than the scope, which kills that mutant, and says why. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): refuse the unsupported RPCs with a reason AudioEncode, AudioDecode, AudioTransformStream, AudioToAudioStream and VoiceEmbed have no counterpart in audio.cpp's VoiceTaskKind. Each now returns UNIMPLEMENTED naming the loaded family, what that family does support, and the upstream limitation, instead of the generated base class's bare status. The reasons live in a table in capability_routing.cpp so they are data rather than literals copied into five handlers, and so a test can assert every one of them. The five claims this was planned against were re-read at the pinned upstream e800d435d130dc776baf6f3e6129bb62b1495c89, and one did not hold. "audio.cpp streams tts and asr only" is false: silero_vad advertises vad with RunMode::Streaming. The refusal stands on the narrower claim that survives, that no family advertises streaming for any task AudioTransform routes to, and a test asserts the refuted wording does not come back. VoiceEmbed is the one refusal whose request carries a ModelIdentity, so it runs the #10952 check before answering: a stale route must get NOT_FOUND and the router's sentinel, not "audio.cpp cannot embed speakers" about a model that is not loaded here. It cannot use snapshot_for, whose no-model branch would tell the caller to load a model when no model can help, so it takes the reference through snapshot_unchecked and checks identity itself. That function's comment now names three classes of caller instead of two. The two bidirectional surfaces refuse without reading their stream, verified with a client that writes a config and eight frames first and gets the status rather than hanging. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): correct the vevo2 clause, and assert the absences Review found a false clause in the AudioToAudioStream refusal. It said s2s is "offline voice conversion ... which converts one clip into another speaker's voice", which is true of miocodec and false of vevo2: vevo2's s2s route is `editing` and only `editing` (default_route_for_task and route_matches_task in src/models/vevo2/session.cpp), documented as "Edit source speech into new target text while using the target voice" and requiring --target-text, so it rewrites what was said. vevo2's voice conversion is its separate vc task. It now reads "offline clip-to-clip processing against a target voice, declared only by miocodec (voice conversion) and vevo2 (speech editing)", and a test asserts the miscast cannot come back. The conclusion is unchanged: neither family converses. That defect was undetectable on the wire, since vevo2 does not load here, which is the argument for upstream_absence_ctest.cpp. It links engine_runtime purely to interrogate make_default_registry() and asserts the five premises the refusal reasons rest on: no codec task kind, no family advertising spk, no streaming for sep/vc/svc/s2s, miocodec advertising exactly vc and s2s, and s2s advertised by exactly miocodec and vevo2. The last two are exact sets, so an addition fails here rather than leaving a message stale. A positive control proves the registry is populated and the query works before any absence is believed, and every assertion has a reproduced negative control. This turns an AUDIO_CPP_VERSION bump from "remember to re-read five prose paragraphs" into a test failure. unsupported_surface now switches over UnsupportedRpc with no default label, so -Wswitch reports a sixth enumerator added without a row at build time; the runtime bounds guard it replaces is deleted. The AudioTransformStream reason had a true premise and an overreaching conclusion: an offline sep family could be buffered into a stream, as other LocalAI backends do. It now says this backend declines to offer a buffered offline call in disguise, rather than implying impossibility. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the missing-switch-case diagnostic fatal unsupported_surface() switches UnsupportedRpc onto the table row that explains it, with no default label, so -Wswitch reports an enumerator nobody handled. As a warning that is not enough: adding a sixth enumerator and building the shipping target gives exit 0, a binary and one warning, and the trailing `return surfaces[0];` then answers the new RPC with AudioEncode's codec reason. That is a confident, specific and false statement about audio.cpp on the wire, on the one code path whose entire job is to be truthful about what this backend cannot do, and it is worse than the runtime fallback it replaced, which at least named itself as a bug in this file. capability_routing.cpp therefore joins loaded_model.cpp on the existing -Werror=switch pin, whose comment already made this argument for the engine enum. The comment now covers both files. The pin stays per-file rather than project-wide because upstream's own ace_step/vae_decoder.cpp has unhandled -Wswitch cases of its own. Verified: a sixth enumerator now fails `make grpc-server` with exit 2 and no binary; appending a 14th VoiceTaskKind upstream still fails loaded_model.cpp, so the two pins fire independently; both reverted clean. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): package the backend image Bundles the dependency closure for the from-scratch image, the dlopened ggml CPU-variant shared objects that ldd cannot see, and upstream's bundled silero_vad and marblenet_vad assets so VAD works with no download. The bundled loader sits in the package ROOT rather than at lib/ld.so. run.sh execs it, which makes /proc/self/exe name the loader, and this backend has two consumers of that path: ggml discovers the libggml-cpu-*.so by listing dirname(/proc/self/exe), and resolve_model_path expands bundled:<name> under the same directory. Rooting the loader makes the binary, the ggml objects and assets/ share the one directory all three resolution mechanisms agree on. llama-cpp's lib/ld.so layout would need assets/ moved into lib/ as well. The image builds against apt gRPC and protobuf, like Dockerfile.ds4 and unlike Dockerfile.privacy-filter. The from-source gRPC that install-base-deps.sh and the base-grpc-* images supply vendors protobuf 26, which pulls abseil into message_lite.h; with SPM_PROTOBUF_PROVIDER=package that collides with sentencepiece's vendored mini-abseil and every absl::internal reference becomes ambiguous. Noble's protobuf 3.21.12 predates the abseil dependency and is the pair every earlier verification of this backend ran against. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): exempt the driver libraries from the packaging gate package.sh already left libcuda.so* and libnvidia-* to the host when copying, because the driver has to match the kernel module on whatever host runs the image, but the validation gate had no matching exemption. With BUILD_TYPE=cublas ggml is static and links CUDA::cuda_driver, so grpc-server carries DT_NEEDED libcuda.so.1 and the gate would have rejected the very absence the copy loop created, failing every cublas build in CI. One regex now feeds both. Building a control for that found a second defect: ld.so --list refuses to trace an object with an unresolvable dependency at all, exiting 127 without emitting a per-library line, so the "=> not found" rule was dead code and no exemption could have applied to it. The gate now traces with LD_TRACE_LOADED_OBJECTS and LD_LIBRARY_PATH, which reports the missing name and exits 0, and which is also what run.sh does at run time. Adds a layout assertion so a future move of the loader into lib/ fails the build instead of shipping a package that resolves bundled: models into lib/assets and finds no ggml CPU backend, and records for Task 16 that the Darwin script must not be a straight copy of privacy-filter-darwin.sh, which never calls package.sh and would silently drop assets/. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): register the backend with CI and the gallery Adds the five Linux matrix entries (cpu amd64/arm64 sharing a tag-suffix so the manifest merge fires, cuda 12, cuda 13, vulkan), the path-filter case that keeps later PRs touching backend/cpp/audio-cpp/ from getting zero CI jobs, the bump-bot entry pointing at the AUDIO_CPP_VERSION pin in the backend Makefile, the gallery meta plus its -development variant and the image entries for every variant, and the Makefile docker-build wiring. The matrix entries carry base-image only, with no builder-base-image, unlike the llama-cpp and privacy-filter blocks they sit next to. The prebuilt quay.io/go-skynet/ci-cache:base-grpc-* images ship a from-source gRPC whose protobuf v26 depends on abseil, and this backend's sentencepiece is built with SPM_PROTOBUF_PROVIDER=package, so it sees real abseil's absl::lts_20240116:: internal alongside its own vendored plain absl::internal and every absl::internal:: reference becomes ambiguous. Building against base-grpc-amd64 fails at sentencepiece-static.dir/error.cc.o with "reference to 'internal' is ambiguous". Dockerfile.audio-cpp installs apt's gRPC/protobuf 3.21.12 itself, which is also the pair every unit and end-to-end run of this backend has been verified against, and the CUDA toolkit therefore has to come from base-image. No Darwin matrix entry and no metal gallery entries: the Metal build needs scripts/build/audio-cpp-darwin.sh, a backends/audio-cpp-darwin make target and a routing step in backend_build_darwin.yml, none of which exist yet, so an entry added now would be routed to build-darwin-go-backend and look for backend/go/audio-cpp/. The inferBackendPathDarwin case and the DARWIN_BESPOKE_BUILDERS membership are in place, inert, so that adding the entry later is a one-line change that cannot be claimed by the generic Go path. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): pin the CUDA architectures, drop the vulkan variant Upstream sets CUDA_ARCHITECTURES to `native` on the engine_runtime target whenever CMAKE_CUDA_ARCHITECTURES is unset at root scope, and docs/build/ linux.md says so outright. ggml's own default does not rescue it: it list(APPEND)s in the ggml subdirectory scope, which never reaches the root scope where the engine_runtime property is decided. No CI runner has a GPU for `native` to enumerate, so both cublas entries would have gone red on the very commit that first turns a CUDA build on. Pin the list in backend/cpp/audio-cpp/Makefile, selected by CUDA_MAJOR_VERSION, which Dockerfile.audio-cpp now forwards from the CI build-arg it was previously discarding. The values are copied from ggml's own version guards rather than invented, so engine_runtime and ggml compile for the same set: CUDA 12 keeps the Maxwell/Pascal/Volta virtual archs and stops at 120a-real, CUDA 13 drops them and adds 121a-real. The `a` suffix is used rather than `f` because the latter needs CMake 3.31.8 and Ubuntu Noble ships 3.28.3. Verified by driving CMake 3.28.3's own CUDA architecture validator over both lists, with 120f-virtual as the rejected control. Drop the vulkan matrix entry, its two gallery entries, the vulkan capability key on both metas and the Vulkan tag. Every other vulkan backend gets its Mesa ICD drivers from .docker/install-base-deps.sh, which package-gpu-libs.sh then bundles; Dockerfile.audio-cpp calls neither and installs only libvulkan-dev and glslc, so the image would ship a Vulkan loader that finds no GPU. No CI job runs a vulkan image against real hardware, so that would have passed green and failed in users' hands. BUILD_TYPE=vulkan stays supported for local builds. Also note on the cublas entries that cuda-major-version now selects the architecture list and that cuda-minor-version and the base-image tag encode the same toolkit, and correct the stale entry counts on matrixEntryKey. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): build for Darwin Metal Bespoke C++ Darwin path like ds4 and privacy-filter: an includeDarwin matrix entry, a backends/audio-cpp-darwin make target, a gated workflow step, and the metal image entries plus metal/metal-darwin-arm64 capability keys in the backend gallery. The build script deliberately does NOT reassemble the package the way privacy-filter-darwin.sh does. It runs the backend's own `make package` and copies the result, so the Darwin package keeps the root-level layout the Linux one has: grpc-server, run.sh, the ggml objects and assets/ in one directory, with lib/ for the dylib closure. Hand-assembling would drop assets/, and assets/ is what makes the bundled: model paths resolve with nothing downloaded. The dylib walk is a full transitive closure rather than the single level ds4 and llama-cpp do, because Homebrew's grpc++ pulls libgrpc, abseil, upb, cares and OpenSSL that grpc-server does not link itself, and a level-1 walk ships a package that only works on a machine that already has Homebrew grpc. Two fixes folded in, both in the backend Makefile: - an EMPTY CUDA_MAJOR_VERSION fell through to the CUDA 12 architecture list, which contains 120a-real and so needs nvcc >= 12.8. A local BUILD_TYPE=cublas build on a 12.0-12.7 host failed to compile where upstream's documented default (native) worked. EMPTY now maps to native, 12 and 13 keep their lists, and any other non-empty value is an error on cublas builds. CI always passes a major, so CI is unaffected. - the Darwin branch now points CMake at Homebrew's keg-only libomp. AppleClang ships no OpenMP runtime and nothing is symlinked into /opt/homebrew, so FindOpenMP finds neither the library nor the header, and audio.cpp calls find_package(OpenMP REQUIRED) whenever ENGINE_ENABLE_OPENMP is on. Without the hint the macOS build would have died at configure time. If the keg is absent the build disables OpenMP instead of failing. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the Darwin fallbacks loud and the rpath walk complete Review follow-up on the Darwin Metal build. The OpenMP fallback was silent. If brew --prefix libomp ever comes back empty, CI produced a green Metal package with 108 #pragma omp directives across ~30 files compiled out, and clang says nothing about an ignored omp pragma without -Wsource-uses-openmp, so the only trace was one absent flag inside a set -x cmake line. That regression would have been blamed on Metal. It now warns. The @rpath arm of the dylib walk had no live candidate when it was written, on the reasoning that a Metal build links ggml statically. The OpenMP fix in the same commit made libomp.dylib one, and whether Homebrew records it as an absolute opt path or as @rpath/libomp.dylib is not observable from Linux. The walk now expands @rpath, @loader_path and @executable_path against the object's own LC_RPATH entries, and only fails when nothing on disk answers, printing the rpath list with the error so a failure on a machine nobody can attach to explains itself. Also: ADDITIONAL_LIBS now go through the closure rather than a bare cp, so they are deduplicated and their own dependencies bundled; build/darwin/lib is created explicitly instead of relying on package.sh pre-creating it; the libomp probe uses nested ifneq rather than $(and ...), which needs GNU make 3.81 and would otherwise expand empty and take the OFF branch on an older make; and -DOpenMP_ROOT is quoted like its CUDA sibling. Verified with a Linux harness that runs the script verbatim against a stubbed otool: a level-2 transitive dep, an @rpath dep reachable only through LC_RPATH, and an ADDITIONAL_LIBS dep are all bundled, a dependency cycle terminates, system libraries are skipped, the packaged tree has assets/ at the root beside grpc-server with the dylibs in lib/, and both failure paths exit non-zero. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make bundled: reachable from a model YAML resolve_model_path() tested the bundled: prefix on `candidate`, which prefers ModelFile and falls back to Model. LocalAI fills ModelFile by joining ModelPath onto the configured model string (pkg/model/loader.go, LoadModelWithFile), and only sets it from a managed artifact otherwise, so a model YAML saying `model: bundled:silero_vad` arrives as ModelFile "/models/bundled:silero_vad" and Model "bundled:silero_vad". The prefix therefore never matched through the normal load path: it matched only for a hand-written LoadModel call that left ModelFile empty, which is exactly how task 15 verified it, and every model YAML using the form failed with "model path does not exist: /models/bundled:silero_vad". Both fields are now checked, Model first, so the zero-download VAD path the package ships assets for is reachable the way it is documented. A caller that puts the form in ModelFile still works, so task 15's verification stands. Compiled clean; the runtime check could not run on this host, whose system libprotobuf/libre2 have gone missing (the pre-existing grpc-server binary no longer resolves its libraries either), so it wants a container run. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): advertise the backend and document its options Registers audio-cpp as preference-only in /backends/known: the family lives in GGUF metadata that an importer cannot read from a remote repo, and one repo hosts thirty families, so there is no honest auto-detect signal. Modality is a single string and the import form chips on a fixed key set, so it registers as tts with the other modalities named in the description rather than under an invented key the UI would bucket as "other". Adds a features page covering the option namespacing, the routing table per endpoint, the RPCs this backend declines and why, the bundled VAD path, the separation stem behaviour, and the family gotchas (supertonic needs the orig package; chatterbox advertises cloning and no plain tts; nemotron_asr defers its whole decode to finalize so live transcription emits nothing until the client half-closes, unlike higgs_audio_stt and voxtral_realtime). Every option name and family capability in it was read off the pinned upstream checkout. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): test resolve_model_path, and correct the family names The bundled: fix in 842443cd7 shipped without a test, which is how the bug got there: task 15 verified the form with a hand-written LoadModel that left ModelFile empty, and that is the one shape the server never produces. Four cases in streaming_driver_ctest, which already links loaded_model.cpp, pin the PRODUCTION shapes instead. The first fails against the pre-fix source (returns the joined /models/bundled:silero_vad); the other three are the branches the bundled: lookup now runs in front of and must fall through for. Three family names in the docs were the source directory rather than the registered family, on pages whose whole argument is that these names cannot be guessed: demucs is htdemucs (demucs/loader.cpp:22), roformer is mel_band_roformer (roformer/assets.h:15), and moss is TWO families, moss_tts_local and moss_tts_nano. The hyphenated ASR names are underscored to match, here and in the compatibility table. The supertonic dtype note claimed more than the evidence carries. The f16 abort is a local observation, identical through TTS and TTSStream; upstream's docs/gguf.md leaves the 16-bit column untested and records q8_0 as "No (unsupported weight dtype)", which says unusable rather than fatal. Both are still refused, because the allow list is what the family can run. Corrected in family_gate.h, family_gate.cpp and the docs together, since the docs inherited the wording from the code. The importers tripwire says in the file that it is a tripwire: it exercises no audio-cpp behaviour, and the registration assertion lives in backend_test.go. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: add audio.cpp models covering every served RPC One representative model per RPC group of the audio-cpp backend, plus the two bundled VAD models, which need no download at all because the assets ship inside the backend package. Every hash was computed with sha256sum on the downloaded file. Quantizations come from upstream's tested-status table in docs/gguf.md rather than a default of q8_0: supertonic ships the orig package (its q8_0 is recorded as an unsupported weight dtype and its f16 aborts in ggml_concat), and nemotron_asr and htdemucs ship f16 because their q8_0 builds are recorded with drift while 16-bit is a clean pass. Diarization and separation use the diarization and audio_transform usecases, not transcript: /v1/audio/diarization and /audio/transform filter the default model on FLAG_DIARIZATION and FLAG_AUDIO_TRANSFORM respectively, so a transcript flag would have hidden both models from their own endpoints. The forced aligner sets parameters.language, which the transcription endpoint uses as the fallback when no language form field is sent, because the family requires both a transcript and a language. All ten entries were run twice: once against the raw gRPC server, and once installed with local-ai models install and called through the HTTP endpoint. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: correct the audio.cpp entries' licenses Swept all ten entries against the real upstream named in audio.cpp's tools/model_manager.py rather than against the audio.cpp repo's own license. Three were wrong: supertonic apache-2.0 -> openrail weights come from mlx-community/supertonic-3-mlx, and both it and Supertone/supertonic are openrail citrinet apache-2.0 -> other pulled from NGC nvidia/nemo/stt_en_citrinet_256, governed by the NGC Terms of Use sortformer other -> cc-by-nc-4.0 nvidia/diar_sortformer_4spk-v1 is CC BY-NC 4.0, and the gallery already uses that exact string, so there is no reason to obscure a non-commercial bar The license field is one word, so citrinet and sortformer also gained a sentence saying why they are restricted. The other seven were confirmed correct against their sources. Also drops an unverified claim from the nemotron description. It said the model drives the realtime transcription session; that endpoint actually calls TranscribeStream, and the live RPC reaches LocalAI only through realtime_semantic_vad.go. Neither path was exercised here, so the description now states only the two calls that were. MarbleNet gains the NeMo upstream under urls: for parity with silero. No sha256, quantization, usecase or model choice changed. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(audio-transform): bound sample_rate, keep same-named uploads apart Four defects the whole-branch review found on the Go side, plus two comment corrections. sample_rate is a disk-exhaustion hazard. The branch added the `form:` tag that makes the field bind for the first time, so the resample path went from dead to live, and utils.AudioResample interpolates the int straight into ffmpeg's -ar with no bound. Measured with ffmpeg 7: -ar 999999999 on a 0.01 s clip writes 20 MB and exits 0, which scales linearly to the reported 3.9 GB for one second, into a GeneratedContentDir nothing sweeps, and convertStems repeats it once per separation stem. Clamped to 8000..192000 in the handler, before the temp dir and before the model is touched, and rejected with a 400 outside it. The low end was reported as "a 0-byte file". It is not: -ar 1 writes a 78-byte header with no audio behind it, whose declared data size still claims 70 bytes, so go-audio parses it as a 35 SECOND file and a size check does not see it. The guard therefore compares the declared data chunk against the bytes actually on disk, and AudioResample now fails rather than returning a WAV carrying nothing. Both parts of a transform request land in one temp dir, and the raw copy was named only after the client's basename, so `-F audio=@mic/clip.wav -F reference=@loopback/clip.wav` wrote "raw-clip.wav" twice. Since AudioToWavPreservingShape hardlinks an already-PCM16 WAV rather than copying it, the reference part's os.Create truncated the inode audio.wav pointed at: mic and reference came out identical, which makes an echo canceller null everything and return near-silence with a 200. The raw copy now carries the form field name. audio-cpp had no BackendCapabilities entry, so VoiceCloningForModel returned nil before it ever consulted the model's tts.voice_cloning override and every `voice: "profile:<id>"` request was refused with a 400, on a backend that ships audio-cpp-chatterbox whose family serves cloning and not plain TTS. Registered with its RPCs, usecases and the reference-audio contract, and deliberately without the 16 kHz mono fold, which its separation families cannot survive. GetBackendCapability was exact-match only, so every pinned gallery variant read as an unknown backend: vulkan-localvqe lost the 16 kHz mono fold that used to be unconditional and started failing inside LocalVQE, and the usecase gate does not stand in for it because BuildFilteredFirstAvailableDefaultModel returns early once the client names a model. Lookup now falls back to the meta name by stripping the gallery's hardware prefix and release-channel suffix, exact match first so nothing can be shadowed. Same class as #10945. Also corrected: the AudioTransformRequest comment claimed echo's binder falls back to the field name, which it does not in either direction (bindData binds ONLY tagged fields and `continue`s otherwise; `model` arrives from setModelNameFromRequest's c.FormValue). And the stable_audio `src` heap corruption caveat now lives on ElevenLabsSoundGenerationRequest, where the Go developer who would add the field can see it, instead of only in C++. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(audio-cpp): refuse a task pin the RPC cannot serve, and stop empty frames holding the lane The model's `task:` option is copied into the request shape by all nine handlers, which is correct, but resolve_route then replaced the RPC's candidate list with the pin WHOLESALE and never asked whether the pin was something that RPC routes to. One pin therefore bled across all nine surfaces, and because the family still supported the pinned task the result was a wrong 200 rather than an error. Reproduced live: nemotron with task:asr made Vad return 200 with zero segments after a full ASR decode, so 14 seconds of speech was reported as silence, and Diarize did the same; silero_vad with task:vad made AudioTranscription return 200 with empty text and four segments whose spans were VAD segments, which combined with response_format in {text,srt,vtt,lrc} building the body solely from Segments[].Text yields a well formed SRT of four timed EMPTY cues. It also contradicted the documented contract, that a family which cannot serve a request is refused rather than rerouted. A pin is now checked against the RPC's admissible task set before it is adopted, and the refusal names both the pin and the RPC. The set is derived from task_candidates with every shape flag set rather than restated, so a task added to an RPC's candidates cannot become inadmissible by omission. Every legitimate pin survives, and the test asserts all fifteen of them alongside the eight crossings that must not. The live watchdog was defeated by empty frames. idle.touch() ran on ANY message, before the has_audio and pcm.empty() filters, so a peer writing unset-oneof or zero-length frames faster than the window held the lane indefinitely while feeding the decoder nothing. There is one lane per model and one model per process, so that is a single client denying the whole backend, which is what the watchdog exists to prevent, and the thrown text already said "no audio frame arrived". The touch moved below the filters, which are now a named predicate so the distinction is testable rather than a call order nobody can see. Three comments corrected against measurement rather than reasoning: - CMakeLists claimed zero google::protobuf:: definitions remain in the executable. nm -C --defined-only reports 2515, and that is expected: they are generated code, sentencepiece::ModelProto's own _InternalParse among them. The claim that holds, and the one the ABI fix is actually about, is that no vendored protobuf RUNTIME is linked and ParseContext::ParseMessage is UNDEFINED in the executable, resolving to libprotobuf.so. - refuse_cloning_without_a_clip's "cannot misfire" paragraph had its reasoning backwards. Routing picks VoiceCloning as the FALLBACK when there is no clip, which is the case being caught; chatterbox, which ships in the gallery, advertises clon and no tts at all, so every voice-less request lands there. - audio_units read "2.1 min at 96 kHz" for index 11289602, which is 1.96 min. 2.1 min is 96 kHz's OWN first failure at 12288002. Both were remeasured and the note is now a per-rate table. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(audio-cpp): exclude the upstream checkout from the C++ gate, harden the darwin walk run-unit-tests.sh pruned */llama.cpp/* but not */audio.cpp/*. It is safe today only by luck: upstream's 44 tests all put "test" at the FRONT of the filename (17 test-*.cpp, 27 test_*.cpp, zero *_test.cpp), so the glob misses every one of them, and nothing enforces that. This gate runs on every PR for every backend and compiles each match as a standalone translation unit with nothing but nlohmann/json on the include path, so the day upstream adds or renames one test the gate goes red repo-wide on an Apache-2.0 file nobody here wrote. audio-cpp-darwin.sh now logs the raw otool -L output and the parsed LC_RPATH list unconditionally, before the walk. Both awk filters in that script assume a column layout nobody working on this can observe, since it runs only on the CI Mac, and a green first Darwin run proves nothing about the assumption: an awk that silently matched nothing yields an empty dependency list, which reads exactly like "no non-system dependencies" and packages happily. Both filters otherwise feed process substitutions, so their input never reached the log. It also lists every symlink in the package and fails on one that cannot resolve inside the image. A dangling link does not fail anything else here, because every assertion tests with -e, which follows links; it fails at dlopen on a user's Mac. Links are NOT banned outright, which the review suggested but which would break the libggml.dylib -> libggml.0.dylib chain the `cp -a` above exists to preserve. What is banned is a link that resolves on the build host and will not resolve in the image: a broken one, or an absolute one pointing outside the package. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(audio-cpp): drop em dashes from the audio-cpp capability entry Follow-up to a84b3c4b9, no behaviour change. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): key the voice-cloning model rule on the resolved backend Making GetBackendCapability strip the gallery hardware prefix and release channel fixed pinned variants of /audio/transform, but VoiceCloningForModel kept keying its per-backend switch on the caller's spelling. A pinned name therefore resolved the capability by stripping and then missed every case in the switch, falling through to the permissive default: cuda12-vibevoice-cpp advertised voice cloning for the realtime 0.5B model, metal-coqui for tacotron2, cuda12-crispasr for a pure ASR model, cpu-qwen3-tts-cpp for CustomVoice. Each of those is a model that cannot clone, so /v1/audio/speech accepted a profile: voice it had to fail on inside the backend rather than rejecting it with a 400, and the UI advertised the capability too. resolveBackendCapability now returns the key the entry was found under, and callers that branch on backend identity use that key instead of the name they were handed. The exact-match-first order is unchanged, so a backend genuinely registered under a variant-looking name still keys on its own name. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * gallery(audio-cpp): declare audio_transform on the chatterbox entry Chatterbox advertises VoiceCloning AND VoiceConversion (src/models/chatterbox), and the entry's own description already said so, but known_usecases listed only tts. /audio/transform selects its default model by FLAG_AUDIO_TRANSFORM, so voice conversion was reachable only by naming the model explicitly and was invisible to every usecase-driven surface. It is the one audio.cpp task with a shipped gallery model and no way to find it. Verified against the real model rather than inferred from the capability list: AudioTransform with chatterbox-q8_0, speech as audio_path and a speaker clip as reference_path, returns a 5.08 s 24 kHz mono WAV at -25.5 dB mean and zero stems, which is the single-output shape voice conversion should have. The description now says which endpoint reaches that half and warns that installing this next to a source-separation model gives /audio/transform two candidates, so the model should be named rather than defaulted. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * gallery(audio-cpp): add voice-design and singing-voice-conversion entries Two of the three audio.cpp task kinds that had no gallery model now have one. Both were driven end to end against the real weights through the backend before being written, not inferred from the capability tables. audio-cpp-irodori-voicedesign covers vdes. TTS carrying `instructions` routes to the vdes task, so the voice is described in words rather than supplied as a clip. Verified: "a calm elderly woman speaking slowly with a warm, gentle tone" over an 8.76 s 48 kHz mono render at -16.8 dB mean, and a closed-loop citrinet pass recovers the sentence with the accent drift expected from a Japanese-first model read by an English recogniser. audio-cpp-seedvc-singing covers svc, and pins task:svc because nothing else can reach it. seed_vc advertises svc and ordinary voice conversion, no request signal means "this input is singing", and auto-routing resolves the tie to voice conversion every time. Verified with the pin: 5.04 s 44.1 kHz output whose closed-loop citrinet transcription is exact. s2s deliberately has no entry, and the reason is not effort. miocodec is the only upstream family whose speech-to-speech route needs no text, and it returned audio with correct duration and level but no recoverable speech in four independent attempts: the stale build, v2 q8_0, v2 orig (the variant upstream records as a clean Pass), both tasks, and matched 44.1 kHz inputs on both sides. vevo2's route refuses with "Vevo2 text/prosody route requires text_input or target_text", and session.cpp:897 fills target_text only from request.text_input, which AudioTransform has no field to carry. The same vevo2 weights convert voice correctly through the default route with an exact ASR round trip, so the model and the plumbing are both healthy; it is the s2s route specifically that this RPC cannot express. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * backend(audio-cpp): carry transform text through params, add the s2s entry AudioTransform is audio-in / audio-out and its proto message has no text field, but not every task it routes to is audio-only. vevo2's speech-to-speech route is a text and prosody route: session.cpp:897 fills refs.target_text from request.text_input and nowhere else, and the run refuses without one with "Vevo2 text/prosody route requires text_input or target_text". The params map is the only channel this RPC has that reaches the engine, so the text travels through it and apply_transform_text_input unpacks it after the params have been copied into task.options. Before this, s2s was not awkward to reach through /audio/transform, it was unreachable, and it was the last audio.cpp task kind with a real model and no way to get to it. target_text is canonical and text is its alias, the order vevo2's own option table declares them in, so a request setting both gets the canonical one rather than whichever the map happened to store first. An empty value falls through to the next candidate instead of ending the search. language rides along only when a text was found: on its own it conditions nothing, and manufacturing a text_input for it would route a plain separation request carrying a language hint through the text path. The keys are left in task.options rather than erased, because vevo2's loader advertises target_text as a request option and a family reading it there keeps working. Nine tests, all confirmed failing on behaviour against a stub that returned false before the implementation was written. Verified end to end afterwards: vevo2-q8_0 with task:s2s and params[text] returns a 5.12 s 24 kHz output whose closed-loop citrinet transcription is exact, and htdemucs separation with no text param still returns its four stems, with and without params[stem]. audio-cpp-vevo2-speech-to-speech ships that route. Every audio.cpp task kind with a loadable family now has a gallery entry; spk remains the only gap and has no family upstream at all. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * docs(audio-cpp): document params[text] and the pinned transform tasks The text channel and the two task pins are both invisible from the endpoint contract alone: nothing in the AudioTransform form tells a reader that a speech-to-speech model needs the line it is resynthesising, and nothing says that asking for singing voice conversion without task:svc silently gets plain voice conversion instead. Both are the kind of thing a user only discovers from a refusal or, worse, from output that looks right and is not. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(utils): annotate the two G304 sites this branch introduced gosec flags os.Open on a variable path, and both new call sites in ffmpeg.go are its alerts on this PR. Neither is reachable by an outside caller: isPCM16Wav opens the exact path it is about to hand ffmpeg as input, which in the upload path is a server-created temp file named from path.Base of the client name so no traversal survives, and wavAudioBytes opens AudioResample's own dst, a name this package derives from src and has just had ffmpeg write. Annotated in the repo's existing style rather than restructured, with the reason spelled out, because a bare suppression is worth nothing to the next reader. The three other G304 sites in this file, in passthroughWAV and isTargetWav, predate the branch and are left untouched. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(audio-cpp): build arm64 with gcc-14 for the armv9.2 SME variants The arm64 CPU image failed to build: cc1: error: invalid feature modifier 'sme' in '-march=armv9.2-a+dotprod+fp16+sve+i8mm+sve2+sme' ggml's CPU_ALL_VARIANTS table includes armv9.2 variants compiled with +sme, and Ubuntu Noble's default gcc-13 rejects that feature modifier. Every entry in the table has to compile even though a host only ever dlopens the one its own CPU supports, so a single unbuildable variant fails the whole image. gcc-14 accepts it, which is exactly the fix llama-cpp already carries in .docker/llama-cpp-compile.sh; this is the same problem reached by a different Dockerfile. Applied to every arm64 BUILD_TYPE rather than to the CPU one alone, and that differs from llama-cpp on purpose. llama-cpp needs it only for its pure-CPU image because its GPU builds run llama-cpp-fallback, which builds no variant table. This backend's Makefile turns ENGINE_ENABLE_CPU_ALL_VARIANTS on for every non-Darwin build, GPU included, so an arm64 GPU image would hit the identical error. The matrix has no arm64 GPU entry today, which is precisely why gating on an empty BUILD_TYPE would leave the trap armed for whoever adds the first one. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-30 12:11:56 +02:00
Richard Palethorpe	9058a2bb46	feat: Add 3d generation UI/API and trellis2cpp backend (#10979) ... * feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint Adds the plumbing for image-conditioned 3D asset generation (binary glTF / GLB output), modeled on the video generation path: - backend.proto: Generate3D RPC + Generate3DRequest (staged image src, glb dst, seed/step/cfg_scale/texture_steps, quality and background enums, params map for backend-specific extras) - pkg/grpc: thread Generate3D through client, server, embed, base and the backend interfaces; connection-evicting and distributed-node wrappers (in-flight tracking + file staging) included - core/config: FLAG_3D usecase (guessed only for the trellis2cpp backend), '3d' canonical usecase string mapped to the Generate3D method, and a '3d' output modality - REST: POST /v1/3d/generations (+ unversioned alias) returning OpenAIResponse with a /generated-3d URL or b64_json; conditioning image accepted as URL, base64, or data URI; quality/background validated at the edge; .glb served as model/gltf-binary - auth: '3d' route feature (default ON); /api/instructions entry Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(trellis2cpp): add the trellis2.cpp image-to-3D backend Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2, pbr-textures branch) as a Go+purego backend, following the stablediffusion-ggml pattern: - backend/go/trellis2cpp: purego bindings to the flat C ABI (v9, asserted at startup), eager pipeline load with model-set validation (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured exactly like the upstream demo), Generate3D via t2_generate + t2_bake_glb writing a binary glTF to dst. Weight-free unit tests cover resolution/validation/param mapping — CI never downloads the multi-GB GGUF set or runs inference. - CPU SIMD variants build into per-variant directories (the shared libggml sonames collide across variants, unlike sd-ggml's flat renamed-.so scheme); run.sh picks one via /proc/cpuinfo. - CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta + latest/master image entries, bump_deps tracking of the pbr-textures branch, changed-backends.js mapping, top-level Makefile targets. - Importer: auto-detects trellis GGUF repos/URIs (registered before llama-cpp so the .gguf match isn't stolen) and expands any trellis URI to the full 10-file component set spanning the three LocalAI-io HF repos. - Gallery: trellis2-4b (full PBR + 1024 cascade) and trellis2-4b-geometry (512 untextured) with verified sha256s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(ui): 3D generation page with native GLB viewer and IndexedDB history Adds a Studio tab + /app/3d page for the new image-to-3D endpoint: - GlbViewer ports the trellis2cpp demo's dependency-free WebGL2 renderer (quaternion trackball, metallic-roughness PBR, ACES, hidden-line wireframe with a bounded index budget) and pairs it with a minimal GLB parser for the two forms t2_bake_glb emits — dense vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as normalized integers) and the opt-in UV-atlas textured form. Parsing happens before any GL so stats and errors render without WebGL2. - use3DHistory stores past generations (params, input thumbnail, and the GLB blob itself) in IndexedDB with keep-newest-20 eviction — GLBs are multi-MB binaries localStorage can't hold — and the page offers a download button for the active GLB. - Wiring: CAP_3D capability constant (FLAG_3D — the exact string /api/models/capabilities serves), threeDApi, router entries, Studio tab, vite dev proxy, en locale keys. - e2e: render-smoke entry plus a focused spec that feeds a real one-triangle vertex-PBR GLB through the parser/viewer and exercises IndexedDB persistence, selection, deletion, and API errors. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(3d): address API correctness and UX issues Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it. Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add previewable print remeshing Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download. Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(trellis2cpp): centralize remesh dependency pins Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement Generate3D stub for new proto RPC The Generate3D RPC added to backend.proto for the trellis2cpp backend made tonic's generated Backend trait require generate3_d, breaking the kokoros-grpc build. Return unimplemented like the other unsupported modalities. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>	2026-07-29 16:15:04 +02:00
Tai An	a7fa678d83	fix(tts): forward the OpenAI speed field to the backend (#11097) (#11120) ... * fix(tts): forward the OpenAI speed field to the backend (#11097) /v1/audio/speech accepted the documented OpenAI `speed` field and then dropped it: schema.TTSRequest had no Speed member, so the value never reached proto.TTSRequest and the request returned 200 with an unchanged playback rate. Accept speed and normalise it into the existing per-request params map, which core/backend forwards verbatim to the backend. An explicit params["speed"] still wins, and a value outside the documented 0.25-4.0 range is now rejected with 400 instead of being silently ignored. Signed-off-by: Anai-Guo <antai12232931@outlook.com> * fix(tts): distinguish explicit speed=0 from an omitted field Make TTSRequest.Speed a *float32 so an explicit `"speed": 0` (invalid, below the documented 0.25 minimum) is rejected with 400 instead of being treated as unset and silently defaulted. An omitted field stays nil and leaves the backend default untouched. Add a request-boundary regression that distinguishes an omitted speed from an explicit zero, addressing review feedback. Signed-off-by: Anai-Guo <antai12232931@outlook.com> * docs: drop the speed field from the TTS docs Per review: no backend consumes params.speed today, so documenting it would be misleading. The API-level plumbing and validation stay. Signed-off-by: Anai-Guo <antai12232931@outlook.com> --------- Signed-off-by: Anai-Guo <antai12232931@outlook.com>	2026-07-27 19:03:32 +02:00
mudler's LocalAI [bot]	83a0f16a21	feat(gallery): let one gallery entry offer several builds of the same model (#10943) ... * feat(system): expose raw detected capability for model meta resolution Model meta gallery entries express hardware fallback through candidate ordering rather than a capability map, so they need the undecorated detected capability string without Capability's default/cpu fallback chain. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(system): drop duplicate capability accessor, cover DetectedCapability ReportedCapability was added with a body identical to the existing DetectedCapability. Keep one accessor and move the specs onto it, since DetectedCapability had no direct coverage of its no-fallback behavior. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): parse IEC binary size suffixes (KiB..PiB) ParseSizeString accepted only SI suffixes, so a "20GiB" floor was rejected outright. Model and VRAM sizes are conventionally quoted in IEC units, and silently reading GiB as GB would understate a floor by about 7%. Purely additive: these inputs previously returned an unknown-suffix error. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add Candidate type for meta model entries Candidate is one option in a meta entry's ordered variant list. It names a concrete gallery entry and declares when that entry suits the host. EffectiveMinVRAM resolves the VRAM floor, letting an authored min_vram win over a nightly-inferred one. An unparseable floor errors instead of being treated as absent: swallowing a typo would turn a constrained candidate into an unconstrained one and select a too-large variant rather than fail loudly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add hardware-aware model variant resolver Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): allow gallery model entries to declare variant candidates A gallery entry with a non-empty candidates list is a meta entry: it names an ordered list of concrete entries and resolves to the first one the host can satisfy, instead of describing model files directly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): resolve meta model entries to hardware-appropriate variants at install Meta gallery entries carry an ordered candidate list; at install time the first candidate the host satisfies is resolved and its payload installed under the meta's name, so the model keeps a stable name regardless of which variant backs it. The resolution is recorded in the installed gallery config so a reinstall honors a prior pin and operators can see the backing variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): key meta pin recall on the installed name and detach resolved entries Six review findings on the meta-entry install path. Pin recall was keyed on the gallery entry name while applyModel writes the record under the install name (req.Name when supplied), so a meta installed under a custom name with a pin lost that pin on reinstall and was silently re-resolved onto a different variant, possibly swapping its backend. Compute the install name with applyModel's own precedence before the recall. ResolveMetaModel returned a shallow struct copy, so the resolved entry's Overrides aliased the gallery entry's map and the install path's in-place mergo merge wrote the caller's request into the shared catalog. Detach Overrides, ConfigFile, AdditionalFiles, URLs and Tags. Not exploitable today only because this path re-unmarshals the gallery per call, which is a property nobody should have to rely on. Also: overlay the meta's name onto the persisted config for meta installs so the gallery file no longer records the variant's name; move the pinned-VRAM warning below the variant validation so a pin naming a nonexistent entry does not warn about VRAM before failing for an unrelated reason; and stop seeding config.URLs in the config_file branch, which duplicated every declared URL. Add seven network-free specs driving InstallModelFromGallery with a meta entry: variant payload wins over the meta's legacy url fallback, the resolution record round-trips to disk, a pin is recorded and honored on reinstall including under a custom install name, and the resolved entry does not alias the gallery's maps. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): deep-copy meta overrides and make two specs functional ResolveMetaModel detached the resolved entry's Overrides and ConfigFile with maps.Clone, which only copies the top level. Gallery overrides are nested in practice (parameters.model is near-universal) and the install path merges the caller's request with mergo.WithOverride, which recurses into nested maps and overwrites them in place, so the gallery entry's own inner maps were still reachable and still got rewritten by the last caller to install. Copy both maps all the way down instead, recursing through the container shapes a YAML decoder produces. ConfigFile is not mutated on the install path today, but it carries the same kind of nested payload and leaving it shallowly cloned would invite the bug back. Also fix two specs that passed whether or not their target fix was present: - "does not write the caller's overrides back into the gallery entry" re-read the catalog from disk, which re-unmarshals fresh structs and so cannot observe in-memory aliasing. It now asserts against the in-memory gallery entry and drives the real mergo merge. - "round-trips the resolution record to disk under the meta's name" asserted a name that is already correct in the config_file branch. It now drives the url branch via a file:// fixture, where the meta-name overlay actually applies. Both were verified red by reverting their fix. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(gallery): lint meta model entry invariants in index.yaml Adds Ginkgo specs that parse the shipped gallery/index.yaml and enforce the invariants that keep meta entries safe: a legacy url fallback equal to the final candidate's url, references only to existing non-meta entries, a min_vram floor on every candidate but the last-resort one, a capability drawn only from the vocabulary the system can report, and descending VRAM floors within a capability group. The capability check is the only compensating control for a typo there. Candidate matching is a case-sensitive exact comparison against SystemState.DetectedCapability(), so an unknown value never matches and falls through silently instead of erroring. The vocabulary therefore mirrors the raw return set of getSystemCapabilities(), which notably excludes "cpu": that is a fallback key inside Capability(capMap) on the meta backend path, never a reported capability. A CPU-only host reports "default". These pass vacuously until the pilot meta entry lands; the guard is intentionally in place before the thing it guards. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(gallery): close coverage gaps in the meta entry lint The ordering invariant grouped candidates by capability and asserted floors descend within a group. A candidate with an EMPTY capability matches every host, so it does not belong in its own group: it dominates every later candidate whose floor is at or above its own, across capability groups. Track a running minimum floor over the unconditional candidates instead, which subsumes the old same-group check for the empty capability. Every spec skipped non-meta entries, so with zero meta entries in the index all five bodies were no-ops. Aligning GalleryModel.IsMeta() with GalleryBackend.IsMeta(), whose semantics are deliberately opposite, would have made all of them pass while checking nothing. Extract each invariant into a helper over a slice of entries returning the violations it finds, and cover those helpers with synthetic fixtures so the logic stays tested at zero meta entries. The index-driven specs are now a thin application of already proven logic. Also assert the index parses non-empty, report every violation in one run rather than aborting on the first, and parse the index once for the suite. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(gallery): add nightly denormalization of meta model candidates Fills the read-only backend, quantization and inferred_min_vram fields on meta gallery candidates and opens a PR, modeled on the existing checksum_checker job. Computing these needs network access, so it happens nightly rather than at install time. An authored min_vram is never modified: a human who measured a real load knows more than a pre-download estimate does. The index is rewritten via yaml.Node rather than a document round-trip. A full round-trip reflows all ~26k lines of gallery/index.yaml, which would bury the computed values and make the nightly PR unreviewable. The rewrite touches only the three derived keys, so authored styling survives and a run that computes nothing leaves the file untouched. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): keep the gallery denormalize diff reviewable and self-healing The nightly denormalization job edits YAML nodes instead of round-tripping structs so its PR stays small enough for a human to review, but the write path undid that: yaml.Marshal re-encoded the node tree at yaml.v3's default 4-space indent and dropped the leading document marker, reflowing roughly 6000 lines around the handful of real changes. Encode through yaml.NewEncoder at the index's authored 2-space indent and restore the header. A write that changes three fields now changes three lines. Stale inferred_min_vram values were also never cleared. Both skip paths (an authored min_vram is present, or the candidate is the last resort) returned before touching the field, so a candidate that gained a floor or became the last resort after a reorder kept an inferred value that EffectiveMinVRAM reported as a real constraint, failing the meta lint with no way for the job to self-heal. Clear the field before both skips. The workflow discarded a whole night's work on any single failure: the program exits 1 when a candidate cannot be estimated, which aborted the job before the PR step, so one unreachable candidate blocked every other refresh indefinitely. Capture the status, open the PR with what was computed, mark the PR body as partial, and fail the run afterwards so the problem still surfaces. Also preserve the index's existing file mode instead of forcing 0644, and drop the redundant //go:build ignore tag, since Go already skips dot directories and the sibling modelslist.go carries no tag. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add nanbeige4.1-3b meta entry with hardware-resolved variants Adds the first real meta entry to the gallery index. It resolves to the Q8_0 build on hosts with at least 6GiB of VRAM and to the Q4_K_M build everywhere else, installing either payload under the stable name nanbeige4.1-3b. The entry carries a url equal to its final candidate's url. LocalAI releases that predate candidates support parse the index non-strictly and drop the key silently, so without that url they would list the entry and install nothing. A regression spec parses the index the way those releases do and asserts every meta entry stays installable for them. Also teaches core/schema/gallery-model.schema.json about candidates. The schema sets additionalProperties: false at the top level, so an author following CONTRIBUTING.md and adding the yaml-language-server comment would otherwise get a validation error on this entry. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): make candidate entries complete, installable entries Reworks hardware-resolved gallery variants after a design pivot. There is no longer a separate "meta" entry kind. A gallery entry is a normal, complete entry that may additionally carry candidates:, a list of hardware-gated upgrades over itself, and the entry is itself the last-resort candidate. The previous design relied on a bare url: as the fallback for LocalAI releases that predate candidates support. That fallback is empty in practice: none of the 80 gallery/*.yaml files carry a top-level files:, and 1216 of 1281 index entries carry their payload in the index entry itself, so a url alone yields a config template with nothing to download. Since every released LocalAI reads gallery/index.yaml live from master, merging a payload-less entry would have shown every existing user a model that installs to a broken state. Making the entry its own base candidate removes the problem at the root: old clients drop the candidates key and install the entry exactly as they do today. Resolution order is now explicit pin, then capability plus VRAM over the declared upgrades, then the entry itself. The entry ALWAYS installs: when its own min_vram or capability is unmet the installer warns and installs it anyway, because there is nothing below it and refusing would make the gallery behave worse the newer the client is. A pin naming the entry's own name is valid and is how an operator declines an upgrade. IsMeta() becomes HasCandidates(), ResolveMetaModel becomes ResolveVariant, and the persisted meta_name record key becomes entry_name. GalleryBackend.IsMeta() is a separate concept and is untouched. The lint drops the three rules the pivot makes wrong (url equality with the final candidate, no inline payload, unconstrained final candidate) and gains one: the entry's own floor must sit strictly below every candidate's, since a base that outranks a candidate makes that candidate unreachable. The pilot entry is now the existing nanbeige4.1-3b-q4, which gains a 2GiB floor of its own and a single 6GiB upgrade to nanbeige4.1-3b-q8, replacing the separate nanbeige4.1-3b entry added in d0d441bb4. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): select model variants by hardware fit, not authored order Gallery entries could already carry a list of alternatives, but selection was an authored, ordered, first-match policy: every candidate declared a `capability` string and the VRAM floors had to descend in a hand-tuned order. That pushed hardware knowledge onto whoever edits the gallery and made ordering load-bearing, so a reordered list silently changed what users installed. None of it was necessary. SystemState.IsBackendCompatible already derives hardware support from a backend name alone: it knows MLX and metal are Darwin-only, CUDA is NVIDIA-only, ROCm AMD-only, SYCL Intel-only. Selection can read that instead of asking authors to restate it. Authoring is now just a list of names: - name: qwen3.6-27b min_memory: 4GiB variants: - model: qwen3.6-27b-mlx-8bit - model: qwen3.6-27b-gguf-q8 min_memory: 28GiB and all the intelligence moved into the selector. Given a host it drops the variants whose backend cannot run here, drops those whose known memory requirement exceeds what the host has, and takes the LARGEST of what is left, because a bigger footprint is a higher quality quantization of the same model. A variant of unknown size is kept, since nothing proves it does not fit, but it ranks last so a proven fit always beats a guess. An explicit pin still wins outright, and if nothing survives the entry installs its own payload: the base always installs, this never refuses. Available memory is VRAM when a GPU was detected and system RAM otherwise, read through xsysinfo so a cgroup limit is honored and a container gets its own limit rather than the node's RAM. Capability disappears entirely, from the types, the schema and the lint. VRAM and RAM collapse into one `min_memory`, because a model's footprint is roughly the same wherever it lives and one figure is compared against whichever applies. The lint rules about ordering, the capability vocabulary and floor relationships are deleted with the hazards they described; what remains is that every variant names an entry that exists and does not itself declare variants, plus that any memory figure actually parses. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: size model variants with a live probe, drop the nightly denormalizer Selection needs each variant's size to decide whether it fits and to rank largest-first. That figure was written into the index by a nightly job, which made the gallery carry a derived value that could drift from the entry it was derived from. Derive it at install time instead. pkg/vram already sizes a model without downloading it, and the gallery UI already uses it: a remote GGUF header range-fetch, then an HTTP HEAD for the content length, then any declared size:. It caches its results, so reuse it rather than writing a second probing path. A probe failure must never fail an install, so an unprobeable variant is treated as unknown: it survives the memory filter, because nothing proves it does not fit, and it ranks last, so a known-good fit always beats a guess. If every probe fails, selection still terminates on the base entry. The probe is injected through ResolveEnv rather than called directly, for the same reason the backend compatibility check is: specs pin an exact size, or an exact failure, without reaching the network. With that in place three things are dead weight and go: - The nightly job and the fields it populated. Variant.Backend was redundant because the backend is resolved live from the referenced entry during selection, and Quantization was display-only that nothing read. - min_memory on the base entry. The base always installs and its floor could only warn, so it could not change any outcome. - The lint rules and schema entries for both. min_memory on individual variants stays, as the override for when the probed size is wrong. An authored figure now suppresses the probe entirely rather than merely outranking it, so it costs no round trip. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): expose model variants for selection over API, CLI and MCP A gallery entry may carry `variants:`, alternative builds of the same model. Selection already worked at install time, but nothing could see what an entry offered or ask for a specific build, so the feature was undrivable. Listing: `GET /api/models` now reports `variants` and `auto_variant` for the entries that declare variants. Each variant carries its resolved backend, its measured size and whether it fits this host. `auto_variant` is what installing without a choice would pick right now. The new gallery.DescribeVariants runs the same variantOptions + SelectVariant pass the installer runs, so the reported default cannot drift from what installing actually does, and HostResolveEnv is extracted so both derive the host and share pkg/vram's probe cache from one place. Performance: an entry that declares no variants returns early without touching the probe, so the ~1280 ordinary entries cost exactly what they cost before. Selection: `variant` is accepted on POST /models/apply, as a query param on POST /api/models/install/:id, on the gallery apply file/string request, as `local-ai models install --variant`, and as a parameter on the install_model MCP tool (both the httpapi and inproc clients). Empty means auto-select. An unknown variant name now fails the install naming what was requested. This closes a real hole: an entry declaring no variants short-circuits before selection runs, so a requested variant was previously dropped silently and the install reported success. startup.InstallModels ends in a variadic model list, so install options could not be appended to it; InstallModelsWithOptions is added alongside and InstallModels delegates to it. No caller signature changed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): drop the redundant variant min_memory field Variant.MinMemory was an authored override for when the live probe misreads a variant's footprint. It duplicated an existing field: probeEntryMemory already passes the entry's declared size: into EstimateModelMultiContext, whose cascade prefers that declared size over its own guesswork. Correcting size: on the referenced entry fixes the figure for every consumer rather than only for variant selection, so min_memory shadowed the right answer. A variant is now nothing but a name. Its effective size is exactly the probe result, and an unknown stays unknown: it survives the filter and ranks last. EffectiveMemory loses its error return along with the field. The authored string was the only thing that could fail to parse, so the error had no remaining source and was propagating dead nil-checks through SelectVariant, DescribeVariants and the pin warning. Selection behaviour is unchanged. The specs covering probe-derived sizing, ranking, filtering, the unknown-size path, pin recall, entry/variant metadata split and deep-copy isolation all survive; the three install specs that needed a definite size now declare it through the referenced entry's own size:, which exercises the documented escape hatch directly. gallery/index.yaml is untouched: no entry ever carried the key. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): rank the entry's own build against its variants Variant selection pulled the declaring entry's own payload, the base, out of the candidate set and consulted it only once every declared variant had been rejected. Two real failures followed. A variant whose size the probe cannot determine deliberately survives the memory filter, because nothing proves it does not fit. As the only survivor it then won outright on any host, however small: a 2GiB machine installed an unmeasured variant in preference to the 4GiB build the entry itself ships, with no warning. 241 of the 1280 current index entries carry no files and no size, which is exactly that shape. "Largest wins" also broke whenever the base was the largest. An author writing a Q8 entry that offers a Q4 downgrade for small hosts, a natural shape that nothing in the lint, schema or docs discourages, had the Q4 installed on every large host instead. Make the base an ordinary participant. It is still exempt from both filters, so selection always terminates on something installable, but it is now ranked against the variants: a proven fit first and largest, then the base, then any variant whose size nothing could measure. Both failures disappear together. The base is probed for its size accordingly, which it was not before, because an unsized base would lose every contest to an unmeasurable variant. FellBackToBase is kept but narrowed to "no declared variant survived", rather than "the base was chosen", since the base now also wins on merit and that is not worth warning about. A recalled variant pin also became a permanent install failure. A pin the caller supplies on this request must stay fatal, but one recalled from ._gallery_<name>.yaml can be invalidated by any later gallery edit, and failing on it turned one rename into a model that could never be reinstalled or upgraded again short of deleting a dotfile the user has never heard of. A stale recalled pin is now dropped with a warning naming it, and selection runs as if it had never been recorded. Also drop the last textual reference to two abandoned designs from the DetectedCapability comment, correct the documented variants JSON example, which showed a memory_bytes of 0 that omitempty makes impossible, and remove an em dash from the install skill. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): budget variant memory from RAM when a GPU reports no VRAM Variant selection read its memory budget from VRAM whenever a GPU capability was detected, and from system RAM only when none was. Apple Silicon satisfies the first branch and fails the premise: arm64 macs report the metal capability unconditionally, without probing anything, while TotalAvailableVRAM has no discrete VRAM pool to find and returns zero. The budget therefore came out as zero on every Mac. Zero drops every variant carrying a known size, so the base build was installed on all of them however much memory the machine had. The feature was inert on the platform, and silently: falling back to the base is a legitimate outcome, so nothing looked wrong. Take VRAM only when it is actually a number, and fall back to RAM otherwise. On a unified-memory host RAM is not an approximation of the budget, it is the budget, since the GPU shares it. A discrete GPU whose VRAM could not be read also lands on RAM, which overstates what the card holds but understates nothing the host has; the previous zero understated both. An unreadable RAM figure still yields zero and still installs the base, so a genuinely unknown host is not talked into a larger download. This is what turned tests-apple red: "installs a fitting variant's payload under the entry's own name" asserts on selection, and the runner resolved to the base because its budget was zero. The added specs pin the branch directly rather than relying on a macOS runner to notice again. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add a model variant picker to the models gallery PR #10943 shipped the server side: a gallery entry may declare `variants:`, `GET /api/models` attaches `variants` and `auto_variant` to declaring entries, and `POST /api/models/install/:id` accepts a `variant` query parameter. Nothing in the UI consumed any of it, so the feature was not reachable from the browser. This wires it up. modelsApi.install takes an optional second argument and appends an encoded `?variant=` only when one is given, so every existing call site keeps sending exactly the request it sent before. On the models table, an entry that declares variants gets a split button. The primary Install still installs the auto-selected build, because auto is the default and the point of the feature; the chevron opens a menu for a deliberate override. It follows the Backends.jsx precedent: one shared Popover re-anchored per row, rendering .action-menu items, which brings Escape, outside-click and focus return along with it. An entry that declares no variants renders exactly as it did before. A variant that does not fit is dimmed but stays selectable, since the server honors an explicit choice with a warning rather than refusing it. memory_bytes is omitempty on the wire, so an absent key means the size is unknown and never zero. A single helper guards both the menu and the detail row, because formatBytes would otherwise render a falsy value as "0 B", which reads as "needs nothing". The expanded detail row gains a Variants section listing each build's backend, size, whether it fits, which is the entry's own build, and which one auto-selection would pick, built from the existing DetailRow helper and .badge classes. Eight Playwright specs cover the picker, including that plain Install sends no variant parameter and that choosing one sends it. One pre-existing assertion was scoped with .first(): the Variants section legitimately adds more llama-cpp badges to the detail row, which tripped strict mode. UI line coverage 49.42% -> 49.36% against a 40.0 baseline and 0.8pp tolerance; branch coverage rose 72.04% -> 72.66%. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): describe model variants from a companion endpoint Variant description probes each referenced entry's weight files over the network: an HTTP HEAD plus a ranged GET, serial, five seconds per probe with no aggregate deadline. Running it inline in GET /api/models made one listing cost (entries x variants) round trips. The Manage page fetches with items=9999, so at 200 declaring entries that is ~1000 serial probes, minutes of a blocked handler and gigabytes of range traffic for a single page load. Only one entry declares variants today, but the feature exists so that many will. Follow the precedent already set for VRAM estimates. The listing now reports only has_variants, a length check on loaded metadata that touches nothing, and GET /api/models/variants/:id returns the description for one entry, mirroring estimate/:id in route shape, auth and error handling. DescribeVariants itself is unchanged; only its caller moved. The picker fetches lazily at the two points where a user asks to see variants, opening the split-button menu and expanding the detail row, and caches per entry for the page session. An entry declaring no variants issues no request at all. A spec counts real HTTP hits on the weight files, so it goes red if description becomes reachable from the listing path again through any caller. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): filter the model gallery to entries that declare variants The gallery is heading towards showing parent entries and hiding the individual builds they reference, so a user sees one row per model rather than six quantizations of it. Adoption is a single entry today, so defaulting to that would leave a one-row gallery. This ships the migration-phase inverse instead: the default is untouched, and a toggle narrows the list to only the entries that declare variants. It previews the end state and changes nothing until someone asks for it. The filter is server-side, next to term/tag/backend/capability and above the pagination arithmetic. The listing paginates at 9 items, so narrowing on the client would leave totalPages and availableModels describing the unfiltered set and hand the user empty pages. It selects on HasVariants(), which reads already-loaded metadata, so it issues no variant probes. The parameter is named has_variants after the listing field it selects on, and is compared against "true" like the other boolean query params (all_users, save_checkpoint), so has_variants=false reads as absent. With it omitted the response is byte-for-byte what it was before. The control is the shared Toggle component, matching the fitsFilter toggle already on this page: same wrapper class, same icon and label shape, same localStorage persistence. Unlike fitsFilter it resets to page 1 on change, which a server-side filter has to do. Stacking the toggle with a tag or backend filter easily yields nothing while one entry declares variants, so the empty state now names the variants filter as the cause rather than leaving a user to conclude the gallery is broken. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): render gallery model descriptions as Markdown Gallery descriptions are Markdown, but the React UI dumped them raw, so a model whose description opens with an ATX heading showed a literal "# Qwen3.6-27B [](https://chat.qwen.ai)" in the list. Full-description areas now render through renderMarkdown (marked + DOMPurify), matching how Backends.jsx and the Manage detail panels already handle the same content: - Models.jsx expanded detail row - VoiceLibrary.jsx voice detail header The truncated one-line previews must not render block Markdown: a leading "#" would become an <h1> and wreck the row height and rhythm. They get a new stripMarkdown() helper instead, which reduces Markdown to a single line of readable plain text. It is used for the cell text and for the title tooltip, since a tooltip full of "[](url)" is no better than a cell full of it: - Models.jsx gallery table description cell - Manage.jsx model and backend resource-row descriptions stripMarkdown walks marked's lexer output rather than running regexes over the source, so what it strips is by construction what renderMarkdown would have rendered, and it needs no new dependency. Output lands in JSX text nodes, so React escapes it; no new dangerouslySetInnerHTML beyond the two full-description sites, both of which run DOMPurify. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): strip Markdown from the backends table description cell Commit b35d630cf fixed this for gallery models but left the Backends admin page with the same asymmetry: its detail panel renders the description through renderMarkdown, while the collapsed table row dumped the raw gallery string into both the cell body and the title tooltip. That is user-visible. 40 of the 949 entries in backend/index.yaml carry Markdown - insightface uses inline code backticks, others use lists and links - and backend descriptions also contain embedded newlines, so the one-line cell showed literal syntax. The cell now runs stripMarkdown over the description once and uses the result for the text and the title, matching Models.jsx and the ResourceRowDesc component in Manage.jsx. The '-' placeholder is preserved, and now also fires when a description reduces to nothing after stripping. The detail panel is untouched and no new dangerouslySetInnerHTML is introduced: stripMarkdown output lands in a JSX text node, so React escapes it. Three Playwright specs cover it: a description with a heading, inline code and a link renders as clean text with no literal syntax and no block element in the cell, the title tooltip carries the same stripped text, and a backend without a description still shows the placeholder. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ui(models): polish the variant detail view and scope rendered Markdown The gallery detail pane rendered every field through the same two-column label/value row, including the description. Multi-paragraph prose in a value cell ran eight rows tall at the top of the pane on a ~1200px measure, breaking the grid's rhythm exactly where the eye enters. Move it into its own full-width block above the table, capped at a 68ch measure, keeping the label. Rendered Markdown had no scoped typography anywhere in the app, so a description opening with `#` inherited the browser default 2em inside a 13px surface while a `##` further down was indistinguishable from body text. Add a reusable .markdown-body block mapping h1-h6, paragraphs, lists, links, code, blockquotes, images and tables onto the existing type scale, and apply it to every renderMarkdown() consumer: the models detail, the backends detail, both Manage details and the voice library detail. Rebalance the variants list so the name leads. Backend and size drop from badge/secondary weight to muted metadata; the FITS badge goes entirely, since it was true of nearly every row and so said nothing, while the variant that does not fit keeps a warning badge and a dimmed name. AUTO-SELECTED stays marked because it answers what a plain Install produces. Rows share the parent's grid tracks via subgrid so name, backend, size and status line up down the list instead of raggedly following name length. Finally, make each variant row actionable. It looked like a list of choices but was inert text, with per-variant install hidden behind the split-button chevron elsewhere; each row is now a button onto the existing handleInstall(modelId, variant) path, with hover, keyboard focus and a disabled state while an install is in flight. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): collapse the listing to one row per model The listing supported has_variants=true, which narrowed to entries that DECLARE variants. With adoption at three entries that showed three rows, which is useless; it was always a placeholder. Replace it with the view that is actually useful: the deduplicated gallery. Show every entry installable in its own right and nothing twice, which means the parents plus every entry nobody references, and hide only the builds another entry already offers as a variant, since those are reachable through their parent. The parameter is renamed to collapse_variants accordingly: the filter is no longer a predicate on a row's own metadata but a view over the whole gallery. Default stays off, so the response with the parameter absent is unchanged. VariantReferencedIDs never reports an entry that declares variants of its own, so parents are always visible. That guarantees every hidden entry has a visible entry offering it, and no chain can strand a row. Variant resolution already refuses to install such a reference, but the listing has to stay coherent in the presence of a gallery that has one rather than silently swallowing entries. Self-references and dangling references hide nothing. The referenced set is computed over the whole gallery rather than over what the other filters left, so an entry is hidden because a parent offers it and never because of what the user searched for. The pass is over metadata already in memory: it resolves nothing over the network and triggers no variant description or size probe, so the listing's zero-probe contract still holds. The UI toggle keeps its behaviour (persistence, page reset, clear filters) and becomes "One row per model", which says what the user gets. Its localStorage key moves too, since the stored value meant a different filter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show the collapsed model listing by default The gallery listing is what a user reaches for to answer "what can I install". Answering that with several rows for the same model, one per build, makes the reader do the deduplication the collapsed view already does, so the collapsed view is the one to land on. The UI now asks for collapse_variants=true unless the toggle says otherwise. The server default is deliberately untouched: a request with the parameter absent still returns the full listing, because other API clients depend on that response and collapsing it under them would be a breaking change. Opting out omits the parameter rather than sending false, so it asks for exactly the listing everyone else gets. The stored preference changes vocabulary from '1'/'0' to 'on'/'off'. The previous build wrote it from an effect that runs on mount, so a stored '0' recorded that the page had been opened rather than that anyone chose the expanded view, and honouring it would pin every earlier visitor to a default they never picked. Only the new vocabulary counts as a choice; a legacy '1' meant the collapsed view and is what the new default gives anyway, so no earlier deliberate choice is lost. Collapsing being the default also changes what the empty state may say about it. An opted-into filter can be named as the cause of an empty result; a default cannot, so the filters keep the top line and the collapsed view drops to a hint below it, shown only once filters are narrowing the set. For the same reason "Clear filters" now restores the collapsed default instead of switching it off, and the toggle alone no longer counts as a filter worth offering to clear. The label stays "One row per model": it describes the view the user is looking at rather than an action, so it reads the same whether it is opted into or out of. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: group alternative builds of the same weights under variants Sweep the gallery for entries that are alternative builds of the same weights (different quantization, precision, or runtime format) and declare them as variants of a single parent row, so the listing offers one row per model instead of one row per quantization and the installer picks the largest build that this host can actually run. 41 families over 95 entries, turning 54 entries into variants. The parent is the bare-named entry wherever one exists, so nothing changes about what any existing entry installs. Ranking already selects the largest fitting build regardless of which entry is nominally the parent, so the parent only decides the pathological case where nothing fits. For the ten families that have no bare-named entry, the smallest build is the parent, since that is the one that has to install when nothing fits. Grouping was verified against the actual model filenames rather than the entry names alone. Different parameter sizes, languages, finetunes, and products that merely share a name prefix are left as separate rows: the qwen3.6 APEX and pi-tune finetunes, the DFlash and MTP speculative-decoding pairings, English-only versus multilingual Whisper, the QAT versus non-QAT Gemma 4 weights, and the abliterated FLUX build are all distinct models. Six parents define YAML anchors that other entries pull in with a merge key, which would have handed their variants to every merging child. For the two depth-anything anchors that would have made fourteen unrelated entries advertise the base model's builds as their own. All 26 merging children therefore carry an explicit empty variants list, which overrides the merged key and is equivalent to the key being absent. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): rank model variants by host backend preference Variant auto-selection filtered candidates by whether their backend can run on the host, then ranked the survivors by size alone. The backend never influenced the choice beyond that gate, so a Mac offered both an MLX build and a llama.cpp build kept neither filtered and installed whichever was larger, leaving the native accelerated runtime unused. The same held for CUDA against CPU on NVIDIA and ROCm against Vulkan on AMD. Rank by the host's backend preference between the fit tier and size: fit stays a filter, preference decides among the builds the host can equally hold, and size still separates builds on equally preferred runtimes. The preference data stays in one declarative table in pkg/system, now read by a prefix lookup instead of a switch, so adding a capability or reordering one host's runtimes is a one-line edit and the gallery's ranking code carries no per-backend branching. MLX joins the metal rule ahead of metal itself, which is inert for the existing alias-resolution consumer because no alias group holds a candidate named for mlx. An unrecognised backend, an unrecognised capability and an absent preference list all collapse to the previous size-only ordering rather than erroring or dropping candidates. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): rank variants by engine name, not backend build tag Variant auto-selection ranked candidates with SystemState.BackendPreferenceTokens, but that function and the variant ranker speak different vocabularies. BackendPreferenceTokens returns BUILD TAGS ("cuda", "rocm", "sycl", "vulkan", "metal", "cpu"). It exists to match installed backend build directory names like "llama-cpp-cuda-12" during alias resolution in ListSystemBackends. Variant ranking instead matches a gallery entry's `backend:` value, which is an ENGINE NAME: "llama-cpp", "vllm", "vllm-omni", "sglang", "mlx" and the rest. No engine name in gallery/index.yaml contains "cuda", "rocm", "sycl" or "vulkan". preferenceRank matches by substring, so on an NVIDIA host the tokens [cuda, vulkan, cpu] matched neither "llama-cpp" nor "vllm", every candidate scored identically and size alone decided. The NVIDIA, AMD, Intel, darwin-x86 and vulkan rules were all inert. Only metal appeared to work, and only because the token "mlx" happens to equal an engine name. The mismatch does not error, it silently deletes the feature. Separate the two vocabularies. backendBuildTagPreferenceRules keeps the build tags and its original output for every capability, including metal, whose "mlx" token is removed again; its alias-resolution consumer is byte-identical to before. engineNamePreferenceRules is new, holds engine names, and is read by the new EnginePreferenceTokens, which HostResolveEnv wires into the renamed ResolveEnv.EnginePreference. Both tables sit adjacent under one block comment naming each vocabulary and each consumer, and share one lookup helper so their semantics cannot drift. On NVIDIA the order is vLLM, then SGLang, then llama-cpp: vLLM is the throughput engine and a model published with a vLLM build is published that way because that build is the one worth running. AMD and Intel get the same order, since rocm and intel builds of both serving engines ship. Metal prefers mlx over llama-cpp. Vulkan prefers llama-cpp, the only LLM engine with a Vulkan build. darwin-x86 and unknown capabilities are deliberately absent rather than guessed at, degrading to the size-only ordering that predates preference. preferenceRank stays generic and names no engine and no capability, so adding a runtime remains a one-line table edit. Specs pin the NVIDIA and metal rules through the live table and the real HostResolveEnv wiring, so emptying the engine table or wiring the build tag source back in both go red. A regression table asserts BackendPreferenceTokens' original output per capability, and mirrored locks assert neither table carries the other's vocabulary. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: record that variant selection ranks by engine before size A gallery entry can now declare variants, and selection ranks the builds a host can run by engine preference before size. Nothing told a contributor adding a backend that engineNamePreferenceRules exists, so a new engine would silently rank below every known one and lose to whatever build happened to be larger on hosts where it should have won. Document the step where a backend is added, warn against the sibling backendBuildTagPreferenceRules table (build tags, not engine names: the wrong table matches nothing, scores every candidate equally and disables the preference without erroring), and index it from AGENTS.md. Fix the authoring and user docs, which still claimed the largest surviving build wins. An author grouping builds under one entry has to be able to predict what a user gets, and size alone no longer decides it. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(cli,mcp): describe variant auto-selection as preference before size The CLI flag help and the install_model tool schema both still said auto-selection takes the largest build that runs. Ranking now puts engine preference ahead of size, so on NVIDIA a vLLM build wins over a larger llama.cpp one. An assistant reading the old schema would tell users the wrong thing. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): prefer llama.cpp over GPU serving engines on hosts with no GPU engineNamePreferenceRules had no row for the "default" capability, which getSystemCapabilities() returns both when no GPU is detected and when a GPU is present but under the 4 GiB VRAM floor. A missing row yields an empty preference list, which preferenceRank reads as "score everything equally", collapsing variant selection to size alone. That would be harmless if the hardware filter dropped GPU serving engines on such a host, but it does not. IsBackendCompatible derives support from the engine NAME, and "vllm" and "sglang" contain none of the darwin, cuda, rocm or sycl tokens it keys on, so they fall through to its closing "return true". A vLLM variant therefore survives on a CPU-only box and wins whenever its build is the larger of the two on offer: the machine installs vLLM in preference to llama.cpp. darwin-x86 had the identical hole. It was documented as a deliberate omission because nothing accelerates on an Intel Mac, which is true about acceleration and wrong about consequence: with every engine tied, download size decides. Add rows for both putting llama-cpp first. The GPU engines are enumerated behind it rather than left unmatched: an unmatched engine already ranks below every listed one, so llama.cpp would win either way, but unmatched engines also tie with each other and let size decide among them. Naming them fixes that order. MLX is left off the darwin-x86 row on purpose so it ranks last, since IsBackendCompatible admits darwin-tokened engines on that capability even though MLX needs Apple silicon. Preference orders survivors and never filters, so a model published only as a vLLM build is still installed on a host with no GPU; there is a spec for it. Surveyed every other value getSystemCapabilities() can return. nvidia, amd, intel and vulkan have rows; the l4t and cuda-refined values reach the nvidia row by prefix; "apple" and "" cannot reach the vendor fallthrough because the darwin and no-GPU branches return earlier. default and darwin-x86 were the only live holes. BackendPreferenceTokens and its build-tag table are untouched, and preferenceRank stays generic, naming no engine and no capability. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: prefer speculative-decoding builds when they fit Rank serving features between engine preference and size, so a host that can hold a DFlash or MTP build of a model's weights installs it instead of the plain build. Both answer faster for the same output, so whenever one survives the filters there is no reason to take the plain build. Precedence is now fit, then engine, then serving feature, then size. Engine outranks the feature deliberately: a serving feature makes the right engine faster, it does not make a wrong engine right, so a plain vLLM build still beats a DFlash llama.cpp build on NVIDIA. Fit outranks both, and a drafter pairing is strictly larger than the plain build, so the existing size filter drops it on a host too small for it before this axis is consulted. The order lives in a third preference table in pkg/system, alongside the build tag and engine name tables. It is the odd one of the three: not keyed by capability, because no hardware prefers a plain build over an equivalent faster one, and matched against whole segments of a gallery ENTRY NAME rather than as a substring of a backend value. Nothing on a gallery entry declares a serving feature, and tags are not a usable substitute: gemma-4-e2b-it:sglang-mtp carries an mtp tag while ornith-1.0-9b-mtp and qwen3.6-27b-nvfp4-mtp carry none. Entry names are author-supplied free text, unlike the closed engine vocabulary, so a short marker can turn up inside an unrelated word and whole segment matching is what keeps smtp-assistant from ranking as an MTP build. The block comment over the tables now documents all three together and states what each is matched against; the ranking code names no feature, so adding one stays a one-line edit to the table. 29c49203b rejected these entries as serving configurations rather than alternative builds of the same weights. The definition is now "alternative ways to serve the same model", which includes them, so regroup 14 entries under 12 parents. Judged by the files each entry points at: the qwen3.6, qwen3.5, qwen3 and deepseek pairings are the base GGUF plus a drafter, the gemma-4 QAT MTP entries are the same QAT weights at a different quantization plus an MTP drafter, and the two sglang MTP entries describe themselves as the same model served with speculative decoding. Left separate: qwen3.6-27b-mtp-pi-tune, a finetune with its own weights, and every entry whose base model LocalAI does not ship as its own row, which is the whole Qwopus line plus gemmable-4-12b-mtp, mimo-7b-mtp:sglang and qwen3.5-4b-dflash. None of the twelve parents defines a YAML anchor, so no variants key can leak through a merge key and no empty override was needed this time. The index was edited by line insertion only. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test: check env restore errors in capability and variant specs errcheck flagged ten unchecked os.Setenv and os.Unsetenv returns in the specs added while the pre-commit hook was being skipped. Restoring an env var is exactly the place a silent failure leaks state into the next spec, so assert on it rather than suppressing the linter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): make the mtp tag authoritative for serving-feature ranking Variant auto-selection ranks survivors by fit, then engine, then serving feature, then size. The serving-feature lookup read only whole alphanumeric segments of a variant's entry name, because tags were inconsistent: every dflash entry carried a dflash tag, but only 7 of 20 MTP entries carried an mtp tag. Tag the 13 untagged MTP entries, then teach the lookup to read tags as well as names. A tag is now the authoritative signal and is compared whole and case-insensitively, which is safe precisely because a tag is a deliberate declaration rather than free text: there is no word-inside-a-word failure mode, so the segment splitting the name half needs is unnecessary there. The name check stays as a fallback rather than being replaced. Switching to tags only would have regressed the six already-grouped entries on the day it shipped, and would depend on tagging discipline that does not exist yet. The lookup still names no feature, so adding one remains a one-line edit to servingFeaturePreferenceTokens. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): make a declared tag the sole serving-feature signal Variant auto-selection ranks survivors by fit, then engine, then serving feature, then size. The serving-feature lookup recognised a speculative build by either a declared tag or a whole segment of its entry name. Drop the name half: a tag is now the only signal. A name is author-supplied free text and a naming convention is not a contract, so reading a marker out of one infers a capability nobody declared. The gallery already had the failure in it: the four NVFP4 entries name MTP-bearing weights while setting no option that enables speculative decoding, and being live variants they were winning the feature axis without answering any faster. overrides.options was considered as the replacement and rejected. It carries spec_type:draft-mtp / spec_type:draft-dflash, which is what actually turns the feature on, but that spelling is llama.cpp's config vocabulary: ds4 spells the same feature mtp_path and sglang spells it speculative_algorithm in a referenced config. Keying a cross-backend ranking decision on one backend's option syntax would rank the other backends' builds as plain. Options are the curation-time check instead, and never reach the selection logic. With no fallback left, tag correctness is load bearing, so audit every entry against the rule "tagged when the entry configures that feature, in whatever vocabulary its backend uses". Three entries configure MTP untagged and gain the tag (hy3, glm-5.2, qwythos-9b-claude-mythos-5-1m, all spec_type:draft-mtp with no marker in their names). Four carry the tag while configuring nothing and lose it: qwen3.6-27b-nvfp4-mtp, qwen3.6-35b-a3b-nvfp4-mtp, qwopus3.6-27b-coder-mtp-nvfp4 and qwopus3.6-27b-v2-mtp-nvfp4, whose only option is use_jinja:true. The dflash side was checked independently rather than assumed consistent: all five dflash entries declare spec_type:draft-dflash and all five are tagged, so it needed no edits. Four entries keep a tag that a literal spec_type-only reading would strip, because they configure MTP through a different backend: deepseek-v4-flash-q2-mtp via ds4's mtp_path/mtp_draft, and the three sglang entries via speculative_algorithm in their referenced configs. Stripping those would contradict the reason spec_type was rejected as the signal and would demote four genuinely faster builds to plain. The index was edited by line insertion and deletion only, never round-tripped through a serializer. A resolved-tag diff across all 1272 named entries, taken after merge keys are applied, shows exactly these 7 changing and no entry gaining or losing a tag through an anchor. The two specs that pinned the name fallback are inverted rather than deleted, since a name silently promoting a build is the regression worth guarding. The whole-token guard survives on the tag path, where smtp must still not match mtp. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): make deepseek-v4-flash variant targets installable Clicking install on deepseek-v4-flash failed with "invalid gallery model". The parent entry is fine, but all four entries it was grouped with declared neither url: nor config_file:, and applyModel needs one of the two to have anything to build a config from. They carry urls: (plural), the informational HuggingFace link list, which is a different field. None of the four was ever independently installable, so grouping them routed a previously-working install into a broken entry. Give each the url: the parent already resolves through. virtual.yaml is a no-op base, and applyModel passes overrides to InstallModel separately from the fetched config, so backend: ds4, the parameters and the ssd/mtp options all still land exactly as authored. This is the same pattern the parent and many other GGUF entries in the index already use. Add the lint rule that should have caught this. checkVariantReferences only proved a target exists and is not itself a parent, which is structural validity: an entry can exist, declare no variants, and still be uninstallable. checkVariantTargetsInstallable mirrors applyModel's precondition instead, and names the parent, the target and the missing fields, because whoever hits it is reading a gallery entry and has no reason to know applyModel exists. The two index-driven resolution specs live in their own Ordered container: an Ordered container stops at its first failure, so sharing one with the lint rules let a lint breach skip them silently. Nine further entries gallery-wide have the same defect and are unrelated to variants, so they are broken installs that predate this branch. They are left alone here rather than buried in a regression fix, and widening the rule to cover every entry is deferred with them so the gate can ratchet up in one step instead of needing a skip list. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): install entries with no url or config_file on an empty base applyModel had three branches: fetch a base config from url:, build one from an inline config_file:, or fail with "invalid gallery model". An entry declaring neither is now installed on an empty base config, with overrides: and files: supplying everything. This is what the ~345 entries pointing at gallery/virtual.yaml were already getting. That stub is five lines carrying name, description and license. description and license are overwritten from the gallery entry immediately after the fetch, and the name never reaches disk because InstallModel prefers the install name. Crucially applyModel passes model.Overrides to InstallModel as a separate argument rather than merging it into the fetched config, so nothing an author writes depends on that base existing. The fetch bought a round trip to GitHub and nothing else. That makes f4ef80173 the wrong fix, so it is unwound. The four url: lines it added to the deepseek-v4-flash variants are reverted: they are a pointless network fetch now, and the family installs without them. Relaxing the branch would hide a real authoring mistake, so a payload rule replaces the base-config rule. An entry with no url, no config_file, no overrides and no files installs nothing and would leave an empty model directory while reporting success, so it is refused by name. The caller's request counts toward the payload, because its overrides and files are merged into the install exactly as the entry's own are. urls: (plural) is the informational link list and does not count, which is what the four entries that shipped broken had and why they were still uninstallable. checkVariantTargetsInstallable asserted every variant target declares a url: or a config_file:, which is no longer true and would now reject correct authoring. checkEntriesInstallSomething pins what survives instead, and covers every entry rather than only variant targets: the hazard is a half-written stanza and a parent can be one as easily as a target. The old rule was scoped to targets precisely because nine unrelated entries would have failed a gallery-wide version; those nine are valid now, so the deferred ratchet happens here in one step. 1280 entries, zero violations. Those nine (aurore-reveil_koto-small-7b-it, lfm2-1.2b, the six liquidai_lfm2 entries and deepseek-v4-pro-q2-ssd) become installable for free. Each carries overrides: and files:, and one of them is driven through the real install path in a spec. The no-fetch spec is paired rather than bare: an assertion that nothing was fetched proves nothing unless something could have been, so a control runs the same fixture with a url: pointing at a base config that is not there and asserts the install fails. Only then does the identical fixture without the url passing mean the read was skipped. Follow-up, deliberately not here: the ~345 entries still naming virtual.yaml can drop their url:. That is 345 index edits with their own risk, and mixing them in would bury this change. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ui(models): let search bypass the variant collapse, drop the toggle The models page collapsed the gallery to one row per model by default and offered a toggle to see every individual build. Because the collapse composed with the search term, a build another entry offers as a variant could not be found by typing its name, so the toggle was the only way to reach those builds in the UI. A user who typed a name they knew existed got "no models found", which reads as "that model does not exist". Collapse is for browsing; search is for finding. An explicit search term now bypasses the collapse in the listing handler, so a name lookup returns matching entries whether or not a parent offers them. The term is trimmed once at the top of the handler, so whitespace is neither a search nor a bypass; previously an untrimmed blank term also narrowed the listing to whatever contained a space. Tag and backend deliberately do not bypass: they refine a listing the user is still reading rather than name an entry already known to exist. That makes the toggle redundant, so it goes, along with its i18n strings in all six locales, its localStorage persistence, its participation in "Clear filters" and the empty-state hint telling users to turn it off. The hint was doubly stale: it pointed at a control that no longer exists, and it was untrue exactly when a user has a search term, since searching now sees every build. The page always requests the collapsed listing. The stored preference key is left inert rather than cleaned up: nothing reads it, so a user who had the toggle off simply gets the collapsed view. collapse_variants stays on the API, off by default, because other clients want either view and the UI dropping its control is no reason to remove a working parameter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): give the models gallery filter form a deliberate structure The filter area had accreted controls into one undifferentiated flow. The "Fits in GPU" toggle and the backend select were direct children of .filter-bar, the same wrapping container as the 18 taxonomy chips, so their position was decided by how many chips happened to wrap at the current width rather than by any layout intent. At narrow widths they were pushed past the right edge of that container's horizontal scroll and became unreachable entirely. Restructure into three bands inside the house .filter-bar-group wrapper that components/FilterBar.jsx already uses on Backends and the System tabs: 1. query scope: search plus the backend select 2. taxonomy: the chip row, alone, free to wrap 3. refinements: fits-in-GPU and context size, under a hairline rule The backend select leads the chips rather than trailing them because picking a backend disables the use cases that backend cannot serve, so it gates the row below it. Fits-in-GPU and context size share a band because they are one control group: the context size is the length the VRAM estimate is computed at, and that estimate is what the fits filter tests against. Chips had no visible keyboard focus indicator. The global focus ring is wrapped in :where(), so it carries the specificity of a bare :focus-visible, ties with .filter-btn and loses on source order, leaving focused chips showing their resting drop shadow. Restate the ring where it outranks both resting and hover. Also: aria-pressed on the chips, a real label association and aria-valuetext on the context slider (it steps over an index, so it announced "2"), disabled chip styling moved off inline styles, a prefers-reduced-motion block for the chip transition, and the hard-coded English "Context:" moved into all seven locales. No behaviour change: same filters, same state, same requests. Page reset on change, localStorage persistence and "Clear filters" verified unchanged. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): let the models recommendations panel fade into the background The "Recommended for your hardware" strip rendered at full height on every visit regardless of how many models were already installed, costing 186px at 1600px wide (287px at 1100px, where its cards wrapped to two rows) and pushing the first gallery row to y=554 / y=703. Make its prominence track how much the user still needs it. The panel now defaults to a one-line summary once anything is installed, and both the collapse choice and the existing dismissal persist: collapsed = explicit user choice, if one exists : installedCount > 0 The preference is three-valued on purpose. A boolean cannot tell "the user expanded it" apart from "the user has never chosen", and those need opposite handling when the installed count later crosses zero: someone who deliberately opened the panel on an empty instance should not have it collapse out from under them when their first model finishes installing. Collapsed keeps the card, icon, title and a suggestion count, so the panel is recovered by clicking what you are already looking at rather than by hunting. Expanded is unchanged, because for a user with nothing installed it was never the problem. Collapsed reclaims 145px at 1600 and 420, and 246px at 1100. Models.jsx gains a statsLoaded flag: stats initializes to installed:0, so reading it before the fetch resolves would render expanded and collapse a frame later, which is exactly the layout shove this removes. The dismissal key moves to the page's localai-models-* convention; the old localai_rec_models_dismissed is still read, never written, so an existing dismissal is honoured rather than resurrected by the rename. Accessibility: the disclosure is a real button whose accessible name is the visible title alone, with state on aria-expanded and aria-controls resolving in both states, because the grid is hidden via the hidden attribute rather than unmounted. That also keeps the four install buttons out of the tab order while collapsed. The app's global focus ring applies; no per-component outline is added, per the warning in App.css. Reveal animates opacity and transform only, never height, and both it and the chevron rotation are disabled under prefers-reduced-motion. Only en had a recommended block, so the other six locales were falling back to English for the whole panel. Translated the complete block rather than adding one orphaned key to files that would still render the title in English. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(downloader): recover from a leftover .partial on non-HTTP URIs An interrupted download leaves a `<file>.partial` behind. The partial handling in DownloadFileWithContext gated resume on `err == nil && uri.LooksLikeHTTPURL()`, so for any URI that is not literally http(s) the branch fell through to `else if !errors.Is(err, os.ErrNotExist)`, which with a nil err is true. The download then failed with an error wrapping nil: failed to check file ".../Ternary-Bonsai-27B-Q2_g64.gguf" existence: <nil> Every gallery file URI uses `huggingface://`, so a single interrupted download made that model permanently uninstallable until someone deleted the partial by hand. The `<nil>` in the message compounded it by pointing debugging at a filesystem failure that never happened. Restructure the handling as an explicit switch over the four real states: partial exists and is resumable, partial exists and is not resumable (discard and restart, as already done for an HTTP server without range support), no partial, and a genuine stat failure. The error branch is now only reachable with a non-nil error, names the path that was actually stat'd, and wraps with %w. Discarding is required for correctness and not merely convenience: the writer opens the partial with O_APPEND, so an un-resumed download would concatenate a fresh body onto stale bytes. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): tell the models gallery's variant rows apart, and let browsing see every build Both variant surfaces rendered name, backend and size. For two builds of one model that is close to no information: a variant exists precisely because the same weights are offered another way, so the backend usually matches and the sizes usually land within a few hundred megabytes. Comparing ternary-bonsai-27b-pq2 against ternary-bonsai-27b-q2-g64 meant reading two names that differ by a suffix nobody has defined anywhere in the UI. Report the quantization and the serving features on VariantView, and derive both server-side from the referenced entry rather than parsing names in the browser, so every client reads the same format out of the same file the installer will hand the backend. Quantization comes from overrides.parameters.model first, falling back to the file list. That order is load bearing: entries routinely ship a vision tower alongside the language model at a different quantization, so reading the file list first reports the mmproj's format. Matching walks `-` and `.` delimited segments right to left; `_` deliberately does not split, because it separates the parts INSIDE a quant token and splitting on it reports Q4 for a Q4_K_M build. A second, looser pass takes a segment's `_`-delimited tail, which catches the gemma-4-E2B_q4_0-it.gguf style; it runs second so a precise match can never lose to a fuzzy one further right in the name. An entry naming no format reports nothing, which is the honest answer for a backend served from a directory of weights. Features are the same tag-against-vocabulary match servingFeatureRank already ranks on, over the same host preference list. A build can therefore never be shown as faster than one selection did not actually reward, nor rewarded without being shown; a spec pins that agreement rather than trusting it. The compact dropdown gets the quantization on its meta line and the bare feature token. The detail row, which has the room, gets the quantization as its own monospaced column so precision lines up down the list, and the feature spelled out, because DFLASH names nothing to a user who has not met it. The referenced entry's description stays out of both: the detail row already renders the parent's prose above the table, and a second block per variant would push a three-variant list past a screen to restate what the columns now say precisely. The collapse toggle comes back. 462583f38 dropped it once search bypassed the collapse, on the reasoning that nothing was unreachable any more. That holds for finding a build whose name you know and does not hold for browsing: no sequence of actions enumerated the 68 builds the default view hides. Collapse is for browsing and search is for finding, and the toggle was the browsing half. It goes in the refinements band 0d4823362 established, not back among the taxonomy chips where its position depended on how many chips happened to wrap. It leads that band because it decides how many rows the other two refine over, and because unlike fits-in-GPU it is unconditional: a host with no GPU still browses. The search bypass is untouched and re-checked by a spec in the toggle's default state, since restoring the control must not restore the dead end it replaced. The empty-state hint returns but only without a search term, because a term bypasses the collapse and the hint would otherwise point at a control that cannot change the result. The stored preference reads 'on'/'off' only: an older build wrote '1'/'0' from an effect that ran on mount, so those record that the page was opened, not that anyone chose a view. Also fixes a latent flake it exposed. The collapse_variants spec compared whole response bodies byte for byte, and the listing envelope carries live host telemetry that drifts between two calls milliseconds apart, so it was asserting on the machine's memory pressure. It now compares everything the parameter governs -- the entries, their serialization and the paging -- and is green 25/25 where it was failing about one run in three. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): let the models gallery show a variant's full details The variant list in an entry's expanded detail row says how the builds differ: name, backend, quantization, size, and the auto-selected, base and serving-feature markers. It cannot say what any one of them is. A variant's own description, tags, license, source links and file list are unreachable anywhere in the UI, because while the collapse is on a variant has no gallery row of its own. Give each variant row an info control that reveals its entry, rendered by the same ModelDetail a top-level row gets, so a field added to the detail view appears here too. variantData is withheld from the nested render: a variant may declare variants of its own, and recursing would nest a picker inside a picker two levels deep already. An inline disclosure rather than a modal. The control sits inside a table row that is already expanded, inside a variant list within that; a dialog opened from there stacks a dismissal on a dismissal for a handful of extra fields about the entry the user is already reading, and breaks the page's own expand idiom. The third level is carried by an inset and a left rule instead of another card. The entry is fetched by exact name from the listing, once, on first use. The listing already returns every field the detail view renders, and a search term bypasses the variant collapse server-side, so no new endpoint is needed and neither the listing nor DescribeVariants gains any work. Expanding a row costs nothing; a variant nobody opens costs nothing. A name the listing no longer returns is stated, not blanked: an empty panel reads as a rendering fault rather than as a lookup that came back empty. The control is a sibling of the install button, not a descendant, so asking about a build can never install it. The variant list keeps its content-sized columns via a trailing filler track instead of max-content sizing, so the rows are unchanged while the panel spanning them gets the pane width its file table needs. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): let search respect the collapse instead of switching it off The models listing collapsed to one row per model, and an explicit search term turned that off wholesale. Searching while collapsed therefore answered with the individual builds a parent already offers, which are exactly the rows the view the user asked for has no place for: typing "mtp" returned qwen3.6-27b-nvfp4-mtp, a row that is invisible the moment the box is cleared. The bypass was the right shape of fix for the wrong half of the problem. What a search must not do is answer "no models found" for a build the gallery does hold; that does not require abandoning the grouping the user asked for. So the term is now matched against every entry either way, hidden builds included, and the collapse decides how a match is reported rather than which matches exist. Collapsing stops being a filter that drops rows and becomes a substitution: a match on a build another entry offers is reported as that entry, the one installable in its own right. Nothing becomes unfindable and nothing comes back that the requested view cannot show. Substitution happens after search, tag and backend, so every filter is judged against the build that really carries the name, tag or backend rather than against a parent that merely offers it; the other order would let backend=vllm match a parent whose own backend is something else. The price is that the surfaced row shows the parent's own metadata while the match was on a variant, which is what grouping means, and the alternative is claiming the gallery holds no such build. It happens before the count and the page math, so both describe the rows actually handed out rather than the matches that produced them. A parent already in the result keeps its own position and absorbs its matching variants there, which is what leaves the browsing listing ordered exactly as it was; a parent surfaced only by a variant takes the position of the first variant that surfaced it. Either way it appears once, however many of its builds matched and whether or not it matched itself. Search preserves gallery order rather than scoring, so a surfaced parent has a real position rather than an invented one. VariantParents never reports an entry that declares variants of its own, so a parent is never itself hidden and one hop always lands on a visible row. The handler follows exactly one anyway: refusing the second is what makes a gallery the linter would have rejected terminate rather than loop. The empty-state hint pointing at the toggle goes with it for every server-side filter. Substitution means a match is always reported as some row, so the collapse can no longer be why a term, a chip or a backend came back empty, and naming it there sends the user to a control that cannot change the result. It survives for the fits filter alone, which runs in the browser after the substitution and judges the surfaced entry's own size: there the build that fits really can be filtered out along with a parent that does not. Searching a build's exact name while collapsed now answers with its parent, so the result no longer contains the string the user typed. That is intended, and the row is the one they can act on, but it is a real rough edge: nothing on the row explains the connection. Closing it properly means reporting which variant matched so the UI can say so, which the listing does not do today. ResetGalleryModelCache is added for tests. The model cache is a package global keyed by nothing, so a background refresh one spec triggers can land in the middle of the next and answer it with the previous spec's gallery; the extra specs here made that fail about one run in five. It waits for the in-flight refresh to publish before clearing, since clearing alone only narrows the window. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-20 18:43:02 +02:00
LocalAI [bot]	b00422e45f	feat(backends): add LongCat video and avatar generation (#10792) ... * feat(backends): add LongCat video and avatar generation Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web] * refactor(config): declare model I/O modalities Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook. Assisted-by: Codex:GPT-5 [web] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-12 23:58:46 +02:00
LocalAI [bot]	40dae953f4	feat: interleaved thinking with tool calls (reasoning_content alias + Anthropic thinking blocks) (#10744) ... * feat(schema): accept reasoning_content as inbound alias for reasoning Interleaved-thinking clients (cogito, vLLM/DeepSeek-style) emit reasoning_content on assistant turns. Accept it as an inbound alias so reasoning survives the tool-result loop; canonical reasoning wins when both are present. Emission is unchanged (still reasoning). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(schema): pin interleaved reasoning+tool_calls round-trip Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(openai): pin reachedTokenBudget truncation detection Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): add thinking and signature fields to content blocks Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): parse inbound thinking blocks into reasoning Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): emit thinking blocks with synthetic signature on tool turns Extract buildAnthropicContentBlocks so non-streaming content assembly is unit-testable, and prepend a thinking block (with an opaque synthetic signature) before text/tool_use blocks when the request opts into thinking. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): stream thinking_delta and signature_delta before tool_use Extract anthropicStreamSequence so the streaming block order is unit-testable, and emit content_block_start(thinking) -> thinking_delta -> signature_delta -> content_block_stop before the tool_use block sequence when thinking is enabled. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add interleaved thinking with tool calls guide Add a features guide describing interleaved thinking: an assistant turn carrying reasoning and tool_calls together, the reasoning-round-trip contract (including the reasoning_content inbound alias and Anthropic thinking blocks with a synthetic signature), per-backend enablement (reasoning_format for llama.cpp, reasoning_parser/tool_call_parser for vLLM/SGLang plus the vLLM auto-config hook), a worked request/response example, and known limitations. Cross-link from model-configuration, text-generation, and openai-functions. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-08 16:45:43 +00:00
LocalAI [bot]	b0959d4756	feat(api): add GET /v1/models/capabilities endpoint (#10687) ... Additive superset of /v1/models that enriches each model entry with the capabilities it supports plus its input/output modalities (text / image / audio / video). Clients that only understand /v1/models are unaffected -- they simply never call the new route. Audio and video *input* are derived from the model's multimodal limits (vLLM limit_mm_per_prompt), which no single usecase FLAG expresses. That gap is exactly why a plain capability list is insufficient and this enriched endpoint exists: an attachment router can now decide whether an image/audio/video file can go to the active model directly, or must be converted/transcribed first. Capability derivation lives in core/config as the single source of truth (ModelConfig.Capabilities / InputModalities / OutputModalities / VisionSupported / ...); the Ollama capability surface now delegates to it instead of keeping a parallel copy. Vision is gated on chat/completion capability so a MediaMarker hydrated onto a non-chat model (e.g. a pure ASR/TTS backend) no longer reports a false vision capability. Read-only listing: no new FLAG_* flag, reuses the existing `models` swagger tag, and intentionally exposes no MCP admin tool (there is nothing to manage conversationally). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-05 08:51:55 +02:00
Richard Palethorpe	eb32cd9073	feat(realtime): eager blocking pipeline warm-up + /backend/load API (#10662) ... Realtime sessions previously lazy-loaded each pipeline sub-model (VAD, transcription, LLM, TTS) on first use, so every cold session paid a per-request model-load stall and load errors only surfaced mid-stream. Warm the whole pipeline eagerly and blockingly at session start (including the voice-gate speaker-recognition model, which an enforced gate blocks each utterance on; compaction's summary_model stays lazy since it only runs off the response path): - Add backend.PreloadModel / PreloadModelByName as the single load path for every modality (no transcription special-case; backend-omitted configs are deprecated). - The realtime session blocks on Model.Warmup and returns a model_load_error to the client if any stage fails to load; updateSession warms in the background. Opt out per pipeline with pipeline.disable_warmup, exposed as a UI toggle via the config-metadata registry. Add a LocalAI-native POST /backend/load (and /v1/backend/load) that pre-loads a model -- expanding realtime pipelines into their sub-models -- as the inverse of /backend/shutdown. There is one preload engine (backend.PreloadStages): the realtime Warmup methods, /backend/load and the --load-to-memory startup flag all use it, so --load-to-memory now also expands pipeline models and records load-failure traces. Pipeline sub-model alias resolution is likewise shared (ModelConfigLoader.LoadResolvedModelConfig). Surface the endpoint everywhere an admin manages models: - MCP admin tool load_model (httpapi + inproc clients, safety/catalog prompts, catalog/dispatch tests). - "Load into memory" action in the React models UI. - Swagger regenerated; docs moved to the general backend-monitor page since it is not realtime-specific. Fix a Traces UI crash ("json: unsupported value: -Inf"): audio-snippet RMS/peak now floor at a finite dBFS, and backend-trace data is sanitized to drop non-finite floats before marshaling. The sanitizer is copy-on-write -- it runs on every RecordBackendTrace, so containers are only re-allocated on the paths that actually changed. Migrate core/http/openresponses_test.go onto the prebuilt mock-backend the rest of the http suite already uses -- it was the last spec still pointing at a real HuggingFace model, so it 404'd wherever no vision backend was built -- and fix its item_reference specs to send the spec's "id" field instead of "item_id", which the handler never accepted. Assisted-by: Claude:claude-opus-4-8 Claude Code Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-07-03 18:00:37 +02:00
Richard Palethorpe	5d0c43ec6e	feat(realtime): Semantic VAD EOU token (#10444) ... * feat(realtime): EOU-driven semantic_vad turn detection Add a `semantic_vad` turn-detection mode to the realtime API that feeds the transcription model live and decides "the user finished speaking" from the `<EOU>` end-of-utterance token rather than from silence alone. When EOU fires the turn commits immediately (~0.3s); otherwise it falls back to an eagerness-scaled silence threshold (low/med/high = 8/4/2s). Plumbing, bottom to top: - proto: `AudioTranscriptionLive` bidirectional RPC (config-first oneof, mono float PCM @16k, ready-ack / Unimplemented degrade signal) plus `TranscriptResult.eou` for the unary retranscribe gate. - pkg/grpc: client/server/base/embed scaffolding for the bidi stream, modeled on AudioTransformStream; release stream conns on terminal Recv. - parakeet-cpp: live transcription RPC with per-C-call engine locking (one live stream per turn, finalize+free at commit); bump parakeet.cpp to ABI v5 — incremental StreamingMel (no more quadratic per-feed mel recompute that delayed EOU on long turns) and the <EOU>/<EOB> split; strip the literal <EOU>/<EOB> from offline text and set Eou. - core/backend: LiveTranscriptionSession wrapper + pipeline `turn_detection:` config block (type/eagerness/retranscribe). - realtime: semantic_vad integration — live input captions streamed as transcription deltas while the user speaks, EOU-immediate commit with eagerness fallback, optional retranscribe gate (batch re-decode must also end in <EOU> to confirm), clause synthesis off the LLM token callback, and per-turn live-transcription / model_load telemetry. - UI: show the realtime pipeline components as a vertical list. Docs and tests included; opt-in via the pipeline YAML or per-session `session.update`. Non-streaming STT backends degrade to silence-only. Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Write] [Bash] Assisted-by: Claude Code:claude-fable-5 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): explicit formally-verified state machines + parakeet streaming driver The realtime API had several implicit state machines whose state was inferred from scattered booleans, channels, and five separate mutexes, leaving illegal/inconsistent states reachable. Make them explicit and keep the implementation in step with a formal design; rework the parakeet streaming backend along the same lines. Realtime state machines (M1-M5). Each is a sealed sum-type State/Event/Effect with a total, pure Next(state,event)->(state,[]effect) behind a single-writer Coordinator: M1 conncoord connection lifecycle: VAD toggle + once-only teardown (replaces vadServerStarted + a `done` channel closed from two sites). M2 turncoord turn detection: collapses speechStarted and the live-stream "turn open" flag into one state, so discardTurn can no longer desync them and suppress the next onset. M3 respcoord response coordination: serializes the dual-writer start/cancel so at most one response is live; one response.done per response.create. M4 compactcoord conversation compaction: single-flight (replaces the `compacting atomic.Bool` CAS). M5 ttscoord TTS pipeline: open->closing->closed, idempotent wait(), rejects enqueue-after-close (was a silent drop). The Coordinator/Sink/Next plumbing — only the sealed types and Next differed per machine — is extracted once into core/http/endpoints/openai/coordinator as a generic Coordinator[S,E,F]; each machine keeps its public API via type aliases, so no sink, call-site, or test moved. Hierarchy. session_lifecycle.fizz models M1 as the parent region with its children (M2/M3/M4) as one statechart and asserts ChildrenDieWithParent (conn torn => all children terminal, none start after teardown). respcoord and compactcoord gain an absorbing Terminated state + Shutdown event; conncoord's teardown drives the children terminal. This closes a compaction teardown gap: a fire-and-forget compaction could outlive a torn session — compactionSink now takes a session-scoped cancellable context + WaitGroup and joins the in-flight summarize+evict on shutdown. Formal verification. formal-verification/ holds one authoritative FizzBee spec per machine plus the composition spec, each with an always-assertion and a documented one-line edit that makes the checker fail (verified non-vacuous). scripts/realtime-conformance.sh is fail-closed: all Go conformance suites under -race AND a model-check of every .fizz spec; a missing FizzBee is a hard error (only the loud REALTIME_CONFORMANCE_SKIP_FIZZBEE=1 bypasses it, never in CI). FizzBee is pinned by sha256 and installed via scripts/install-fizzbee.sh into .tools/ (gitignored). Wired as make test-realtime-conformance, a CI workflow, and a pre-commit path filter. Go conformance tests are Ginkgo/Gomega (per the repo's forbidigo lint): transition tables + fixed-seed property walks + concurrent/-race specs, no rapid dependency. Design map: docs/design/realtime-state-machines.md. Parakeet streaming backend. The same treatment applied to the parakeet-cpp streaming paths: - AudioTranscriptionStream returns codes.Unimplemented for non-streaming models instead of decoding offline and emitting it as one delta + final. A client that asked for streaming learns the model cannot stream rather than receiving a batch result shaped like a stream. New grpcerrors.StreamTranscriptionUnsupported carries that signal; the HTTP /v1/audio/transcriptions stream path surfaces it as an SSE error event. Mirrors AudioTranscriptionLive, which already did this. - utteranceBoundary (boundary.go): a single definition of the end-of-utterance latch, replacing three open-coded finalEou toggles. Modelled as a two-valued type so illegal states are unrepresentable. - Shared decode driver (driver.go): streamFeedResult (one per-feed event) + feedChunk (hides the ABI v4 JSON vs text-only split) + feedSlices + flushTail. The feed loop is written once. - AudioTranscriptionLive becomes a bidi adapter: it streams the per-feed {delta,eou,eob,words} the realtime turn detector consumes and a terminal FinalResult carrying only Text. Segments/duration/eou are offline-only and no longer produced (nor read) on the live path; liveTraceState drops the terminal eou and keeps the per-feed eou_events count. - AudioTranscriptionStream + streamJSON merge into one driver-based function; streamSegmenter is generalized to the unified event with a text-only fallback that preserves the legacy (no-words) library's per-utterance segmentation. Verified: build/vet/gofumpt clean, golangci-lint 0 issues, all coordinator and parakeet packages under -race, the fail-closed conformance gate green, and make test-realtime (12 e2e WS+WebRTC). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-30 09:01:22 +02:00
LocalAI [bot]	f0d0bff232	fix(llama-cpp): stop reinterpreting plain-string message content as JSON (#10524) (#10538) ... The llama-cpp gRPC backend reconstructs OpenAI messages from proto for the tokenizer-template path and blindly json::parse'd each message's content string. LocalAI's Go layer always flattens content to a plain string, so a user prompt that merely looks like JSON (e.g. mealie's ingredient array ["1/4 cup brown sugar", ...]) was reinterpreted as structured content parts and rejected by oaicompat_chat_params_parse with "unsupported content[].type". Normalize content per role instead: user/system/developer content is opaque text and is never JSON-sniffed; assistant/tool content still collapses a literal JSON null/object (tool-call bookkeeping) to a string, but a plain string is never turned into an array/scalar. The array defense is role-independent, so the role gate only governs the benign null/object case. While here, extract the duplicated per-message reconstruction and the pre-template content sanitization into shared, unit-tested helpers (message_content.h) so the streaming (PredictStream) and non-streaming (Predict) paths cannot drift. This removes ~490 lines of copy-pasted defensive code, the dead tool-role parse branches, and the redundant Predict-only tool_calls branch, while preserving the prior #7324 (null content -> "") and #7528 (tool array content -> string) fixes. Tests: - backend/cpp/llama-cpp/message_content_test.cpp: standalone C++ unit tests for all three helpers (#10524, #7324, #7528, multimodal), discovered and run by `make test-backend-cpp` and a new generic tests-backend-cpp CI job. Also wired as an opt-in CMake/ctest target (-DLLAMA_GRPC_BUILD_TESTS=ON). - core/schema/message_test.go: Go regression pinning that ToProto flattens a JSON-array-looking text part to the verbatim string. - prepare.sh now copies message_content.h into the build tree. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-27 01:42:05 +02:00
LocalAI [bot]	600dafd20b	feat(ced): sound-event classification backend (CED audio tagger) (#10425) ... * feat(ced): sketch sound-classification backend (CED audio tagger) Wires ced.cpp (CED, 527-class AudioSet sound-event tagger; baby cry, footsteps, glass, alarms, dog bark) into LocalAI as a Go/purego backend. SKETCH (backend skeleton real; core REST wiring + CI/gallery is a checklist in DESIGN.md): - backend/backend.proto: new SoundDetection rpc + SoundClass messages (run `make protogen-go` to regenerate pkg/grpc/proto). - backend/go/ced: main.go (purego dlopen libced.so + ced_capi.h), goced.go (Ced gRPC backend: Load + SoundDetection), Makefile (clone-at-pin CED_VERSION, ggml static-PIC shared build), run.sh, package.sh, .gitignore. - DESIGN.md: REST /v1/audio/classification wiring (handler/route/capability registration checklist), gallery/index + CI registration, and a scoping note for the realtime/websocket live-recognition path (sliding-window classify over the existing ws transport + voicegate; the ced C-API per-PCM entry point is already window-friendly). Backend code does not compile until protogen-go regenerates the pb types and a libced.so is built (Makefile clones+builds it). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): REST /v1/audio/classification endpoint + capability registration Wires the ced sound-event classification backend (AudioSet audio tagger) end to end through the REST surface, mirroring the transcription path. - Handler: core/http/endpoints/openai/sound_classification.go parses the multipart audio upload, temp-files it, resolves the model config and calls the SoundDetection RPC; returns {model, detections[]} JSON. - Backend wrapper: core/backend/sound_classification.go (ModelSoundDetection) loads the model and normalizes the proto response into schema types. - Schema: core/schema/sound_classification.go (SoundClassificationResult). - gRPC layer: SoundDetection wired through the LocalAI wrapper (interface, Backend client, Client, embed, server, base default) so the loader-typed client exposes the RPC; proto regenerated via make protogen-go. - Route: POST /v1/audio/classification (+ /audio/classification alias) with the audio/multipart default-model middleware in routes/openai.go. - Capability surfaces: swagger @Tags/@Router on the handler; FLAG_SOUND_ CLASSIFICATION usecase flag + UsecaseSoundClassification + UsecaseInfoMap + GuessUsecases + ModalityGroups + GetAllModelConfigUsecases; meta usecase option; /api/instructions audio area updated; auth RouteFeatureRegistry + FeatureAudioClassification (APIFeatures, default ON) + FeatureMetas; UI usecaseFilters, capabilities.js CAP_SOUND_CLASSIFICATION, Models.jsx filter + i18n; docs page features/audio-classification.md + whats-new + crosslink. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): realtime sound-event detection over the websocket API When a realtime pipeline configures a sound-classification model, each VAD-committed utterance (the same window the transcription path produces) is also run through the CED sound-event classifier and the scored AudioSet tags are emitted as a new server event. No new backend rpc is needed: the SoundDetection gRPC method already exists on this branch. - config: add Pipeline.SoundDetection (yaml/json sound_detection,omitempty) beside Transcription/VAD. - realtime: add Model.SoundDetection(ctx, audio, topK, threshold) to the ModelInterface; implement it on wrappedModel and transcriptOnlyModel by calling backend.ModelSoundDetection with the session's sound-classification model config (mirrors how Transcribe dispatches). Load the optional config in newModel / newTranscriptionOnlyModel; nil config keeps it additive. - types: add ConversationItemSoundDetectionEvent (item_id, content_index, detections[]{label,score,index}) with type conversation.item.sound_detection, its ServerEventType constant and MarshalJSON, mirroring the transcription completed event. - realtime: add emitSoundDetection (unary path: classify the committed window, build the event, t.SendEvent) and wire it at the utterance-commit hook right after emitTranscription; gated on session.SoundDetectionEnabled (resolved from Pipeline.SoundDetection at session setup, defaults top_k=5, threshold=0). Its error is logged via xlog but never aborts the turn. - test: Ginkgo specs for emitSoundDetection (tags emitted, empty detections, classifier error) plus a SoundDetection method on the fakeModel double. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): implement SoundDetection in nodes backend test doubles The SoundDetection method added to the grpc backend interface left two test doubles (fakeBackendClient, fakeGRPCBackend) incomplete, so core/services/nodes failed to compile under `go vet`/`go test` (go build missed it: the doubles live in _test.go). Add the method to both, mirroring their existing Detect mock. Repairs CI for the nodes package. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): decouple realtime sound detection from VAD (sound-only sessions) Sound-event detection must activate on sounds, not speech, so it no longer runs through the voice VAD/transcription path. A sound-detection-only pipeline (sound_detection set, no transcription/LLM) now: - is accepted by prepareRealtimeConfig (sound_detection counts as a pipeline stage), - builds a lightweight model via newSoundDetectionOnlyModel (no VAD/STT/LLM/TTS loaded), and - defaults the session to turn_detection none (no VAD) with no transcription stage, so the client drives windowing via input_audio_buffer.commit (option A: client-side sliding window). The per-PCM C-API already supports arbitrary windows. commitUtterance gains a sound-only branch: it emits the conversation.item.sound_detection event (scored AudioSet tags) and stops - no transcription, no LLM response. generateResponse is now guarded on a transcription stage being present, so a sound-only turn never invokes the LLM. Existing transcription/VAD sessions are unchanged (additive). Added a commitUtterance sound-only Ginkgo spec asserting it emits the sound event and neither transcribes nor generates a response. go vet + golangci-lint (new-from-merge-base) clean; openai suite green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): register sound-classification backend in gallery + CI Mechanical backend-image registration for the ced sound-event classifier, mirroring the parakeet-cpp Go/purego backend everywhere it is wired up. - .github/backend-matrix.yml: add the ced build matrix, field-for-field copies of the parakeet-cpp entries (cpu amd64/arm64, cublas cuda 12/13 amd64, l4t cuda-13 arm64, l4t-jetpack cuda-12 arm64, sycl f32/f16, vulkan amd64/arm64, rocm hipblas, and the metal darwin entry), changing only backend and tag-suffix. dockerfile stays ./backend/Dockerfile.golang. - backend/index.yaml: add the &ced meta anchor (capabilities map per platform) plus ced-development and the per-arch image entries, each uri/mirror tag-suffix matching the matrix exactly. The model gallery (GGUF) entry is intentionally deferred pending the HuggingFace publish (TODO note inline). - scripts/changed-backends.js: add an explicit item.backend === "ced" branch in inferBackendPath mapping to backend/go/ced/, same mechanism and ordering as the parakeet-cpp branch (before the generic golang fallthrough). - .github/workflows/bump_deps.yaml: register mudler/ced.cpp -> CED_VERSION in backend/go/ced/Makefile so the daily bot bumps the pin. - swagger/{docs.go,swagger.json,swagger.yaml}: regenerated via make swagger so the existing /v1/audio/classification annotations land in the generated spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): server-side windowing for realtime sound detection (option B) Adds an optional server-driven sliding-window classifier so a sound-only realtime client only has to stream audio (no input_audio_buffer.commit): - Pipeline.sound_detection_window_ms / sound_detection_hop_ms config knobs. When both > 0 on a sound-only session, the server classifies the last window of streamed audio every hop and emits a conversation.item.sound_ detection event; the input buffer is trimmed to one window so a long stream stays bounded. When unset, the session stays client-driven (option A). Runs independent of VAD (sound events are not speech). - handleSoundWindow (ticker) + classifySoundWindow (one tick, extracted so it is unit-testable) + writeWindowWAV, which declares the true InputSampleRate (NewWAVHeaderWithRate) so the classifier resamples correctly. Goroutine is started after toggleVAD and torn down with the session (close + wg.Wait). - Register pipeline.sound_detection (+window_ms/hop_ms) in the config meta registry; the earlier realtime commit added pipeline.sound_detection without a registry entry, failing TestAllFieldsHaveRegistryEntries. This fixes that and covers the two new knobs. Tests: classifySoundWindow emits an event + trims the buffer to one window, no-ops on too-little audio; writeWindowWAV declares the given sample rate. go build/vet + golangci-lint (new-from-merge-base) clean; config + openai suites green. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add ced-base GGUF model gallery entries (f16 + q8_0) The ced-base weights are now published at mudler/ced-base-gguf (Apache-2.0, converted from mispeech/ced-base). Adds gallery/ced.yaml (backend: ced + known_usecases: sound_classification) and two gallery/index.yaml entries (ced-base-f16 default, ced-base-q8 smallest) with sha256-pinned files, and removes the now-resolved TODO from backend/index.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ced): add tiny/mini/small GGUF model gallery entries Publishes the rest of the CED family (same architecture, metadata-driven port verified end-to-end on ced-tiny) to mudler/ced-{tiny,mini,small}-gguf and adds their f16 + q8_0 gallery entries: ced-tiny (5.5M, edge/Pi-class) f16 11MB / q8_0 6MB ced-mini (9.6M) f16 19MB / q8_0 11MB ced-small (22M) f16 42MB / q8_0 23MB All sha256-pinned. ced-base remains the accuracy default. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): point gallery entries at the consolidated mudler/ced-gguf repo All CED quantizations (tiny/mini/small/base, f16/q8_0) now live in a single HuggingFace repo, mudler/ced-gguf, instead of per-model repos. Repoint the 8 gallery model entries' urls + file uris accordingly. sha256 and filenames are unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): bump CED_VERSION to the short-clip fix Pin the ced backend to ced.cpp 99c6ed3, which fixes a crash on any clip shorter than target_length (~10.11s): time_pos_embed was added at its full 63-frame grid instead of being sliced to the clip's actual time grid, tripping ggml_can_repeat in ggml_add. Surfaced by the live realtime e2e (sub-10s windows) and gated with a short-clip parity test upstream. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(ced): list ced.cpp as a LocalAI-team engine + backend-guide directive - README.md: add ced.cpp to the "native C/C++/GGML engines developed and maintained by the LocalAI project" table. - docs/content/features/backends.md: add a Sound Classification backend category (sound-event classification / audio tagging) listing ced.cpp. - .agents/adding-backends.md: add a "Documenting the backend" section and two verification-checklist items requiring new backends to be documented in the backends.md category list, and in-house native engines to be added to the README maintained-engines table. This directive was missing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ced): repin CED_VERSION to the v0.1.0 release commit ced.cpp history was squashed into a single release commit (tagged v0.1.0), so the previous pin (99c6ed3) no longer exists upstream. Pin to c04ac14, the v0.1.0 release commit, so the backend builds against a commit that exists. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ced): silence gosec G304/G103 + govet unsafeptr on audited paths - sound_classification.go: os.Create(dst) where dst = temp dir + path.Base of the upload (no traversal). #nosec G304, matching the depth-anything-cpp handler. - goced.go: reading a NUL-terminated C string from a libced-owned buffer. #nosec G103 (gosec) + //nolint:govet (golangci-lint's unsafeptr check), since the uintptr is a C-owned malloc'd buffer, not Go-GC memory. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-22 01:00:28 +02:00
Richard Palethorpe	3fa7b2955c	feat(pii): NER tier engine — privacy-filter.cpp backend + NER-centric PII filter (#10360) ... Squashed feat/pii-ner-tier-engine rebased onto master (was 45 commits; see backup/pii-ner-tier-engine-prerebase). Net change: - privacy-filter.cpp: standalone GGML engine for the openai-privacy-filter PII/NER token classifier, wired as a LocalAI gRPC backend (CPU/CUDA/Vulkan). TokenClassify moves off the patched llama.cpp path onto this backend. - PII filter reworked to be NER-centric (encoder/NER detection tier scanning whole conversations as one document), with a recreated bounded restricted- regex secret-matching pattern detector tier alongside it (per-model pii_detection.builtins / .patterns + core/services/routing/piipattern). - Detection labelled by source (ner vs pattern); backend trace / confidence / debug observability; analyze/redact exposed as a synchronous API. - Instance-wide default detector policy + per-usecase default-on; request filtering extended to completions, embeddings, edits & Ollama. - React UI: NER-centric PII editor, detector-models table, pattern/builtins editor, middleware default-policy UI. - Gallery: privacy-filter-multilingual token-classify model + NER install filter; token_classify known_usecase; batch sized to context for NER models. privacy-filter backend registered in the backend gallery (cpu/vulkan/cuda-13 meta + image entries with a capabilities map) matching its CI matrix jobs, and an /import-model auto-detect importer (PrivacyFilterImporter, narrow privacy-filter GGUF detection) replacing the prior pref-only registration. Reconciled against master's independent evolution: - Dropped master's PIIPatternOverrides feature (global-pattern runtime overrides + /api/pii/patterns API + runtime_settings.json persistence). The per-model NER + pattern-detector design supersedes it; it was built on the global redactor pattern set this branch replaced. - Reverted the llama.cpp Score carry-patch (0006-server-task-type-score): removed the patch and restored master's grpc-server.cpp Score RPC (direct llama_decode, slot-loop bypass) and LLAMA_VERSION pin, plus master's model_config validation forbidding score + chat/completion/embeddings on llama-cpp. token_classify is unaffected (it runs on the privacy-filter backend, not llama-cpp). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-18 11:45:22 +01:00
LocalAI [bot]	294170d3ed	feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery (#10352) ... * feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery Mirrors the locate-anything-cpp backend to register a new depth-anything backend that wraps the Depth Anything 3 ggml port (depth-anything.cpp) via purego (cgo-less, no Python at inference). - backend/go/depth-anything-cpp/: gRPC backend (Load + Predict + GenerateImage), purego binding to the da_capi_* C ABI, CMake/Makefile/run/package/test scripts building depth-anything.cpp's DA_SHARED static .so per CPU variant. - backend/index.yaml: depth-anything backend meta + all hardware-variant capability entries (cpu/cuda12/cuda13/intel-sycl-f32+f16/vulkan/nvidia-l4t). - gallery/index.yaml: 8 Depth Anything 3 GGUF models (base q4_k/q8_0/f16/f32, small, large, giant, mono-large). - .github/backend-matrix.yml: one build entry per hardware variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): typed Depth RPC + REST endpoint exposing full DA3 data Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to e0b6814 (ABI 3 dense C-API) The Depth RPC handler calls da_capi_depth_dense / da_capi_points (C-API ABI 3); pin the native build to the commit that exports them. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): pin depth-anything.cpp to v0.1.0 release (b515c31) Repoint the native version from the now-orphaned e0b6814 to the b515c31 release commit, kept alive by the upstream v0.1.0 tag. C-API is unchanged (da_capi_abi_version == 3). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): wire depth-anything-cpp into build, CI bump, and importer The backend dir, gallery index, and CI build-matrix were present but the backend was never wired into the integration points that adding-backends.md requires: - root Makefile: add to .NOTPARALLEL, the test-extra chain, a BACKEND_* definition, the docker-build target eval, and docker-build-backends (mirrors parakeet-cpp; the backend's own Makefile already documented that its `test` target is driven by test-extra). - bump_deps.yaml: register the DEPTHANYTHING_VERSION pin so the daily auto-bump bot tracks mudler/depth-anything.cpp master (it cannot see an unregistered Makefile pin). - import form: add a preference-only KnownBackend entry so depth-anything is selectable at /import-model (mirrors sam3-cpp; no reliable GGUF auto-detect signal, so pref-only per the doc's default). changed-backends.js needs no entry: the generic golang suffix branch already resolves backend/go/depth-anything-cpp/. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(depth): auto-detect importer for depth-anything GGUFs Replace the preference-only entry with a real auto-detect importer (mirrors parakeet-cpp / locate-anything): - DepthAnythingImporter matches a .gguf whose name carries a depth-anything token (depth-anything-<size>-<quant>.gguf), so /import-model recognises mudler/depth-anything.cpp-gguf repos and direct GGUF URLs without an explicit backend preference. preferences.backend= "depth-anything" still forces it. - Registered before LlamaCPPImporter so its GGUF bundles aren't claimed by the generic .gguf importer; the narrow name match means it cannot claim arbitrary llama GGUFs or the upstream safetensors PyTorch repos. - Multi-quant repos pick the smallest quant by default (q4_k -> ... -> f32, depth stays >0.998 corr even at q4_k); quantizations preference overrides. - Drops the now-redundant knownPrefOnlyBackends entry (importer-backed backends are not listed there, matching parakeet-cpp). - Table-driven Ginkgo test covers detection, negative cases (llama GGUF, upstream safetensors), default/override/fallback quant pick, and direct URL import. 10/10 specs pass. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(depth): check conn.Close error in grpc Depth client (errcheck) The new Depth() client method used a bare `defer conn.Close()`. golangci-lint runs with new-from-merge-base, so although the 39 sibling methods use the same bare form (grandfathered), the newly added line trips errcheck. Drop the result explicitly to satisfy the linter. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.1 (embeddable CMake) v0.1.0 (b515c31) used ${CMAKE_SOURCE_DIR} for its include dirs, which points at the parent project when built via add_subdirectory() as this backend does, so the container build failed with missing stb_image.h / da_gguf_keys.h. v0.1.1 (2d42897) switches to project-relative paths. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): resolve gosec findings in the backend wrapper The code-scanning gate flagged three new failure-level alerts in godepthanythingcpp.go (gosec runs with -no-fail; GitHub gates on new alerts): - G301: export dirs were created with 0o755. Tighten to 0o750 (no world access needed for backend-written export output). - G304: writeDepthPNG creates req.GetDst(). That path is chosen by the LocalAI core as the intended output destination (same pattern every image backend uses), not attacker input, so annotate with #nosec G304 and document why. The remaining G103 "audit unsafe" notes on the unsafe.Slice C-buffer copies are warning-level (the same purego interop whisper/parakeet use) and do not gate the check, per the supertonic exclusion precedent in secscan.yaml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 * fix(depth): bump depth-anything.cpp to v0.1.2 (CUDA cross-build arch) v0.1.1 forced CMAKE_CUDA_ARCHITECTURES=native, which breaks the GPU-less l4t/cublas CI builds (nvcc "Unsupported gpu architecture 'compute_'" on CMake 3.22). v0.1.2 (442eea4) drops the override and lets ggml pick its default cross-build arch list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-16 16:28:28 +02:00
Richard Palethorpe	085fc53bbc	fix(router): production-ready request router + auto-size batch for embedding/rerank (#10104) ... * fix(router): score classifier production-readiness Conversation trimming runs through the classifier model's chat template and trims by exact token count, sized to the model's n_batch which is now scaled to context so long probes can't crash the backend. Missing chat_message templates are a hard error at router build time. Router- facing factories (Embedder/Scorer/Reranker/TokenCounter) re-resolve ModelConfig per call so a model installed post-startup doesn't bind a stub Backend="" config and silently fall into the loader's auto- iterate path. New 'vector_store' backend trace recorded inside localVectorStore on every Search/Insert — including the backend-load-failure path that previously vanished into an xlog.Warn — with outcome tagging (hit/miss/empty_store/backend_load_error/find_error/insert_error/ok). Companion cleanup drops misleading similarity:0 and input_tokens_count:0 from non-hit and text-mode traces. Gallery local-store-development aliases to 'local-store' so the master image satisfies pkg/model.LocalStoreBackend lookups from the embedding cache. Misc: llama-cpp TokenizeString reads the correct 'prompt' JSON key (the original bug); ModelTokenize nil-guard; non-fatal mitm proxy startup; PII 'route_local' renamed to 'allow' with docs/UI in sync; model-editor footer no longer eats the edit area on small screens; several config-editor template/dropdown/section fixes. Tests: e2e router specs (casual/code-hint + long-conversation trim), vector_store trace specs, lazy-factory specs, gallery dev-alias resolution, Playwright trace badge + scroll regression. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(backend): auto-size batch to context for embedding and rerank models Embedding and rerank models pool over the whole input in a single physical batch (n_ubatch). With batch left at the 512 default, the backend rejects longer inputs with "input is too large to process", silently capping a large-context embedder (e.g. 8k/32k) at 512 tokens. Size n_batch to the context for these single-pass usecases, mirroring the existing FLAG_SCORE behaviour; an explicit batch: still wins. Extracts EffectiveContextSize/EffectiveBatchSize from grpcModelOpts so the effective decode window has one home for other callers to reuse. Adds an e2e-aio regression test that embeds a >512-token input. The AIO embedding model is switched to nomic-embed-text-v1.5 (2048 context) because the previous granite model was capped at 512 tokens and could not exercise the larger batch. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(gallery): raise arch-router scoring output cap via parallel:64 Scoring decodes the whole prompt+candidate in a single llama_decode and reads one logit row per candidate token. The vendored llama.cpp server caps causal output rows at n_parallel, so the default of 1 aborts with GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) on multi-token route labels. Set options: [parallel:64] on both arch-router quant entries to lift the cap; kv_unified (the grpc-server default) keeps the full context per sequence, so this does not split the KV cache. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-12 16:21:15 +02:00
LocalAI [bot]	27e63b9a78	feat(tts): support per-request instructions and params (#10172) ... The OpenAI-compatible TTS endpoint accepts an `instructions` field, but it was silently dropped at the HTTP->gRPC boundary: neither schema.TTSRequest nor the gRPC TTSRequest proto carried it, so backends could only read such a value from static YAML options (identical for every request). This blocked per-line emotion/style and, for Qwen3-TTS VoiceDesign, limited a model config to a single designed voice. Plumb a generic per-request instruction string end to end, plus an optional backend-specific params map: - proto: add `optional string instructions` and `map<string,string> params` to TTSRequest. - schema: add Instructions (maps OpenAI `instructions`) and Params (LocalAI extension) to schema.TTSRequest. - core: thread both through ModelTTS/ModelTTSStream via a newTTSRequest helper that attaches instructions only when non-empty (so backends can fall back to YAML when unset); forward them from the /v1/audio/speech handler. - qwen-tts: prefer the per-request instruction over the YAML `instruct` option (used by both mode detection and generation) and merge per-request params. - chatterbox: merge per-request params (coerced to float/int/bool) over YAML options into generate() kwargs. Fully backward compatible: empty instructions fall back to the YAML option and backends that don't support style/voice instructions ignore the field. Closes #10164 Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-04 11:45:02 +02:00
Tai An	0fd666ee6e	fix(openresponses): populate Content and accept bare {role,content} items (#10039) (#10040) ... * fix(openresponses): populate Content and accept bare {role,content} items (#10039) Fixes mudler/LocalAI#10039 — `/v1/responses` silently returned empty output on any model whose YAML doesn't include a Go-side `template.chat_message` block. Three cooperating bugs: * `convertORInputToMessages` populated only `StringContent` for string input and for the `input.Instructions` system message, leaving the `Content` (any) field nil. * `TemplateMessages` gated all fallback content-rendering branches on `Content != nil && StringContent != ""` — but every branch in that function consumes `StringContent`, not `Content`. The `&&` silently dropped messages that had StringContent set and Content nil, producing an empty prompt that the 5× empty-retry guard then turned into a 200 OK with `output: []`. * The array-input branch of `convertORInputToMessages` dispatched on `itemMap["type"]` with no default, dropping bare `{role, content}` items emitted by the OpenAI Python SDK helper `client.responses.create(input=[{...}])`. Fix: * Set both `Content` and `StringContent` in the two openresponses message-construction sites that only set one. * Treat a bare `{role, content}` item (no `type`) as `type: "message"` for OpenAI-SDK compatibility. * Gate `TemplateMessages` fallback rendering on `StringContent != ""`, which is what every downstream branch in that function actually reads. Regression test added to `evaluator_test.go` covering the fallback path (no `ChatMessage` template) with a StringContent-only message, both with and without a role mapping. * test(openresponses): guard Content population and ToProto path (#10039) Add regression tests for the two seams the original fix touched but left uncovered: * convertORInputToMessages must populate both Content and StringContent for plain string input and for bare {role, content} array items (the OpenAI SDK shape that omits the type discriminator). Both are functional reds against the pre-fix code. * Messages.ToProto reads Content, not StringContent — this is the path UseTokenizerTemplate backends (imported GGUFs) take. The cases pin that contract so a future regression on the producer side is caught. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-28 07:21:48 +00:00
Richard Palethorpe	6a80e23733	feat(middleware): Model routing, PII filtering, Cloud model proxies (#9802) ... Add a routing middleware stack and a cloud-proxy backend. * cloud-proxy: a Go gRPC backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers, with an optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response) and full tool-calling support. * routing: admission control, content-aware model routing (embedding cache + classifier + rerank + Arch-Router score), PII detection/redaction (regex + NER) with streaming filter and OpenAI/Anthropic adapters, and a per-user/per-key billing recorder backed by GORM or in-memory storage. * middleware: UsageMiddleware records usage via the billing recorder, plus admission, route-model, usage-stamp and trace middlewares. * observability: BackendTrace ring buffer stores full request bodies (capped), MITM proxy emits structured trace events, and router classifier decisions surface at /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0). * UI: cloud-proxy model-editor fields, classifier system-prompt and score-normalization config, and a Traces page rendering request bodies. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-25 09:28:27 +02:00
LocalAI [bot]	661a0c3b9d	fix(ollama): accept float-encoded integer options (fixes #9837) (#9849) ... fix(ollama): accept float-encoded integer options (num_ctx, top_k, ...) Home Assistant's Ollama integration encodes integer options as JSON floats (e.g. `"num_ctx": 8192.0`). Stdlib `json.Unmarshal` refuses to decode a number with fractional notation into an `int` field, so the entire request was rejected with HTTP 400 before reaching the backend: Unmarshal type error: expected=int, got=number 8192.0, field=options.num_ctx Add a custom `UnmarshalJSON` on `OllamaOptions` that routes the int fields (`top_k`, `num_predict`, `seed`, `repeat_last_n`, `num_ctx`) through `*json.Number`, then converts via `Int64()` with a `Float64()` fallback. Public field types are unchanged, so endpoint code is untouched. Float fields and `stop` continue to parse via the default path. Fixes #9837 Assisted-by: Claude Code:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-16 18:38:19 +02:00
LocalAI [bot]	8af963bdd9	fix(streaming): comply with OpenAI usage / stream_options spec (#9815) ... * fix(streaming): comply with OpenAI usage / stream_options spec (#8546) LocalAI emitted `"usage":{"prompt_tokens":0,...}` on every streamed chunk because `OpenAIResponse.Usage` was a value type without `omitempty`. The official OpenAI Node SDK and its consumers (continuedev/continue, Kilo Code, Roo Code, Zed, IntelliJ Continue) filter on a truthy `result.usage` to detect the trailing usage chunk; LocalAI's zero-but-non-null usage on every intermediate chunk made that filter swallow every content chunk and surface an empty chat response while the server log looked successful. Changes: - `core/schema/openai.go`: `Usage *OpenAIUsage \`json:"usage,omitempty"\`` so intermediate chunks no longer carry a `usage` key. Add `OpenAIRequest.StreamOptions` with `include_usage` to mirror OpenAI's request field. - `core/http/endpoints/openai/chat.go` and `completion.go`: keep using the `Usage` struct field as an in-process channel for the running cumulative, but strip it before JSON marshalling. When the request set `stream_options.include_usage: true`, emit a dedicated trailing chunk with `"choices": []` and the populated usage (matching the OpenAI spec and llama.cpp's server behavior). - `chat_emit.go`: new `streamUsageTrailerJSON` helper; drop the `usage` parameter from `buildNoActionFinalChunks` since chunks no longer carry usage. - Update `image.go`, `inpainting.go`, `edit.go` to wrap their Usage values with `&` for the new pointer field. - UI: send `stream_options:{include_usage:true}` from the React (`useChat.js`) and legacy (`static/chat.js`) chat clients so the token-count badge keeps populating now that the server is spec-compliant. Tests: - New `chat_stream_usage_test.go` pins the spec invariants: intermediate chunks have no `usage` key, the trailer JSON has `"choices":[]` and a populated `usage`, and `OpenAIRequest` parses `stream_options.include_usage`. - Update `chat_emit_test.go` to reflect that finals no longer embed usage. Verified against the live LocalAI instance: before the fix Continue's filter logic swallowed 16/16 token chunks; with the new shape it yields 4/5 and routes usage through the dedicated trailer chunk. Fixes #8546 Assisted-by: Claude:opus-4.7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(streaming): silence errcheck on usage trailer Fprintf The new spec-compliant `stream_options.include_usage` trailer writes were flagged by errcheck since they're new code (golangci-lint runs new-from-merge-base on master); the surrounding `fmt.Fprintf` data: writes are grandfathered. Drop the return values explicitly to match the linter's contract without adding a nolint shim. Assisted-by: Claude:opus-4.7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-14 08:53:46 +02:00
LocalAI [bot]	a57e73691d	fix(ollama): accept prompt alias on /api/embed for Ollama parity (#9780) ... Ollama's embedding endpoint accepts both `input` and `prompt` as the input string value (see ollama/ollama docs/api.md#generate-embeddings). LocalAI only accepted `input`, which broke client libraries that send the `prompt` form. Add `Prompt` to OllamaEmbedRequest and have GetInputStrings fall back to it when Input is unset. Input still wins when both are provided. Fixes #9767. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-12 17:21:20 +02:00
LocalAI [bot]	bc3fb16105	feat(ollama): report model capabilities + details on /api/tags and /api/show (#9766) ... Ollama-compatible clients (Open WebUI, Enchanted, ollama-grid-search, etc.) rely on the `capabilities` list and `details.{parameter_size, quantization_level,families}` fields returned by /api/tags and /api/show to decide which models are eligible for a given task -- for example to filter the "embedding model" picker. Upstream Ollama returns these; LocalAI's compat layer was leaving them empty, so embedding models were silently rejected by clients that only allow chat models for chat and only allow embedding models for embeddings. This wires up the existing config signals already present in ModelConfig: - modelCapabilities() derives the Ollama capability strings from the config: "embedding" (FLAG_EMBEDDINGS), "completion" (FLAG_CHAT / FLAG_COMPLETION), "vision" (explicit KnownUsecases bit or MMProj / multimodal template / backend media marker), "tools" (auto-detected ToolFormatMarkers, JSON/Response regex, XML format, grammar triggers), "thinking" (ReasoningConfig with reasoning not disabled) and "insert" (presence of a completion template). - modelDetailsFromModelConfig() now fills families, parameter_size and quantization_level. The latter two are parsed from the GGUF filename via regex -- conservative tokens only (Q*/IQ*/F16/F32/BF16 and \d+(\.\d+)?[BM] surrounded by separators) so we don't accidentally match "Qwen3" as "3B". - modelInfoFromModelConfig() exposes general.architecture and general.context_length in the new ShowResponse.model_info map. Note: HasUsecases(FLAG_VISION) cannot be used directly -- GuessUsecases has no FLAG_VISION case and returns true at the end for any chat model. hasVisionSupport() instead reads KnownUsecases explicitly plus MMProj / template / media-marker signals. Tests are written first (TDD) using Ginkgo/Gomega -- DescribeTable for the capability mapping (embedding-only, chat, vision, thinking, tools via markers, tools via JSON regex, no-capability rerank) plus integration tests against ShowModelEndpoint that round-trip JSON through a real ModelConfigLoader populated from a temp YAML file. Fixes #9760. Assisted-by: Claude Code:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-12 00:16:19 +02:00
Andreas Egli	af83518532	feat: support word-level timestamps for faster-whisper (#9621) ... Signed-off-by: Andreas Egli <github@kharan.ch> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2026-05-06 00:32:52 +02:00
Ettore Di Giacinto	e86ade54a6	feat(api): add /v1/audio/diarization endpoint with sherpa-onnx + vibevoice.cpp (#9654) ... * feat(api): add /v1/audio/diarization endpoint with sherpa-onnx + vibevoice.cpp Closes #1648. OpenAI-style multipart endpoint that returns "who spoke when". Single endpoint instead of the issue's three-endpoint sketch (refactor /vad, /vad/embedding, /diarization) — the typical client wants one call, and embeddings can land later as a sibling without breaking this surface. Response shape borrows from Pyannote/Deepgram: segments carry a normalised SPEAKER_NN id (zero-padded, stable across the response) plus the raw backend label, optional per-segment text when the backend bundles ASR, and a speakers summary in verbose_json. response_format also accepts rttm so consumers can pipe straight into pyannote.metrics / dscore. Backends: * vibevoice-cpp — Diarize() reuses the existing vv_capi_asr pass. vibevoice's ASR prompt asks the model to emit [{Start,End,Speaker,Content}] natively, so diarization is a by-product of the same pass; include_text=true preserves the transcript per segment, otherwise we drop it. * sherpa-onnx — wraps the upstream SherpaOnnxOfflineSpeakerDiarization C API (pyannote segmentation + speaker-embedding extractor + fast clustering). libsherpa-shim grew config builders, a SetClustering wrapper for per-call num_clusters/threshold overrides, and a segment_at accessor (purego can't read field arrays out of SherpaOnnxOfflineSpeakerDiarizationSegment[] directly). Plumbing: new Diarize gRPC RPC + DiarizeRequest / DiarizeSegment / DiarizeResponse messages, threaded through interface.go, base, server, client, embed. Default Base impl returns unimplemented. Capability surfaces all updated: FLAG_DIARIZATION usecase, FeatureAudioDiarization permission (default-on), RouteFeatureRegistry entries for /v1/audio/diarization and /audio/diarization, audio instruction-def description widened, CAP_DIARIZATION JS symbol, swagger regenerated, /api/instructions discovery map updated. Tests: * core/backend: speaker-label normalisation (first-seen → SPEAKER_NN, per-speaker totals, nil-safety, fallback to backend NumSpeakers when no segments). * core/http/endpoints/openai: RTTM rendering (file-id basename, negative duration clamping, fallback id). * tests/e2e: mock-backend grew a deterministic Diarize that emits raw labels "5","2","5" so the e2e suite verifies SPEAKER_NN remapping, verbose_json speakers summary + transcript pass-through (gated by include_text), RTTM bytes content-type, and rejection of unknown response_format. mock-diarize model config registered with known_usecases=[FLAG_DIARIZATION] to bypass the backend-name guard. Docs: new features/audio-diarization.md (request/response, RTTM example, sherpa-onnx + vibevoice setup), cross-link from audio-to-text.md, entry in whats-new.md. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * fix(diarization): correct sherpa-onnx symbol name + lint cleanup CI failures on #9654: * sherpa-onnx-grpc-{tts,transcription} and sherpa-onnx-realtime panicked at backend startup with `undefined symbol: SherpaOnnxDestroyOfflineSpeakerDiarizationResult`. Upstream's actual symbol is SherpaOnnxOfflineSpeakerDiarizationDestroyResult (Destroy in the middle, not the prefix); the rest of the diarization surface follows the same naming pattern. The mismatched name made purego.RegisterLibFunc fail at dlopen time and crashed the gRPC server before the BeforeAll could probe Health, taking down every sherpa-onnx test job — not just the diarization-related ones. * golangci-lint flagged 5 errcheck violations on new defer cleanups (os.RemoveAll / Close / conn.Close); wrap each in a `defer func() { _ = X() }()` closure (matches the pattern other LocalAI files use for new code, since pre-existing bare defers are grandfathered in via new-from-merge-base). * golangci-lint also flagged forbidigo violations: the new diarization_test.go files used testing.T-style `t.Errorf` / `t.Fatalf`, which are forbidden by the project's coding-style policy (.agents/coding-style.md). Convert both files to Ginkgo/Gomega Describe/It with Expect(...) — they get picked up by the existing TestBackend / TestOpenAI suites, no new suite plumbing needed. * modernize linter: tightened the diarization segment loop to `for i := range int(numSegments)` (Go 1.22+ idiom). Verified locally: golangci-lint with new-from-merge-base=origin/master reports 0 issues across all touched packages, and the four mocked diarization e2e specs in tests/e2e/mock_backend_test.go still pass. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * fix(vibevoice-cpp): convert non-WAV input via ffmpeg + raise ASR token budget Confirmed end-to-end against a real LocalAI instance with vibevoice-asr-q4_k loaded and the multi-speaker MP3 sample at vibevoice.cpp/samples/2p_argument.mp3: both /v1/audio/transcriptions and /v1/audio/diarization now succeed and return correctly attributed speaker turns for the full clip. Two latent issues surfaced once the diarization endpoint actually exercised the backend with a non-trivial input: 1. vv_capi_asr only accepts WAV via load_wav_24k_mono. The previous code passed the uploaded path straight through, so anything that wasn't already a 24 kHz mono s16le WAV failed at the C side with rc=-8 and the very unhelpful "vv_capi_asr failed". prepareWavInput shells out to ffmpeg ("-ar 24000 -ac 1 -acodec pcm_s16le") in a per-call temp dir, matching the rate the model was trained on; both AudioTranscription and Diarize now route through it. This is the same shape sherpa-onnx uses (utils.AudioToWav), but vibevoice needs 24 kHz rather than 16 kHz so we don't reuse that helper. 2. The C ABI's max_new_tokens defaults to 256 when 0 is passed. That's fine for a five-second clip but not for anything past ~10 s — vibevoice stops mid-JSON, the parse fails, and the caller sees a hard error. Pass a much larger budget (16 384 ≈ ~9 minutes of speech at the model's ~30 tok/s rate); generation stops at EOS so this is a cap rather than a target. 3. As a defensive belt-and-braces, mirror AudioTranscription's existing "fall back to a single segment if the model emits non-JSON text" pattern in Diarize, so partial / unusual model output never produces a 500. This kept the endpoint usable while diagnosing (1) and (2), and is the right behaviour to keep. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * fix(vibevoice-cpp): pass valid WAVs through directly so ffmpeg is not required at runtime Spotted by tests-e2e-backend (1.25.x): the previous fix forced every incoming audio file through `ffmpeg -ar 24000 ...`, which meant the backend container — which does not ship ffmpeg — failed even for the existing happy path where the caller already uploads a WAV. The container-side error was: rpc error: code = Unknown desc = vibevoice-cpp: ffmpeg convert to 24k mono wav: exec: "ffmpeg": executable file not found in $PATH Reading vibevoice.cpp's audio_io.cpp, `load_wav_24k_mono` uses drwav and already accepts any PCM/IEEE-float WAV at any sample rate, downmixes multi-channel input to mono, and resamples to 24 kHz internally. So the only inputs that genuinely need an external converter are non-WAV formats (MP3, OGG, FLAC, ...). Detect WAVs by RIFF/WAVE magic at bytes 0..3 / 8..11 and pass them straight through with a no-op cleanup; everything else still goes through ffmpeg with the same 24 kHz mono s16le target. The result: * Container builds without ffmpeg keep working for WAV uploads (the e2e-backends fixture is jfk.wav at 16 kHz mono s16le). * MP3 and other non-WAV inputs still get the new ffmpeg conversion path so the diarization endpoint stays useful. * If the caller uploads a non-WAV but ffmpeg isn't on PATH, the surfaced error is still descriptive enough to act on. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * fix(ci): make gcc-14 install in Dockerfile.golang best-effort for jammy bases The LocalVQE PR (bb033b16) made `gcc-14 g++-14` an unconditional apt install in backend/Dockerfile.golang and pointed update-alternatives at them. That works on the default `BASE_IMAGE=ubuntu:24.04` (noble has gcc-14 in main), but every Go backend that builds on `nvcr.io/nvidia/l4t-jetpack:r36.4.0` — jammy under the hood — now fails at the apt step: E: Unable to locate package gcc-14 This blocked unrelated jobs: backend-jobs(*-nvidia-l4t-arm64-{stablediffusion-ggml, sam3-cpp, whisper, acestep-cpp, qwen3-tts-cpp, vibevoice-cpp}). LocalVQE itself is only matrix-built on ubuntu:24.04 (CPU + Vulkan), so it doesn't actually need gcc-14 anywhere else. Make the gcc-14 install conditional on the package being available in the configured apt repos. On noble: identical behaviour to today (gcc-14 installed, update-alternatives points at it). On jammy: skip the gcc-14 stanza entirely and let build-essential's default gcc take over, which is what the other Go backends compile with anyway. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-05 15:10:13 +02:00
Richard Palethorpe	bb033b16a9	feat: add LocalVQE backend and audio transformations UI (#9640) ... feat(audio-transform): add LocalVQE backend, bidi gRPC RPC, Studio UI Introduce a generic "audio transform" capability for any audio-in / audio-out operation (echo cancellation, noise suppression, dereverberation, voice conversion, etc.) and ship LocalVQE as the first backend implementation. Backend protocol: - Two new gRPC RPCs in backend.proto: unary AudioTransform for batch and bidirectional AudioTransformStream for low-latency frame-by-frame use. This is the first bidi stream in the proto; per-frame unary at LocalVQE's 16 ms hop would be RTT-bound. Wire it through pkg/grpc/{client,server, embed,interface,base} with paired-channel ergonomics. LocalVQE backend (backend/go/localvqe/): - Go-Purego wrapper around upstream liblocalvqe.so. CMake builds the upstream shared lib + its libggml-cpu-*.so runtime variants directly — no MODULE wrapper needed because LocalVQE handles CPU feature selection internally via GGML_BACKEND_DL. - Sets GGML_NTHREADS from opts.Threads (or runtime.NumCPU()-1) — without it LocalVQE runs single-threaded at ~1× realtime instead of the documented ~9.6×. - Reference-length policy: zero-pad short refs, truncate long ones (the trailing portion can't have leaked into a mic that wasn't recording). - Ginkgo test suite (9 always-on specs + 2 model-gated). HTTP layer: - POST /audio/transformations (alias /audio/transform): multipart batch endpoint, accepts audio + optional reference + params[*]=v form fields. Persists inputs alongside the output in GeneratedContentDir/audio so the React UI history can replay past (audio, reference, output) triples. - GET /audio/transformations/stream: WebSocket bidi, 16 ms PCM frames (interleaved stereo mic+ref in, mono out). JSON session.update envelope for config; constants hoisted in core/schema/audio_transform.go. - ffmpeg-based input normalisation to 16 kHz mono s16 WAV via the existing utils.AudioToWav (with passthrough fast-path), so the user can upload any format / rate without seeing the model's strict 16 kHz constraint. - BackendTraceAudioTransform integration so /api/backend-traces and the Traces UI light up with audio_snippet base64 and timing. - Routes registered under routes/localai.go (LocalAI extension; OpenAI has no /audio/transformations endpoint), traced via TraceMiddleware. Auth + capability + importer: - FLAG_AUDIO_TRANSFORM (model_config.go), FeatureAudioTransform (default-on, in APIFeatures), three RouteFeatureRegistry rows. - localvqe added to knownPrefOnlyBackends with modality "audio-transform". - Gallery entry localvqe-v1-1.3m (sha256-pinned, hosted on huggingface.co/LocalAI-io/LocalVQE). React UI: - New /app/transform page surfaced via a dedicated "Enhance" sidebar section (sibling of Tools / Biometrics) — the page is enhancement, not generation, so it lives outside Studio. Two AudioInput components (Upload + Record tabs, drag-drop, mic capture). - Echo-test button: records mic while playing the loaded reference through the speakers — the mic naturally picks up speaker bleed, giving a real (mic, ref) pair for AEC testing without leaving the UI. - Reusable WaveformPlayer (canvas peaks + click-to-seek + audio controls) and useAudioPeaks hook (shared module-scoped AudioContext to avoid hitting browser context limits with three players on one page); migrated TTS, Sound, Traces audio blocks to use it. - Past runs saved in localStorage via useMediaHistory('audio-transform') — the history entry stores all three URLs so clicking re-renders the full triple, not just the output. Build + e2e: - 11 matrix entries removed from .github/workflows/backend.yml (CUDA, ROCm, SYCL, Metal, L4T): upstream supports only CPU + Vulkan, so we ship those two and let GPU-class hardware route through Vulkan in the gallery capabilities map. - tests-localvqe-grpc-transform job in test-extra.yml (gated on detect-changes.outputs.localvqe). - New audio_transform capability + 4 specs in tests/e2e-backends. - Playwright spec suite in core/http/react-ui/e2e/audio-transform.spec.js (8 specs covering tabs, file upload, multipart shape, history, errors). Docs: - New docs/content/features/audio-transform.md covering the (audio, reference) mental model, batch + WebSocket wire formats, LocalVQE param keys, and a YAML config example. Cross-links from text-to-audio and audio-to-text feature pages. Assisted-by: Claude:claude-opus-4-7 [Bash Read Edit Write Agent TaskCreate] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-04 22:07:11 +02:00
Ettore Di Giacinto	f5eb13d3c2	feat(insightface): add antispoofing (liveness) detection (#9515) ... * feat(insightface): add antispoofing (liveness) detection Light up the anti_spoofing flag that was parked during the first pass. Both FaceVerify and FaceAnalyze now run the Silent-Face MiniFASNetV2 + MiniFASNetV1SE ensemble (~4 MB, Apache 2.0, CPU <10ms) when the flag is set. Failed liveness on either image vetoes FaceVerify regardless of embedding similarity. Every insightface* gallery entry now ships the MiniFASNet ONNX weights so existing packs light up after reinstall. Setting the flag against a model without the MiniFASNet files returns FAILED_PRECONDITION (HTTP 412) with a clear install message — no silent is_real=false. FaceVerifyResponse gained per-image img{1,2}_is_real and img{1,2}_antispoof_score (proto 9-12); FaceAnalysis's existing is_real/antispoof_score fields are now populated. Schema fields are pointers so they are fully absent from the JSON response when anti_spoofing was not requested — avoids collapsing "not checked" with "checked and fake" under Go's omitempty on bool. Validated end-to-end over HTTP against a local install: - verify + anti_spoofing, both real -> verified=true, score ~0.76 - verify + anti_spoofing, img2 spoof -> verified=false, img2_is_real=false - analyze + anti_spoofing -> is_real and score per face - flag against model without MiniFASNet -> HTTP 412 fail-loud Assisted-by: Claude:claude-opus-4-7 go vet * test(insightface): wire test target into test-extra The root Makefile's `test-extra` already runs `$(MAKE) -C backend/python/insightface test`, but the backend's Makefile never defined the target — so the command silently errored and the suite was never executed in CI. Adding the two-line target (matching ace-step/Makefile) hooks `test.sh` → `runUnittests` → `python -m unittest test.py`, which discovers both the pre-existing engine classes (InsightFaceEngineTest, OnnxDirectEngineTest) and the new AntispoofingTest. Each class skips gracefully when its weights can't be downloaded from a network-restricted runner. Assisted-by: Claude:claude-opus-4-7 * test(insightface): exercise antispoofing in e2e-backends (both paths) Add a `face_antispoof` capability to the Ginkgo e2e suite and extend the existing FaceVerify + FaceAnalyze specs with liveness assertions covering BOTH paths: real fixture -> is_real=true, score>0, verified stays true spoof fixture -> is_real=false, verified vetoed to false The spoof fixture is upstream's own `image_F2.jpg` (via the yakhyo mirror) — verified locally against the MiniFASNetV2+V1SE ensemble to classify as is_real=false with score ~0.013. That makes the assertion deterministic across CI runs; synthetic/derived spoofs fool the model unpredictably and would be flaky. Makefile wires it up end-to-end: - New INSIGHTFACE_ANTISPOOF_* cache dir + two ONNX downloads with pinned SHAs, matching the gallery entries. - insightface-antispoof-models target shared by both backend configs. - FACE_SPOOF_IMAGE_URL passed via BACKEND_TEST_FACE_SPOOF_IMAGE_URL. - Both e2e targets (buffalo-sc + opencv) now: * depend on insightface-antispoof-models * pass antispoof_v2_onnx / antispoof_v1se_onnx in BACKEND_TEST_OPTIONS * include face_antispoof in BACKEND_TEST_CAPS backend_test.go adds the new capability constant and a faceSpoofFile fixture resolved the same way as faceFile1/2/3. Spoof assertions are gated on both capFaceAntispoof AND faceSpoofFile being set, so a test config that omits the spoof fixture degrades gracefully to "real path only" instead of failing. Assisted-by: Claude:claude-opus-4-7 go vet	2026-04-23 18:28:15 +02:00
Ettore Di Giacinto	181ebb6df4	feat: voice recognition (#9500) ... * feat(voice-recognition): add /v1/voice/{verify,analyze,embed} + speaker-recognition backend Audio analog to face recognition. Adds three gRPC RPCs (VoiceVerify / VoiceAnalyze / VoiceEmbed), their Go service and HTTP layers, a new FLAG_SPEAKER_RECOGNITION capability flag, and a Python backend scaffold under backend/python/speaker-recognition/ wrapping SpeechBrain ECAPA-TDNN with a parallel OnnxDirectEngine for WeSpeaker / 3D-Speaker ONNX exports. The kokoros Rust backend gets matching unimplemented trait stubs — tonic's async_trait has no defaults, so adding an RPC without Rust stubs breaks the build (same regression fixed by eb01c772 for face). Swagger, /api/instructions, and the auth RouteFeatureRegistry / APIFeatures list are updated so the endpoints surface everywhere a client or admin UI looks. Assisted-by: Claude:claude-opus-4-7 * feat(voice-recognition): add 1:N identify + register/forget endpoints Mirrors the face-recognition register/identify/forget surface. New package core/services/voicerecognition/ carries a Registry interface and a local-store-backed implementation (same in-memory vector-store plumbing facerecognition uses, separate instance so the embedding spaces stay isolated). Handlers under /v1/voice/{register,identify,forget} reuse backend.VoiceEmbed to compute the probe vector, then delegate the nearest-neighbour search to the registry. Default cosine-distance threshold is tuned for ECAPA-TDNN on VoxCeleb (0.25, EER ~1.9%). As with the face registry, the current backing is in-memory only — a pgvector implementation is a future constructor-level swap. Assisted-by: Claude:claude-opus-4-7 * feat(voice-recognition): gallery, docs, CI and e2e coverage - backend/index.yaml: speaker-recognition backend entry + CPU and CUDA-12 image variants (plus matching development variants). - gallery/index.yaml: speechbrain-ecapa-tdnn (default) and wespeaker-resnet34 model entries. The WeSpeaker SHA-256 is a deliberate placeholder — the HF URI must be curl'd and its hash filled in before the entry installs. - docs/content/features/voice-recognition.md: API reference + quickstart, mirrors the face-recognition docs. - React UI: CAP_SPEAKER_RECOGNITION flag export (consumers follow face's precedent — no dedicated tab yet). - tests/e2e-backends: voice_embed / voice_verify / voice_analyze specs. Helper resolveFaceFixture is reused as-is — the only thing face/voice share is "download a file into workDir", so no need for a new helper. - Makefile: docker-build-speaker-recognition + test-extra-backend- speaker-recognition-{ecapa,all} targets. Audio fixtures default to VCTK p225/p226 samples from HuggingFace. - CI: test-extra.yml grows a tests-speaker-recognition-grpc job mirroring insightface. backend.yml matrix gains CPU + CUDA-12 image build entries — scripts/changed-backends.js auto-picks these up. Assisted-by: Claude:claude-opus-4-7 * feat(voice-recognition): wire a working /v1/voice/analyze head Adds AnalysisHead: a lazy-loading age / gender / emotion inference wrapper that plugs into both SpeechBrainEngine and OnnxDirectEngine. Defaults to two open-licence HuggingFace checkpoints: - audeering/wav2vec2-large-robust-24-ft-age-gender (Apache 2.0) — age regression + 3-way gender (female / male / child). - superb/wav2vec2-base-superb-er (Apache 2.0) — 4-way emotion. Both are optional and degrade gracefully when transformers or the model can't be loaded — the engine raises NotImplementedError so the gRPC layer returns 501 instead of a generic 500. Emotion classes pass through from the model (neutral/happy/angry/sad on the default checkpoint); the e2e test now accepts any non-empty dominant gender so custom age_gender_model overrides don't fail it. Adds transformers to the backend's CPU and CUDA-12 requirements. Assisted-by: Claude:claude-opus-4-7 * fix(voice-recognition): pin real WeSpeaker ResNet34 ONNX SHA-256 Replaces the placeholder hash in gallery/index.yaml with the actual SHA-256 (7bb2f06e…) of the upstream Wespeaker/wespeaker-voxceleb-resnet34-LM ONNX at ~25MB. `local-ai models install wespeaker-resnet34` now succeeds. Assisted-by: Claude:claude-opus-4-7 * fix(voice-recognition): soundfile loader + honest analyze default Two issues surfaced on first end-to-end smoke with the actual backend image: 1. torchaudio.load in torchaudio 2.8+ requires the torchcodec package for audio decoding. Switch SpeechBrainEngine._load_waveform to the already-present soundfile (listed in requirements.txt) plus a numpy linear resample to 16kHz. Drops a heavy ffmpeg-linked dep and the codepath we never exercise (torchaudio's ffmpeg backend). 2. The AnalysisHead was defaulting to audeering/wav2vec2-large-robust- 24-ft-age-gender, but AutoModelForAudioClassification silently mangles that checkpoint — it reports the age head weights as UNEXPECTED and re-initialises the classifier head with random values, so the "gender" output is noise and there is no age output at all. Make age/gender opt-in instead (empty default; users wire a cleanly-loadable Wav2Vec2ForSequenceClassification checkpoint via age_gender_model: option). Emotion keeps its working Superb default. Also broaden _infer_age_gender's tensor-shape handling and catch runtime exceptions so a dodgy age/gender head never takes down the whole analyze call. Docs and README updated to match the new policy. Verified with the branch-scoped gallery on localhost: - voice/embed → 192-d ECAPA-TDNN vector - voice/verify → same-clip dist≈6e-08 verified=true; cross-speaker dist 0.76–0.99 verified=false (as expected) - voice/register/identify/forget → round-trip works, 404 on unknown id - voice/analyze → emotion populated, age/gender omitted (opt-in) Assisted-by: Claude:claude-opus-4-7 * fix(voice-recognition): real CI audio fixtures + fixture-agnostic verify spec Two issues surfaced after CI actually ran the speaker-recognition e2e target (I'd curl-tested against a running server but hadn't run the make target locally): 1. The default BACKEND_TEST_VOICE_AUDIO_* URLs pointed at huggingface.co/datasets/CSTR-Edinburgh/vctk paths that return 404 (the dataset is gated). Swap them for the speechbrain test samples served from github.com/speechbrain/speechbrain/raw/develop/ — public, no auth, correct 16kHz mono format. 2. The VoiceVerify spec required d(file1,file2) < 0.4, assuming file1/file2 were same-speaker. The speechbrain samples are three different speakers (example1/2/5), and there is no easy un-gated source of true same-speaker audio pairs (VoxCeleb/VCTK/LibriSpeech are all license- or size-gated for CI use). Replace the ceiling check with a relative-ordering assertion: d(pair) > d(same-clip) for both file2 and file3 — that's enough to prove the embeddings encode speaker info, and it works with any three non-identical clips. Actual speaker ordering d(1,2) vs d(1,3) is logged but not asserted. Local run: 4/4 voice specs pass (Health, LoadModel, VoiceEmbed, VoiceVerify) on the built backend image. 12 non-voice specs skipped as expected. Assisted-by: Claude:claude-opus-4-7 * fix(ci): checkout with submodules in the reusable backend_build workflow The kokoros Rust backend build fails with failed to read .../sources/Kokoros/kokoros/Cargo.toml: No such file because the reusable backend_build.yml workflow's actions/checkout step was missing `submodules: true`. Dockerfile.rust does `COPY . /LocalAI`, and without the submodule files the subsequent `cargo build` can't find the vendored Kokoros crate. The bug pre-dates this PR — scripts/changed-backends.js only triggers the kokoros image job when something under backend/rust/kokoros or the shared proto changes, so master had been coasting past it. The voice-recognition proto addition re-broke it. Other checkouts in backend.yml (llama-cpp-darwin) and test-extra.yml (insightface, kokoros, speaker-recognition) already pass `submodules: true`; this brings the shared backend image builder in line. Assisted-by: Claude:claude-opus-4-7	2026-04-23 12:07:14 +02:00
Ettore Di Giacinto	f0c92610a1	feat(importer): expand importer flow to almost all backends (#9466) ... * docs(agents): require importer integration when adding backends Document the importer registry workflow so contributors know that adding a new backend also requires updating the /import-model dropdown source: either a new importer in core/gallery/importers/, extending an existing one for drop-in replacements, or the pref-only slice for backends with no reliable auto-detect signal. Always covered by a table-driven test. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for Batch 0 primitives Introduce failing tests that drive Batch 0 of the importer expansion: - pkg/huggingface-api: assert GetModelDetails populates PipelineTag and LibraryName from /api/models/{repo}, and that a failing metadata endpoint still returns file details (best-effort fetch). - core/gallery/importers/helpers_test.go: new table-driven coverage for HasFile, HasExtension, HasONNX, HasONNXConfigPair, HasGGMLFile. - core/gallery/importers/importers_test.go: assert ErrAmbiguousImport sentinel exists and round-trips through errors.Is. - core/gallery/importers/local_test.go: extend with detection cases for ggml-*.bin (whisper), silero_vad.onnx (silero-vad), and the piper .onnx + .onnx.json pair. - core/http/endpoints/localai/import_model_test.go: assert ImportModelURIEndpoint returns HTTP 400 with a structured {error, detail, hint} body when ErrAmbiguousImport surfaces. All tests fail in the expected places (missing fields, missing helpers, missing sentinel, endpoint still wraps as 500). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): Batch 0 foundation — helpers, sentinel, local detection Implements the Batch 0 primitives that subsequent importer batches build on: - pkg/huggingface-api: ModelDetails gains PipelineTag and LibraryName. GetModelDetails now layers a best-effort GET /api/models/{repo} fetch on top of ListFiles — a metadata outage leaves the fields empty but still returns full file details. Uses a dedicated response struct because the single-model endpoint uses snake_case keys while the list endpoint historically returned camelCase. - core/gallery/importers/helpers.go: generic HasFile, HasExtension, HasONNX, HasONNXConfigPair, HasGGMLFile helpers working on []hfapi.ModelFile so per-backend importers can detect artefact patterns without duplicating string wrangling. - core/gallery/importers/importers.go: adds the ErrAmbiguousImport sentinel. DiscoverModelConfig now returns it (wrapped with fmt.Errorf("%w: ...")) when no importer matched AND the HF pipeline_tag falls in a whitelist of narrow modalities (ASR, TTS, sentence-similarity, text-classification, object-detection). The whitelist is intentionally narrow — unknown tags keep the previous "no importer matched" behaviour to avoid blocking rare repos. - core/gallery/importers/local.go: three new local-path detections, inserted before the existing merged-transformers branch: * ggml-*.bin → whisper * silero*.onnx → silero-vad * *.onnx + *.onnx.json pair → piper - core/http/endpoints/localai/import_model.go: ImportModelURIEndpoint surfaces ErrAmbiguousImport as HTTP 400 with {error, detail, hint} JSON, preserving existing behaviour for unrelated errors. Green tests: go test ./core/gallery/importers/... ./pkg/huggingface-api/... \ ./core/http/endpoints/localai/... Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(importers): red tests for KnownBackend endpoint and importer metadata Add failing tests that drive Batch UI-Dropdown: - importers_test.go: assert importers expose Name/Modality/AutoDetects and that LlamaCPPImporter advertises drop-in replacements via a new AdditionalBackendsProvider interface. A Registry() accessor is also expected. - backend_test.go (new): assert GET /backends/known returns []schema.KnownBackend, covers every importer, exposes drop-in llama-cpp replacements, includes curated pref-only backends, has no duplicates, and is sorted by Modality+Name. These tests fail at compile time against master; they are intentionally red so the follow-up green commit is reviewable. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery): add /backends/known endpoint for importer-aware backend list Extend the Importer interface with Name/Modality/AutoDetects so the import system can self-describe its registry, and introduce the AdditionalBackendsProvider interface so importers can advertise drop-in replacements (llama-cpp advertises ik-llama-cpp and turboquant). Expose the new GET /backends/known endpoint that merges: - the importer registry (auto-detect supported), - drop-in replacements hosted by importers (preference-only), - a curated knownPrefOnlyBackends slice for backends with no dedicated importer (sglang, tinygrad, trl, mlx-vlm, whisperx, kokoros, Qwen TTS variants, sam3-cpp) — kept at the top of backend.go so contributors adding a new pref-only backend have one obvious place to edit, - backends installed on disk but unknown to the importer (marked AutoDetect=false, empty Modality). The endpoint deliberately does NOT filter by gallery membership or host capability (unlike /backends/available): LocalAI may auto-install a backend that is not yet present, so the import form dropdown must show everything the importer knows about. Response is deduplicated (importer wins over pref-only) and sorted by Modality+Name for deterministic output. Registered in core/http/routes/localai.go next to /backends/available under the same admin middleware. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui): source import form backend dropdown from /backends/known Replace the hard-coded BACKENDS constant in ImportModel.jsx with a live fetch of /backends/known on mount. Users now see every backend the importer layer knows about (including preference-only entries) grouped by modality, not a stale subset. Changes: - config.js: add backendsKnown endpoint constant next to backendsAvailable. - api.js: add backendsApi.listKnown() wrapper. - ImportModel.jsx: remove BACKENDS constant, fetch the list via useEffect, and derive grouped options via buildBackendOptions. Preference-only entries render with a " (preference-only)" suffix. Loading state disables the dropdown with a "Loading backends…" placeholder; on fetch failure the form falls back to auto-detect only and surfaces a non-blocking toast. - SearchableSelect.jsx: accept items flagged isHeader=true and render them as non-selectable section dividers. Keyboard navigation skips headers and search queries hide them so filtered output stays relevant. Vitest is not set up in this project (devDependencies ship Playwright only). Per the brief's guard-rail, no frontend test framework is introduced; coverage is provided by the Go handler tests that assert the /backends/known contract consumed by the React form. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for whisper importer Asserts detection on ggerganov/whisper.cpp (via ggml-*.bin filename), the preferences.backend=whisper override path for arbitrary URIs, and the Importer interface metadata (name/modality/autodetect). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add whisper importer Recognises whisper.cpp GGML models by the "ggml-*.bin" filename convention (direct URL or HF repo member) and by the explicit preferences.backend="whisper" override. Emits backend: whisper with the transcript use-case. Registered before llama-cpp so the narrow filename signal wins before any generic GGUF match is attempted. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for moonshine importer Asserts detection on UsefulSensors/moonshine-tiny via owner + ONNX files, the preferences.backend=moonshine override for arbitrary URIs, and the Importer interface metadata (name/modality/autodetect). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add moonshine importer Matches UsefulSensors-owned HF repos whose artefacts or metadata identify them as ASR: on-disk .onnx files (the canonical Moonshine packaging) OR pipeline_tag=automatic-speech-recognition (covers transformers/safetensors-only sibling repos). preferences.backend= moonshine overrides detection. Test uses the live moonshine-tiny repo because the canonical UsefulSensors/moonshine repo currently hits a recursive-subfolder bug in pkg/huggingface-api ListFiles. Registered after WhisperImporter but before LlamaCPPImporter and TransformersImporter so the narrower owner+ASR signal wins before the generic tokenizer.json check routes the repo to transformers. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for nemo importer Asserts detection on nvidia/parakeet-tdt-0.6b-v3 via owner + .nemo file, the preferences.backend=nemo override for arbitrary URIs, and the Importer interface metadata (name/modality/autodetect). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add nemo importer Matches nvidia-owned HF repos that ship a .nemo checkpoint archive, the canonical NeMo ASR packaging. preferences.backend=nemo forces detection. Registered between moonshine and llama-cpp so the narrow owner + extension signal wins before any downstream generic matcher. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for faster-whisper importer Asserts detection on Systran/faster-whisper-large-v3 (owner + model.bin + config.json + ASR pipeline), the preferences.backend= faster-whisper override for arbitrary URIs, and the Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add faster-whisper importer Recognises CTranslate2-packaged whisper checkpoints distributed for the faster-whisper runtime: model.bin + config.json + ASR pipeline_tag, narrowed to Systran-owned repos or repo names containing "faster-whisper" to avoid falsely claiming vanilla OpenAI whisper HF repos. preferences.backend=faster-whisper overrides detection. Registered before llama-cpp and transformers so the narrow signal wins before tokenizer.json routes the repo to the generic transformers importer. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for qwen-asr importer Asserts detection on Qwen/Qwen3-ASR-1.7B via owner + ASR substring in the repo name, the preferences.backend=qwen-asr override for arbitrary URIs, and the Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add qwen-asr importer Matches Qwen-owned HF repos whose name contains "ASR" (case-insensitive), routing them to the qwen-asr backend rather than the generic transformers/vllm path. The substring check scans the repo portion only so the owner field cannot leak a false match. preferences.backend=qwen-asr forces detection. Registered before llama-cpp and transformers so the narrow owner+name signal wins. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): ASR ambiguity surfaces ErrAmbiguousImport Locks in the behaviour added in Batch 0: an HF repo whose pipeline_tag marks it as automatic-speech-recognition but whose artefacts match no ASR importer (and no generic importer) must fail with ErrAmbiguousImport so callers know to pass preferences.backend rather than silently guess. pyannote/voice-activity-detection is the fixture — its file list is only config.yaml + README, leaving every importer's artefact check negative. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for piper importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add piper importer Detects piper TTS voices by the canonical <voice>.onnx + <voice>.onnx.json pair packaging (via HasONNXConfigPair). Narrow enough to skip generic ONNX repos used by other backends (Moonshine ASR, sentence-transformers). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for bark importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add bark importer Detects Suno's Bark TTS checkpoints by HF owner "suno" + repo name prefix "bark". Adds HFOwnerRepoFromURI() helper so importers can fall back to URI parsing when pkg/huggingface-api's recursive tree listing errors on repos with nested subdirectories (suno/bark ships a speaker_embeddings/v2 subtree that trips a pre-existing path-doubling bug in the listFilesInPath recursion). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for fish-speech importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add fish-speech importer Detects Fish Audio TTS releases by HF owner "fishaudio" with a URI-based fallback for repos whose tree recursion trips the pre-existing hfapi path-doubling bug. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for outetts importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add outetts importer Detects OuteAI's OuteTTS releases by HF owner "OuteAI" or a case- insensitive "OuteTTS" substring in the repo name, with a URI-based fallback for recursion-bugged repos. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for voxcpm importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add voxcpm importer Detects OpenBMB's VoxCPM TTS family by repo-name substring (community mirrors re-host the weights under many owners — mlx-community, bluryar, callgg, etc). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for kokoro importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add kokoro importer Detects hexgrad's Kokoro TTS by the "Kokoro" repo-name substring paired with a PyTorch .pth/.pt checkpoint — the pairing excludes ONNX-only mirrors (handled by the pref-only `kokoros` Rust runtime) and GGUF mirrors (handled by llama-cpp). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for kitten-tts importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add kitten-tts importer Detects KittenML's kitten-tts releases by owner or "kitten-tts" repo-name substring, with URI-parsing fallback. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for neutts importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add neutts importer Detects Neuphonic's NeuTTS releases by owner "neuphonic" or "neutts" repo-name substring, with URI-parsing fallback. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for chatterbox importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add chatterbox importer Detects Resemble AI's Chatterbox TTS by owner "ResembleAI" or "chatterbox" repo-name substring, with URI-parsing fallback. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for vibevoice importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add vibevoice importer Detects Microsoft's VibeVoice TTS by "vibevoice" repo-name substring (case-insensitive) so community mirrors still route here. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for coqui importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add coqui importer Detects Coqui AI's TTS releases (XTTS-v2, YourTTS, …) by the authoritative `coqui` HF owner, with URI-parsing fallback. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): TTS ambiguity surfaces ErrAmbiguousImport Adds a Ginkgo spec that imports nari-labs/Dia-1.6B — a real HF repo carrying pipeline_tag="text-to-speech" whose artefacts (*.pth, one safetensors shard, preprocessor_config.json, config.json) match none of the Batch-2 TTS importers nor the generic text/image importers — and asserts DiscoverModelConfig wraps ErrAmbiguousImport via errors.Is. Also pivots the endpoint-level ambiguity fixture from hexgrad/Kokoro-82M to nari-labs/Dia-1.6B. Batch 2 added a dedicated kokoro importer that now claims the original fixture; Dia remains genuinely unclaimed and so exercises the same ambiguity code path at the HTTP layer. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for stablediffusion-ggml importer Covers HF repo detection (city96/FLUX.1-dev-gguf), raw .gguf URL matching on filename arch tokens, preference override, and Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add stablediffusion-ggml importer Detects GGUF-packed Stable Diffusion and FLUX checkpoints (leejet owner, city96 FLUX mirrors, second-state SD dumps, raw .gguf URLs with arch tokens) and routes them to the stablediffusion-ggml backend. Registered BEFORE LlamaCPPImporter so .gguf image checkpoints are not stolen by llama-cpp's generic .gguf match. Reuses HFOwnerRepoFromURI for the hfapi-recursion-bug fallback. preferences.backend overrides detection. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for ace-step importer Covers HF repo-name detection (ACE-Step/ACE-Step-v1-3.5B), preference override, and Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add ace-step importer Routes ACE-Step music generation checkpoints (ACE-Step/ACE-Step-v1-3.5B, ACE-Step/Ace-Step1.5, community mirrors) to the ace-step backend. Matching is case-insensitive on the "ace-step" repo-name substring and owner, with an HFOwnerRepoFromURI fallback for the hfapi recursion bug. KnownUsecaseStrings mirrors the gallery's ace-step-turbo entry (sound_generation, tts). preferences.backend overrides. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): surface ErrAmbiguousImport on text-to-image misses Adds text-to-image to ambiguousModalities whitelist and covers the h94/IP-Adapter-FaceID case — pipeline_tag=text-to-image but ships only .bin/.safetensors so diffusers, stablediffusion-ggml, llama-cpp, transformers, vllm, mlx, and ace-step all miss. DiscoverModelConfig now surfaces ErrAmbiguousImport for that shape instead of the opaque "no importer matched" error. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for vllm-omni importer Introduces the test surface for the forthcoming VLLMOmniImporter: detection via preferences.backend, Qwen owner + Omni repo token, URI-only fallback, negative cases (plain Qwen, random OmniX repo), and Import() emitting backend: vllm-omni with chat + multimodal usecases. Includes a registration-order assertion via DiscoverModelConfig to pin the requirement that vllm-omni wins over vllm for Qwen Omni repos (tokenizer files are usually present too). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add vllm-omni importer Adds VLLMOmniImporter for Qwen Omni-style multimodal checkpoints (Qwen3-Omni, Qwen2.5-Omni, …). Detection is narrow: HF owner "Qwen" combined with "omni" in the repo name, or a repo name matching the -Omni-/Omni- naming pattern. preferences.backend="vllm-omni" always wins; HFOwnerRepoFromURI provides a URI-only fallback for the hfapi recursion-bug edge case. Emitted YAML sets backend: vllm-omni and known_usecases: [chat, multimodal], matching the gallery/index.yaml vllm-omni entries. The importer is registered ahead of VLLMImporter so Qwen Omni repos — which also carry tokenizer files — route to vllm-omni rather than the plain vllm backend. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for llama-cpp drop-in preferences Pins the expected drop-in replacement behaviour: preferences.backend of ik-llama-cpp or turboquant must swap the emitted YAML backend field while keeping the llama-cpp file layout identical. Also covers the unknown-backend case (must stay llama-cpp) and re-asserts AdditionalBackends() returns the two curated entries with non-empty descriptions. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): llama-cpp honours ik-llama-cpp and turboquant drop-in preferences preferences.backend set to ik-llama-cpp or turboquant now swaps the emitted YAML backend field while leaving the file layout, model path, mmproj handling and everything else in the llama-cpp Import pipeline untouched. Unknown values are ignored and fall back to backend: llama-cpp so arbitrary input can't leak into the config. Aligns the AdditionalBackends() descriptions with the user-facing naming conventions surfaced via /backends/known. No changes to the pref-only curated list in endpoints/localai/backend.go: the two drop-in names have always lived on the importer side via AdditionalBackends. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for silero-vad importer Add the SileroVADImporter test fixtures covering metadata, preference overrides, snakers4 + onnx detection, silero_vad.onnx canonical filename, URI fallback, and live HF discovery. Implementation follows in the next commit. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add silero-vad importer Recognise the Silero VAD ONNX packaging: the canonical silero_vad.onnx filename or any ONNX file under the snakers4 owner. Emits a backend: silero-vad config with the vad known_usecase, and attaches the canonical file entry when present so the weights download on import. Registered before the generic importers so the unique-filename signal takes precedence over any downstream tokenizer-based matcher. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for rerankers importer Cover the RerankersImporter contract: interface metadata, preference override, cross-encoder owner detection, case-insensitive 'reranker' substring match (BAAI/bge-reranker, Alibaba-NLP/gte-reranker), URI fallback, and the full-discovery ordering check that a BAAI reranker repo must route to the rerankers importer rather than transformers. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add rerankers importer Recognise reranker repositories — cross-encoder owner or any repo whose name contains 'reranker' (case-insensitive). Emits backend: rerankers with reranking: true and the rerank known_usecase. Registered ahead of sentencetransformers and transformers so reranker repos that happen to ship tokenizer.json or modules.json still route here. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for sentencetransformers importer Cover the SentenceTransformersImporter contract: interface metadata, preference override, modules.json marker file, sentence_bert_config.json marker file, sentence-transformers owner, URI fallback, and the full-discovery ordering check that ensures a sentence-transformers HF URI routes here rather than transformers. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add sentencetransformers importer Recognise sentence-transformers embedding repos by modules.json, sentence_bert_config.json, or the sentence-transformers owner. Emits backend: sentencetransformers with embeddings: true and the embeddings known_usecase. Registered ahead of transformers so ST repos that carry tokenizer.json still route here. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for rfdetr importer Cover the RFDetrImporter contract: interface metadata, preference override, case-insensitive rf-detr and rfdetr substring matches, URI fallback, and negative cases. Implementation follows in the next commit. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add rfdetr importer Recognise RF-DETR object-detection repositories by a case-insensitive 'rf-detr' / 'rfdetr' substring in the repo name. Emits backend: rfdetr with the detection known_usecase. Registered ahead of transformers so RF-DETR repos with tokenizer artefacts still route here. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): surface ErrAmbiguousImport on sentence-similarity misses Add an ambiguity fixture covering the embeddings/rerankers modality. Qdrant/bm25 carries pipeline_tag=sentence-similarity but ships only config.json + stopword .txt files — none of the Batch 5 importers (silero-vad, rerankers, sentencetransformers, rfdetr) or the generic vllm/transformers/llama-cpp/mlx/diffusers importers match. Because the modality is in the ambiguous whitelist, DiscoverModelConfig must surface ErrAmbiguousImport. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(localai/backend): red tests for KnownBackend.Installed flag Extend the /backends/known suite with three failing cases that pin down the forthcoming Installed field: JSON field presence on every entry, flipping to true when an importer-registered backend is also present on disk (and staying false for non-installed pref-only entries), and surfacing system-only backends with empty modality and AutoDetect=false. A small writeFakeSystemBackend helper plants a run.sh under the backends dir so gallery.ListSystemBackends recognises the fixture. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(schema,localai/backend): add Installed flag to KnownBackend Add an Installed bool to schema.KnownBackend and populate it from the /backends/known handler so the React import form can warn users that picking a not-yet-installed backend will trigger an automatic download on submit. Computation: after merging the importer registry, additional backends provider entries and the curated pref-only slice, the handler walks gallery.ListSystemBackends(systemState) and either flips the existing map entry's Installed flag to true (preserving modality / autodetect / description metadata) or inserts a bare {Installed:true} entry for system-only backends the importer layer doesn't know about. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(localai/import_model): structured ambiguous-import response Add red tests covering the extended ambiguity shape the React import form needs: - ImportModelURIEndpoint must return an HTTP 400 body that exposes the detected `modality` (normalised to the importer modality key, e.g. "tts" for pipeline_tag=text-to-speech) and a list of `candidates` (backend names filtered by modality, excluding text-LLM backends). - The importers package must surface a typed AmbiguousImportError so HTTP consumers can read Modality + Candidates without parsing the error string. errors.Is against the existing sentinel keeps working. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(localai/import_model): structured ambiguity response with modality + candidates DiscoverModelConfig now returns a typed AmbiguousImportError that carries the importer modality key, candidate backend names, the original URI, and the raw HF pipeline_tag. Its Is() preserves errors.Is(err, ErrAmbiguousImport) for legacy callers. The importer modality is pre-mapped from the HF pipeline_tag (automatic-speech-recognition → asr, text-to-speech → tts, etc) via PipelineTagToModality — surfaced as an exported helper so downstream consumers can avoid duplicating the table. CandidatesForModality filters the default importer registry plus AdditionalBackendsProvider drop-ins by modality, sorts deterministically, and is the single source of truth used by ImportModelURIEndpoint. ImportModelURIEndpoint now returns HTTP 400 with { error, detail, modality, candidates, hint } when ambiguity fires, letting the React form render a modality-scoped picker inline instead of a generic toast. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): manual pick badge + tooltip Red Playwright coverage for the preference-only → manual pick rename: - The Backend dropdown renders a "manual pick" badge on every option whose KnownBackend.auto_detect is false. - The badge carries a title attribute with hover-tooltip copy that explains auto-detect won't route to this backend. - Auto-detectable backends must NOT carry the badge. - The legacy " (preference-only)" suffix is gone from every label. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import): replace preference-only suffix with manual pick badge SearchableSelect option rows now support an optional badge field — a muted pill rendered to the right of the label with an optional title attribute for native hover tooltips. Plain text so screen readers read it alongside the option name. buildBackendOptions in ImportModel stops appending " (preference-only)" to the label and instead sets badge="manual pick" plus a descriptive tooltip on every option whose auto_detect is false. The Backend help text explains what "manual pick" means so users aren't left wondering about the badge. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): inline ambiguity picker Red Playwright coverage for Batch A2 — when the server returns a 400 ambiguity body, the form must render an inline alert instead of a toast, expose one clickable chip per candidate backend, and support both auto-resubmit on pick and silent dismiss. - Mocks /api/models/import-uri with the structured ambiguity body (error, detail, modality, candidates, hint). - On first click of Import, the alert is visible, carries modality-specific copy, and shows a chip per candidate. - Clicking a chip clears the alert, sets the Backend dropdown, and triggers a second POST to /api/models/import-uri. - Dismissing the alert leaves the Backend dropdown on Auto-detect — no implicit backend assignment. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): inline ambiguity alert with candidate chips Adds AmbiguityAlert — a soft, info-coloured card rendered above the URI input when the server returns a structured 400 with { modality, candidates }. Message is modality-aware (tts/asr/embeddings/image/ reranker/detection get purpose-written copy, everything else falls back to a generic template). Each candidate is a clickable chip that shows a download icon when /backends/known marks the backend as not yet installed, so users aren't surprised by an implicit install. ImportModel wires the alert to handleSimpleImport's error path: - api.handleResponse now attaches { status, body } to the thrown Error so pages can pattern-match on structured responses instead of string error messages. - handleSimpleImport detects `status === 400 && body.error === 'ambiguous import'` and flips into the inline-picker mode instead of toasting. - Clicking a chip sets prefs.backend and auto-resubmits (passing the picked backend as an override so setPrefs's asynchrony doesn't leak a stale value). - Dismissing clears the alert; changing the URI or the backend also clears it so a stale alert never sticks around. Test fixtures mock GET /backends/known + POST /models/import-uri so the Playwright specs don't depend on real network reachability. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): auto-install warning Red Playwright coverage for Batch A3 — when the user picks a backend whose KnownBackend.installed is false, the form must render a muted inline note under the Backend dropdown warning that submitting will download the backend first. Picking an installed backend or leaving Auto-detect selected must keep the note hidden. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): auto-install warning under backend dropdown When the user picks a backend whose KnownBackend.installed is false, render a muted inline note under the Backend dropdown's help text warning that submitting will download the backend first. The note lives inside the same form-group so it lines up with the existing hint text; it's hidden when Auto-detect is selected (the selected backend is unknowable at that point) or when the chosen backend is already on disk. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import): drop redundant section header, adjust icons, rename HF shortcut - Remove the "Import from URI" card-level <h2> — the page title already says "Import New Model" one row up, so the secondary header was duplicating information. - Swap the fa-star on "Common Preferences" for fa-sliders (stars imply favourites/ratings; this is just a preferences block) and move the Custom Preferences fa-sliders-h to fa-plus-circle so the two blocks read as distinct rather than as two sliders. - Rename the HF shortcut from "Search GGUF on HF" → "Browse models on HF" and drop the `search=gguf` filter on the linked URL. The import form now supports ~40 backends; hard-coding GGUF in the copy no longer matches the form's actual reach. - Pure polish — no behaviour change, covered by the existing Batch A Playwright suite. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): batch B — simple/power switch, options, tabs, dialog Adds a failing Playwright suite covering the full Batch B surface ahead of implementation: - B1: SimplePowerSwitch segmented control renders, toggles, persists to localStorage across reloads. - B2: Simple-mode Options disclosure is collapsed by default; expanding exposes only Backend, Model Name, Description (no quantizations, mmproj, model type, or custom prefs). - B3: Power mode has Preferences and YAML tabs with a persistent selection across reloads; URI/name/description typed in Simple carry over to Power; YAML tab swaps the primary action to Create. - B4: Switching Power -> Simple with a custom preference set triggers the 3-button confirmation dialog (Keep / Discard / Cancel) with the documented semantics. Tests fail against master — implementation lands in the following commits. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): add SimplePowerSwitch segmented control Replaces the previous "Advanced Mode / Simple Mode" toggle button in the page header with a two-segment control that flips between Simple and Power. The control reuses the existing .segmented CSS shared with the Sound page for visual consistency. Mode state is persisted to localStorage under `import-form-mode` so reloads land on the same view (default: simple). The boolean alias `isAdvancedMode` is retained internally to minimise diff — subsequent commits reshape the Simple and Power surfaces independently. Closes B1 from the Batch B Playwright suite. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): simple mode collapsible options, power tabs, switch dialog Completes the Batch B surface in a single structural pass so Simple and Power mode can evolve independently: Simple mode - URI input + Ambiguity alert + Import button, plus a collapsible "Options" disclosure that exposes ONLY Backend, Model Name, Description. Quantizations / MMProj / Model Type / Diffusers fields / Custom Preferences are no longer rendered in Simple mode. Power mode - In-page segmented "Preferences · YAML" tab strip. Active tab persists to localStorage under `import-form-power-tab`. - Preferences tab = the full existing preferences + custom prefs panel (no progressive disclosure yet — that's Batch D). - YAML tab = the existing CodeEditor. Primary button reads "Create" here, "Import Model" everywhere else. Switch dialog - Power -> Simple with non-default prefs (advanced pref keys set, any custom-pref key non-empty, or YAML edited away from the template) opens a 3-button dialog: Keep & switch / Discard & switch / Cancel. - Keep preserves all state. Discard resets prefs + customPrefs + YAML to defaults. Cancel leaves the user in Power mode. Page subtitle reflects the current surface (Simple, Power/Preferences, Power/YAML). Estimate banner renders everywhere except Power/YAML. Closes B2/B3/B4 from the Batch B Playwright suite. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): expand Options disclosure in Batch A tests Batch B hid the Backend dropdown behind a collapsible Options disclosure in Simple mode. The Batch A tests that exercise the dropdown directly (manual-pick badge, ambiguity chip sets the selected backend, auto- install warning) now click the disclosure toggle before asserting on dropdown contents. Test intent is unchanged. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import): strip decorative icons from field labels The preference panel had 12 Font Awesome icons decorating field labels (Backend, Model Name, Description, Quantizations, MMProj Quantizations, Model Type, Pipeline Type, Scheduler Type, Enable Parameters, Embeddings, CUDA, plus fa-link on Model URI). Every label screamed equally, flattening the visual hierarchy. Remove them. Keep icons where they carry meaning: page-level section headers, URI format guide entries, primary buttons, the Simple-mode Options disclosure, the ambiguity alert's fa-lightbulb, the auto-install note's fa-download, and the Estimated-requirements banner's fa-memory / fa-microchip / fa-download. No new behaviour, no layout / spacing changes beyond removing the orphaned icon margin. Playwright suite green. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): progressive disclosure of preference fields Cover the Batch D visibility matrix for Power > Preferences: Quantizations, MMProj Quantizations, and Model Type each render only for the backends that can consume them, stay visible when the backend is unset, and preserve any value the user already typed when toggled off and back on. Also pin the shrunk Description textarea at rows=2. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): progressive disclosure + shorter description textarea Gate Quantizations, MMProj Quantizations, and Model Type in the Power > Preferences tab so each field only renders for the backends that can actually consume it. Backend unset keeps everything visible. Hidden fields' state is preserved (the JSX wrapper is guarded, not the underlying prefs state) so users flipping backends back and forth don't lose input. Also shrink the Description textarea from rows=3 to rows=2 — it's shared between Simple Options and Power Preferences so the change applies to both. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): enter-to-submit in Simple mode Red test for Batch F3 — pressing Enter in the URI input must POST /models/import-uri, and Enter in the Description textarea must insert a newline without submitting the form. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): enter-to-submit in Simple mode Wrap the Simple-mode URI input + ambiguity alert + Options disclosure in a <form> whose onSubmit calls handleSimpleImport. Pressing Enter in the URI input (or any Simple-mode text input) now submits the import without having to move the mouse to the header button. The Description textarea keeps its native behaviour — Enter inserts a newline. A hidden submit button is included because the visible Import button lives outside the form in the page header; some browsers only fire implicit Enter-submit when the form contains a submit-capable element. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import,SearchableSelect,components): aria-hidden on decorative icons Every Font Awesome icon in the import form is decorative — its meaning is already conveyed by adjacent visible text. Adding aria-hidden="true" prevents screen readers from announcing the unicode glyph point as content. Covers ImportModel.jsx (all remaining <i> glyphs) and SearchableSelect.jsx (the trigger chevron). AmbiguityAlert and SimplePowerSwitch already set aria-hidden on their icons when the components landed in Batches A and B — no change needed there. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(SearchableSelect): responsive dropdown maxHeight + hover focus guard F2 — replace fixed pixel heights with min(pixel, vh) so the dropdown and its inner scroll region don't overflow short viewports. Outer container: 260px -> min(260px, 60vh); inner listbox: 200px -> min(200px, 50vh). Tall viewports still get the original pixel caps. F5 — short-circuit onMouseEnter when the hovered row is already the focused row. Avoids queueing a setFocusIndex call (and a render) for every mousemove inside the same item — the state would be identical. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import): aria-label on custom preference rows The Key / Value inputs and trash button in each Custom Preferences row previously relied on placeholder text alone. Placeholders are not accessible names — they vanish on input and screen readers do not announce them consistently. Add row-indexed aria-labels so assistive tech can distinguish "Preference key for row 1" from "row 2", and give the trash button an explicit "Remove this preference" label. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): modality chip row Red tests for Batch E — a horizontal modality chip row that filters the Backend dropdown by modality. Covers visibility in Simple-mode Options and Power/Preferences (and absence in Power/YAML), filter behaviour, mismatched-backend clearing with toast, ambiguity-alert auto-selection, and radiogroup keyboard navigation. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): add ModalityChips component + filter integration Horizontal chip row (Any, Text, Speech, TTS, Image, Embeddings, Rerankers, Detection, VAD) filters the Backend dropdown options to the selected modality. Default is Any — no filter, current behaviour. - New ModalityChips component (radiogroup pattern, roving tabindex, arrow-key navigation, Home/End). - buildBackendOptions now accepts an optional modalityFilter so grouped output is narrowed before rendering. - Chips render inside Simple-mode Options disclosure and Power > Preferences tab. Power > YAML stays unaffected. - Switching the filter drops a mismatched backend selection and surfaces a toast so the auto-clear is visible. - Ambiguity alerts auto-activate the matching chip so users see only relevant backends even if they dismiss the alert. Tightens the Batch E tests' option-matching to the label <span> so the "↵" keybind hint on the focused row doesn't break accessible-name lookups. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * fix(ui/import): rename Power to Advanced + stop URI-formats toggle from submitting form The "Supported URI Formats" disclosure button inside the Simple-mode form lacked an explicit type attribute, so it defaulted to type="submit". Every click triggered the form's onSubmit and surfaced the empty-URI validation toast ("Please enter a model URI"). Marking it type="button" lets it behave as a pure toggle. While here, rename the user-visible "Power" label to "Advanced" in the mode switch (button text + tooltip) and the Power-mode tab's aria-label, matching the term users actually expect. The internal mode key stays 'power' so tests, localStorage, and data-testid selectors are untouched. Assisted-by: Claude:claude-opus-4-7 * fix(system): fall back to cpu when meta backend lacks default capability Meta backends like vllm and sglang enumerate concrete variants for nvidia/amd/intel/cpu but omit a default: catch-all entry. On a no-GPU host the reported capability is "default", so the previous Capability() returned "default" unconditionally on a miss — IsCompatibleWith then saw no "default" key and filtered the meta out of AvailableBackends. The import flow's auto-install step then failed with "no backend found with name <meta>", contradicting the UI's promise that the backend would be downloaded on demand. Try the explicit "default" key first, then fall back to "cpu" before giving up. vllm now resolves to cpu-vllm on CPU-only Linux without touching the gallery YAML. Assisted-by: Claude:claude-opus-4-7	2026-04-22 22:42:37 +02:00
Ettore Di Giacinto	20baec77ab	feat(face-recognition): add insightface/onnx backend for 1:1 verify, 1:N identify, embedding, detection, analysis (#9480) ... * feat(face-recognition): add insightface backend for 1:1 verify, 1:N identify, embedding, detection, analysis Adds face recognition as a new first-class capability in LocalAI via the `insightface` Python backend, with a pluggable two-engine design so non-commercial (insightface model packs) and commercial-safe (OpenCV Zoo YuNet + SFace) models share the same gRPC/HTTP surface. New gRPC RPCs (backend/backend.proto): * FaceVerify(FaceVerifyRequest) returns FaceVerifyResponse * FaceAnalyze(FaceAnalyzeRequest) returns FaceAnalyzeResponse Existing Embedding and Detect RPCs are reused (face image in PredictOptions.Images / DetectOptions.src) for face embedding and face detection respectively. New HTTP endpoints under /v1/face/: * verify — 1:1 image pair same-person decision * analyze — per-face age + gender (emotion/race reserved) * register — 1:N enrollment; stores embedding in vector store * identify — 1:N recognition; detect → embed → StoresFind * forget — remove a registered face by opaque ID Service layer (core/services/facerecognition/) introduces a `Registry` interface with one in-memory `storeRegistry` impl backed by LocalAI's existing local-store gRPC vector backend. HTTP handlers depend on the interface, not on StoresSet/StoresFind directly, so a persistent PostgreSQL/pgvector implementation can be slotted in via a single constructor change in core/application (TODO marker in the package doc). New usecase flag FLAG_FACE_RECOGNITION; insightface is also wired into FLAG_DETECTION so /v1/detection works for face bounding boxes. Gallery (backend/index.yaml) ships three entries: * insightface-buffalo-l — SCRFD-10GF + ArcFace R50 + genderage (~326MB pre-baked; non-commercial research use only) * insightface-opencv — YuNet + SFace (~40MB pre-baked; Apache 2.0) * insightface-buffalo-s — SCRFD-500MF + MBF (runtime download; non-commercial) Python backend (backend/python/insightface/): * engines.py — FaceEngine protocol with InsightFaceEngine and OnnxDirectEngine; resolves model paths relative to the backend directory so the same gallery config works in docker-scratch and in the e2e-backends rootfs-extraction harness. * backend.py — gRPC servicer implementing Health, LoadModel, Status, Embedding, Detect, FaceVerify, FaceAnalyze. * install.sh — pre-bakes buffalo_l + OpenCV YuNet/SFace inside the backend directory so first-run is offline-clean (the final scratch image only preserves files under /<backend>/). * test.py — parametrized unit tests over both engines. Tests: * Registry unit tests (go test -race ./core/services/facerecognition/...) — in-memory fake grpc.Backend, table-driven, covers register/ identify/forget/error paths + concurrent access. * tests/e2e-backends/backend_test.go extended with face caps (face_detect, face_embed, face_verify, face_analyze); relative ordering + configurable verifyCeiling per engine. * Makefile targets: test-extra-backend-insightface-buffalo-l, -opencv, and the -all aggregate. * CI: .github/workflows/test-extra.yml gains tests-insightface-grpc, auto-triggered by changes under backend/python/insightface/. Docs: * docs/content/features/face-recognition.md — feature page with license table, quickstart (defaults to the commercial-safe model), models matrix, API reference, 1:N workflow, storage caveats. * Cross-refs in object-detection.md, stores.md, embeddings.md, and whats-new.md. * Contributor README at backend/python/insightface/README.md. Verified end-to-end: * buffalo_l: 6/6 specs (health, load, face_detect, face_embed, face_verify, face_analyze). * opencv: 5/5 specs (same minus face_analyze — SFace has no demographic head; correctly skipped via BACKEND_TEST_CAPS). Assisted-by: Claude:claude-opus-4-7 * fix(face-recognition): move engine selection to model gallery, collapse backend entries The previous commit put engine/model_pack options on backend gallery entries (`backend/index.yaml`). That was wrong — `GalleryBackend` (core/gallery/backend_types.go:32) has no `options` field, so the YAML decoder silently dropped those keys and all three "different insightface-*" backend entries resolved to the same container image with no distinguishing configuration. Correct split: * `backend/index.yaml` now has ONE `insightface` backend entry shipping the CPU + CUDA 12 container images. The Python backend bundles both the non-commercial insightface model packs (buffalo_l / buffalo_s) and the commercial-safe OpenCV Zoo weights (YuNet + SFace); the active engine is selected at LoadModel time via `options: ["engine:..."]`. * `gallery/index.yaml` gains three model entries — `insightface-buffalo-l`, `insightface-opencv`, `insightface-buffalo-s` — each setting the appropriate `overrides.backend` + `overrides.options` so installing one actually gives the user the intended engine. This matches how `rfdetr-base` lives in the model gallery against the `rfdetr` backend. The earlier e2e tests passed despite this bug because the Makefile targets pass `BACKEND_TEST_OPTIONS` directly to LoadModel via gRPC, bypassing any gallery resolution entirely. No code changes needed. Assisted-by: Claude:claude-opus-4-7 * feat(face-recognition): cover all supported models in the gallery + drop weight baking Follows up on the model-gallery split: adds entries for every model configuration either engine actually supports, and switches weight delivery from image-baked to LocalAI's standard gallery mechanism. Gallery now has seven `insightface-*` model entries (gallery/index.yaml): insightface (family) — non-commercial research use • buffalo-l (326MB) — SCRFD-10GF + ResNet50 + genderage, default • buffalo-m (313MB) — SCRFD-2.5GF + ResNet50 + genderage • buffalo-s (159MB) — SCRFD-500MF + MBF + genderage • buffalo-sc (16MB) — SCRFD-500MF + MBF, recognition only (no landmarks, no demographics — analyze returns empty attributes) • antelopev2 (407MB) — SCRFD-10GF + ResNet100@Glint360K + genderage OpenCV Zoo family — Apache 2.0 commercial-safe • opencv — YuNet + SFace fp32 (~40MB) • opencv-int8 — YuNet + SFace int8 (~12MB, ~3x smaller, faster on CPU) Model weights are no longer baked into the backend image. The image now ships only the Python runtime + libraries (~275MB content size, ~1.18GB disk vs ~1.21GB when weights were baked). Weights flow through LocalAI's gallery mechanism: * OpenCV variants list `files:` with ONNX URIs + SHA-256, so `local-ai models install insightface-opencv` pulls them into the models directory exactly like any other gallery-managed model. * insightface packs (upstream distributes .zip archives only, not individual ONNX files) auto-download on first LoadModel via FaceAnalysis' built-in machinery, rooted at the LocalAI models directory so they live alongside everything else — same pattern `rfdetr` uses with `inference.get_model()`. Backend changes (backend/python/insightface/): * backend.py — LoadModel propagates `ModelOptions.ModelPath` (the LocalAI models directory) to engines via a `_model_dir` hint. This replaces the earlier ModelFile-dirname approach; ModelPath is the canonical "models directory" variable set by the Go loader (pkg/model/initializers.go:144) and is always populated. * engines.py::_resolve_model_path — picks up `model_dir` and searches it (plus basename-in-model-dir) before falling back to the dev script-dir. This is how OnnxDirectEngine finds gallery-downloaded YuNet/SFace files by filename only. * engines.py::_flatten_insightface_pack — new helper that works around an upstream packaging inconsistency: buffalo_l/s/sc zips expand flat, but buffalo_m and antelopev2 zips wrap their ONNX files in a redundant `<name>/` directory. insightface's own loader looks one level too shallow and fails. We call `ensure_available()` explicitly, flatten if nested, then hand to FaceAnalysis. * engines.py::InsightFaceEngine.prepare — root-resolution order now includes the `_model_dir` hint so packs download into the LocalAI models directory by default. * install.sh — no longer pre-downloads any weights. Everything is gallery-managed now. * smoke.py (new) — parametrized smoke test that iterates over every gallery configuration, simulating the LocalAI install flow (creates a models dir, fetches OpenCV files with checksum verification, lets insightface auto-download its packs), then runs detect + embed + verify (+ analyze where supported) through the in-process BackendServicer. * test.py — OnnxDirectEngineTest no longer hardcodes `/models/opencv/` paths; downloads ONNX files to a temp dir at setUpClass time and passes ModelPath accordingly. Registry change (core/services/facerecognition/store_registry.go): * `dim=0` in NewStoreRegistry now means "accept whatever dimension arrives" — needed because the backend supports 512-d ArcFace/MBF and 128-d SFace via the same Registry. A non-zero dim still fails fast with ErrDimensionMismatch. * core/application plumbs `faceEmbeddingDim = 0`, explaining the rationale in the comment. Backend gallery description updated to reflect that the image carries no weights — it's just Python + engines. Smoke-tested all 7 configurations against the rebuilt image (with the flatten fix applied), exit 0: PASS: insightface-buffalo-l faces=6 dim=512 same-dist=0.000 PASS: insightface-buffalo-sc faces=6 dim=512 same-dist=0.000 PASS: insightface-buffalo-s faces=6 dim=512 same-dist=0.000 PASS: insightface-buffalo-m faces=6 dim=512 same-dist=0.000 PASS: insightface-antelopev2 faces=6 dim=512 same-dist=0.000 PASS: insightface-opencv faces=6 dim=128 same-dist=0.000 PASS: insightface-opencv-int8 faces=6 dim=128 same-dist=0.000 7/7 passed Assisted-by: Claude:claude-opus-4-7 * fix(face-recognition): pre-fetch OpenCV ONNX for e2e target; drop stale pre-baked claim CI regression from the previous commit: I moved OpenCV Zoo weight delivery to LocalAI's gallery `files:` mechanism, but the test-extra-backend-insightface-opencv target was still passing relative paths `detector_onnx:models/opencv/yunet.onnx` in BACKEND_TEST_OPTIONS. The e2e suite drives LoadModel directly over gRPC without going through the gallery, so those relative paths resolved to nothing and OpenCV's ONNXImporter failed: LoadModel failed: Failed to load face engine: OpenCV(4.13.0) ... Can't read ONNX file: models/opencv/yunet.onnx Fix: add an `insightface-opencv-models` prerequisite target that fetches the two ONNX files (YuNet + SFace) to a deterministic host cache at /tmp/localai-insightface-opencv-cache/, verifies SHA-256, and skips the download on re-runs. The opencv test target depends on it and passes absolute paths in BACKEND_TEST_OPTIONS, so the backend finds the files via its normal absolute-path resolution branch. Also refresh the buffalo_l comment: it no longer says "pre-baked" (nothing is — the pack auto-downloads from upstream's GitHub release on first LoadModel, same as in CI). Locally verified: `make test-extra-backend-insightface-opencv` passes 5/5 specs (health, load, face_detect, face_embed, face_verify). Assisted-by: Claude:claude-opus-4-7 * feat(face-recognition): add POST /v1/face/embed + correct /v1/embeddings docs The docs promised that /v1/embeddings returns face vectors when you send an image data-URI. That was never true: /v1/embeddings is OpenAI-compatible and text-only by contract — its handler goes through `core/backend/embeddings.go::ModelEmbedding`, which sets `predictOptions.Embeddings = s` (a string of TEXT to embed) and never populates `predictOptions.Images[]`. The Python backend's Embedding gRPC method does handle Images[] (that's how /v1/face/register reaches it internally via `backend.FaceEmbed`), but the HTTP embeddings endpoint wasn't wired to populate it. Rather than overload /v1/embeddings with image-vs-text detection — messy, and the endpoint is OpenAI-compatible by design — add a dedicated /v1/face/embed endpoint that wraps `backend.FaceEmbed` (already used internally by /v1/face/register and /v1/face/identify). Matches LocalAI's convention of a dedicated path per non-standard flow (/v1/rerank, /v1/detection, /v1/face/verify etc.). Response: { "embedding": [<dim> floats, L2-normed], "dim": int, // 512 for ArcFace R50 / MBF, 128 for SFace "model": "<name>" } Live-tested on the opencv engine: returns a 128-d L2-normalized vector (sum(x^2) = 1.0000). Sentinel in docs updated to note /v1/embeddings is text-only and point image users at /v1/face/embed instead. Assisted-by: Claude:claude-opus-4-7 * fix(http): map malformed image input + gRPC status codes to proper 4xx Image-input failures on LocalAI's single-image endpoints (/v1/detection, /v1/face/{verify,analyze,embed,register,identify}) have historically returned 500 — even when the client was the one who sent garbage. Classic example: you POST an "image" that isn't a URL, isn't a data-URI, and isn't a valid JPEG/PNG — the server shouldn't claim that's its fault. Two helpers land in core/http/endpoints/localai/images.go and every single-image handler is switched over: * decodeImageInput(s) Wraps utils.GetContentURIAsBase64 and turns any failure (invalid URL, not a data-URI, download error, etc.) into echo.NewHTTPError(400, "invalid image input: ..."). * mapBackendError(err) Inspects the gRPC status on a backend call error and maps: INVALID_ARGUMENT → 400 Bad Request NOT_FOUND → 404 Not Found FAILED_PRECONDITION → 412 Precondition Failed Unimplemented → 501 Not Implemented All other codes fall through unchanged (still 500). Before, my 1×1 PNG error-path test returned: HTTP 500 "rpc error: code = InvalidArgument desc = failed to decode one or both images" After: HTTP 400 "failed to decode one or both images" Scope-limited to the LocalAI single-image endpoints. The multi-modal paths (middleware/request.go, openresponses/responses.go, openai/realtime.go) intentionally log-and-skip individual media parts when decoding fails — different design intent (graceful degradation of a multi-part message), not a 400-worthy failure. Left untouched. Live-verified: every error case in /tmp/face_errors.py now returns 4xx with a meaningful message; the "image with no face (1x1 PNG)" case specifically went from 500 → 400. Assisted-by: Claude:claude-opus-4-7 * refactor(face-recognition): insightface packs go through gallery files:, drop FaceAnalysis Follows up on the discovery that LocalAI's gallery `files:` mechanism handles archives (zip, tar.gz, …) via mholt/archiver/v3 — the rhasspy piper voices use exactly this pattern. Insightface packs are zip archives, so we can now deliver them the same way every other gallery-managed model gets delivered: declaratively, checksum-verified, through LocalAI's standard download+extract pipeline. Two changes: 1. Gallery (gallery/index.yaml) — every insightface-* entry gains a `files:` list with the pack zip's URI + SHA-256. `local-ai models install insightface-buffalo-l` now fetches the zip, verifies the hash, and extracts it into the models directory. No more reliance on insightface's library-internal `ensure_available()` auto-download or its hardcoded `BASE_REPO_URL`. 2. InsightFaceEngine (backend/python/insightface/engines.py) — drops the FaceAnalysis wrapper and drives insightface's `model_zoo` directly. The ~50 lines FaceAnalysis provides — glob ONNX files, route each through `model_zoo.get_model()`, build a `{taskname: model}` dict, loop per-face at inference — are reimplemented in `InsightFaceEngine`. The actual inference classes (RetinaFace, ArcFaceONNX, Attribute, Landmark) are still insightface's — we only replicate the glue, so drift risk against upstream is minimal. Why drop FaceAnalysis: it hard-codes a `<root>/models/<name>/*.onnx` layout that doesn't match what LocalAI's zip extraction produces. LocalAI unpacks archives flat into `<models_dir>`. Upstream packs are inconsistent — buffalo_l/s/sc ship ONNX at the zip root (lands at `<models_dir>/*.onnx`), buffalo_m/antelopev2 wrap in a redundant `<name>/` dir (lands at `<models_dir>/<name>/*.onnx`). The new `_locate_insightface_pack` helper searches both locations plus legacy paths and returns whichever has ONNX files. Replaces the earlier `_flatten_insightface_pack` helper (which tried to fight FaceAnalysis's layout expectations; now we just find the files wherever they are). Net effect for users: install once via LocalAI's managed flow, weights live alongside every other model, progress shows in the jobs endpoint, no first-load network call. Same API surface, cleaner plumbing. Assisted-by: Claude:claude-opus-4-7 * fix(face-recognition): CI's insightface e2e path needs the pack pre-fetched The e2e suite drives LoadModel over gRPC without going through LocalAI's gallery flow, so the engine's `_model_dir` option (normally populated from ModelPath) is empty. Previously the insightface target relied on FaceAnalysis auto-download to paper over this, but we dropped FaceAnalysis in favor of direct model_zoo calls — so the buffalo_l target started failing at LoadModel with "no insightface pack found". Mirror the opencv target's pre-fetch pattern: download buffalo_sc.zip (same SHA as the gallery entry), extract it on the host, and pass `root:<dir>` so the engine locates the pack without needing ModelPath. Switched to buffalo_sc (smallest pack, ~16MB) to keep CI fast; it covers the same insightface engine code path as buffalo_l. Face analyze cap dropped since buffalo_sc has no age/gender head. Assisted-by: Claude:claude-opus-4-7[1m] * feat(face-recognition): surface face-recognition in advertised feature maps The six /v1/face/* endpoints were missing from every place LocalAI advertises its feature surface to clients: * api_instructions — the machine-readable capability index at GET /api/instructions. Added `face-recognition` as a dedicated instruction area with an intro that calls out the in-memory registry caveat and the /v1/face/embed vs /v1/embeddings split. * auth/permissions — added FeatureFaceRecognition constant, routed all six face endpoints through it so admins can gate them per-user like any other API feature. Default ON (matches the other API features). * React UI capabilities — CAP_FACE_RECOGNITION symbol mapped to FLAG_FACE_RECOGNITION. Declared only for now; the Face page is a follow-up (noted in the plan). Instruction count bumped 9 → 10; test updated. Assisted-by: Claude:claude-opus-4-7[1m] * docs(agents): capture advertising-surface steps in the endpoint guide Before this change, adding a new /v1/* endpoint reliably missed one or more of: the swagger @Tags annotation, the /api/instructions registry, the auth RouteFeatureRegistry, and the React UI CAP_* symbol. The endpoint would work but be invisible to API consumers, admins, and the UI — and nothing in the existing docs said to look in those places. Extend .agents/api-endpoints-and-auth.md with a new "Advertising surfaces" section covering all four surfaces (swagger tags, /api/ instructions, capabilities.js, docs/), and expand the closing checklist so it's impossible to ship a feature without visiting each one. Hoist a one-liner reminder into AGENTS.md's Quick Reference so agents skim it before diving in. Assisted-by: Claude:claude-opus-4-7[1m]	2026-04-22 21:55:41 +02:00
Ettore Di Giacinto	87e6de1989	feat: wire transcription for llama.cpp, add streaming support (#9353) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-14 16:13:40 +02:00
Ettore Di Giacinto	d67623230f	feat(vllm): parity with llama.cpp backend (#9328) ... * fix(schema): serialize ToolCallID and Reasoning in Messages.ToProto The ToProto conversion was dropping tool_call_id and reasoning_content even though both proto and Go fields existed, breaking multi-turn tool calling and reasoning passthrough to backends. * refactor(config): introduce backend hook system and migrate llama-cpp defaults Adds RegisterBackendHook/runBackendHooks so each backend can register default-filling functions that run during ModelConfig.SetDefaults(). Migrates the existing GGUF guessing logic into hooks_llamacpp.go, registered for both 'llama-cpp' and the empty backend (auto-detect). Removes the old guesser.go shim. * feat(config): add vLLM parser defaults hook and importer auto-detection Introduces parser_defaults.json mapping model families to vLLM tool_parser/reasoning_parser names, with longest-pattern-first matching. The vllmDefaults hook auto-fills tool_parser and reasoning_parser options at load time for known families, while the VLLMImporter writes the same values into generated YAML so users can review and edit them. Adds tests covering MatchParserDefaults, hook registration via SetDefaults, and the user-override behavior. * feat(vllm): wire native tool/reasoning parsers + chat deltas + logprobs - Use vLLM's ToolParserManager/ReasoningParserManager to extract structured output (tool calls, reasoning content) instead of reimplementing parsing - Convert proto Messages to dicts and pass tools to apply_chat_template - Emit ChatDelta with content/reasoning_content/tool_calls in Reply - Extract prompt_tokens, completion_tokens, and logprobs from output - Replace boolean GuidedDecoding with proper GuidedDecodingParams from Grammar - Add TokenizeString and Free RPC methods - Fix missing `time` import used by load_video() * feat(vllm): CPU support + shared utils + vllm-omni feature parity - Split vllm install per acceleration: move generic `vllm` out of requirements-after.txt into per-profile after files (cublas12, hipblas, intel) and add CPU wheel URL for cpu-after.txt - requirements-cpu.txt now pulls torch==2.7.0+cpu from PyTorch CPU index - backend/index.yaml: register cpu-vllm / cpu-vllm-development variants - New backend/python/common/vllm_utils.py: shared parse_options, messages_to_dicts, setup_parsers helpers (used by both vllm backends) - vllm-omni: replace hardcoded chat template with tokenizer.apply_chat_template, wire native parsers via shared utils, emit ChatDelta with token counts, add TokenizeString and Free RPCs, detect CPU and set VLLM_TARGET_DEVICE - Add test_cpu_inference.py: standalone script to validate CPU build with a small model (Qwen2.5-0.5B-Instruct) * fix(vllm): CPU build compatibility with vllm 0.14.1 Validated end-to-end on CPU with Qwen2.5-0.5B-Instruct (LoadModel, Predict, TokenizeString, Free all working). - requirements-cpu-after.txt: pin vllm to 0.14.1+cpu (pre-built wheel from GitHub releases) for x86_64 and aarch64. vllm 0.14.1 is the newest CPU wheel whose torch dependency resolves against published PyTorch builds (torch==2.9.1+cpu). Later vllm CPU wheels currently require torch==2.10.0+cpu which is only available on the PyTorch test channel with incompatible torchvision. - requirements-cpu.txt: bump torch to 2.9.1+cpu, add torchvision/torchaudio so uv resolves them consistently from the PyTorch CPU index. - install.sh: add --index-strategy=unsafe-best-match for CPU builds so uv can mix the PyTorch index and PyPI for transitive deps (matches the existing intel profile behaviour). - backend.py LoadModel: vllm >= 0.14 removed AsyncLLMEngine.get_model_config so the old code path errored out with AttributeError on model load. Switch to the new get_tokenizer()/tokenizer accessor with a fallback to building the tokenizer directly from request.Model. * fix(vllm): tool parser constructor compat + e2e tool calling test Concrete vLLM tool parsers override the abstract base's __init__ and drop the tools kwarg (e.g. Hermes2ProToolParser only takes tokenizer). Instantiating with tools= raised TypeError which was silently caught, leaving chat_deltas.tool_calls empty. Retry the constructor without the tools kwarg on TypeError — tools aren't required by these parsers since extract_tool_calls finds tool syntax in the raw model output directly. Validated with Qwen/Qwen2.5-0.5B-Instruct + hermes parser on CPU: the backend correctly returns ToolCallDelta{name='get_weather', arguments='{"location": "Paris, France"}'} in ChatDelta. test_tool_calls.py is a standalone smoke test that spawns the gRPC backend, sends a chat completion with tools, and asserts the response contains a structured tool call. * ci(backend): build cpu-vllm container image Add the cpu-vllm variant to the backend container build matrix so the image registered in backend/index.yaml (cpu-vllm / cpu-vllm-development) is actually produced by CI. Follows the same pattern as the other CPU python backends (cpu-diffusers, cpu-chatterbox, etc.) with build-type='' and no CUDA. backend_pr.yml auto-picks this up via its matrix filter from backend.yml. * test(e2e-backends): add tools capability + HF model name support Extends tests/e2e-backends to cover backends that: - Resolve HuggingFace model ids natively (vllm, vllm-omni) instead of loading a local file: BACKEND_TEST_MODEL_NAME is passed verbatim as ModelOptions.Model with no download/ModelFile. - Parse tool calls into ChatDelta.tool_calls: new "tools" capability sends a Predict with a get_weather function definition and asserts the Reply contains a matching ToolCallDelta. Uses UseTokenizerTemplate with OpenAI-style Messages so the backend can wire tools into the model's chat template. - Need backend-specific Options[]: BACKEND_TEST_OPTIONS lets a test set e.g. "tool_parser:hermes,reasoning_parser:qwen3" at LoadModel time. Adds make target test-extra-backend-vllm that: - docker-build-vllm - loads Qwen/Qwen2.5-0.5B-Instruct - runs health,load,predict,stream,tools with tool_parser:hermes Drops backend/python/vllm/test_{cpu_inference,tool_calls}.py — those standalone scripts were scaffolding used while bringing up the Python backend; the e2e-backends harness now covers the same ground uniformly alongside llama-cpp and ik-llama-cpp. * ci(test-extra): run vllm e2e tests on CPU Adds tests-vllm-grpc to the test-extra workflow, mirroring the llama-cpp and ik-llama-cpp gRPC jobs. Triggers when files under backend/python/vllm/ change (or on run-all), builds the local-ai vllm container image, and runs the tests/e2e-backends harness with BACKEND_TEST_MODEL_NAME=Qwen/Qwen2.5-0.5B-Instruct, tool_parser:hermes, and the tools capability enabled. Uses ubuntu-latest (no GPU) — vllm runs on CPU via the cpu-vllm wheel we pinned in requirements-cpu-after.txt. Frees disk space before the build since the docker image + torch + vllm wheel is sizeable. * fix(vllm): build from source on CI to avoid SIGILL on prebuilt wheel The prebuilt vllm 0.14.1+cpu wheel from GitHub releases is compiled with SIMD instructions (AVX-512 VNNI/BF16 or AMX-BF16) that not every CPU supports. GitHub Actions ubuntu-latest runners SIGILL when vllm spawns the model_executor.models.registry subprocess for introspection, so LoadModel never reaches the actual inference path. - install.sh: when FROM_SOURCE=true on a CPU build, temporarily hide requirements-cpu-after.txt so installRequirements installs the base deps + torch CPU without pulling the prebuilt wheel, then clone vllm and compile it with VLLM_TARGET_DEVICE=cpu. The resulting binaries target the host's actual CPU. - backend/Dockerfile.python: accept a FROM_SOURCE build-arg and expose it as an ENV so install.sh sees it during `make`. - Makefile docker-build-backend: forward FROM_SOURCE as --build-arg when set, so backends that need source builds can opt in. - Makefile test-extra-backend-vllm: call docker-build-vllm via a recursive $(MAKE) invocation so FROM_SOURCE flows through. - .github/workflows/test-extra.yml: set FROM_SOURCE=true on the tests-vllm-grpc job. Slower but reliable — the prebuilt wheel only works on hosts that share the build-time SIMD baseline. Answers 'did you test locally?': yes, end-to-end on my local machine with the prebuilt wheel (CPU supports AVX-512 VNNI). The CI runner CPU gap was not covered locally — this commit plugs that gap. * ci(vllm): use bigger-runner instead of source build The prebuilt vllm 0.14.1+cpu wheel requires SIMD instructions (AVX-512 VNNI/BF16) that stock ubuntu-latest GitHub runners don't support — vllm.model_executor.models.registry SIGILLs on import during LoadModel. Source compilation works but takes 30-40 minutes per CI run, which is too slow for an e2e smoke test. Instead, switch tests-vllm-grpc to the bigger-runner self-hosted label (already used by backend.yml for the llama-cpp CUDA build) — that hardware has the required SIMD baseline and the prebuilt wheel runs cleanly. FROM_SOURCE=true is kept as an opt-in escape hatch: - install.sh still has the CPU source-build path for hosts that need it - backend/Dockerfile.python still declares the ARG + ENV - Makefile docker-build-backend still forwards the build-arg when set Default CI path uses the fast prebuilt wheel; source build can be re-enabled by exporting FROM_SOURCE=true in the environment. * ci(vllm): install make + build deps on bigger-runner bigger-runner is a bare self-hosted runner used by backend.yml for docker image builds — it has docker but not the usual ubuntu-latest toolchain. The make-based test target needs make, build-essential (cgo in 'go test'), and curl/unzip (the Makefile protoc target downloads protoc from github releases). protoc-gen-go and protoc-gen-go-grpc come via 'go install' in the install-go-tools target, which setup-go makes possible. * ci(vllm): install libnuma1 + libgomp1 on bigger-runner The vllm 0.14.1+cpu wheel ships a _C C++ extension that dlopens libnuma.so.1 at import time. When the runner host doesn't have it, the extension silently fails to register its torch ops, so EngineCore crashes on init_device with: AttributeError: '_OpNamespace' '_C_utils' object has no attribute 'init_cpu_threads_env' Also add libgomp1 (OpenMP runtime, used by torch CPU kernels) to be safe on stripped-down runners. * feat(vllm): bundle libnuma/libgomp via package.sh The vllm CPU wheel ships a _C extension that dlopens libnuma.so.1 at import time; torch's CPU kernels in turn use libgomp.so.1 (OpenMP). Without these on the host, vllm._C silently fails to register its torch ops and EngineCore crashes with: AttributeError: '_OpNamespace' '_C_utils' object has no attribute 'init_cpu_threads_env' Rather than asking every user to install libnuma1/libgomp1 on their host (or every LocalAI base image to ship them), bundle them into the backend image itself — same pattern fish-speech and the GPU libs already use. libbackend.sh adds ${EDIR}/lib to LD_LIBRARY_PATH at run time so the bundled copies are picked up automatically. - backend/python/vllm/package.sh (new): copies libnuma.so.1 and libgomp.so.1 from the builder's multilib paths into ${BACKEND}/lib, preserving soname symlinks. Runs during Dockerfile.python's 'Run backend-specific packaging' step (which already invokes package.sh if present). - backend/Dockerfile.python: install libnuma1 + libgomp1 in the builder stage so package.sh has something to copy (the Ubuntu base image otherwise only has libgomp in the gcc dep chain). - test-extra.yml: drop the workaround that installed these libs on the runner host — with the backend image self-contained, the runner no longer needs them, and the test now exercises the packaging path end-to-end the way a production host would. * ci(vllm): disable tests-vllm-grpc job (heterogeneous runners) Both ubuntu-latest and bigger-runner have inconsistent CPU baselines: some instances support the AVX-512 VNNI/BF16 instructions the prebuilt vllm 0.14.1+cpu wheel was compiled with, others SIGILL on import of vllm.model_executor.models.registry. The libnuma packaging fix doesn't help when the wheel itself can't be loaded. FROM_SOURCE=true compiles vllm against the actual host CPU and works everywhere, but takes 30-50 minutes per run — too slow for a smoke test on every PR. Comment out the job for now. The test itself is intact and passes locally; run it via 'make test-extra-backend-vllm' on a host with the required SIMD baseline. Re-enable when: - we have a self-hosted runner label with guaranteed AVX-512 VNNI/BF16, or - vllm publishes a CPU wheel with a wider baseline, or - we set up a docker layer cache that makes FROM_SOURCE acceptable The detect-changes vllm output, the test harness changes (tests/ e2e-backends + tools cap), the make target (test-extra-backend-vllm), the package.sh and the Dockerfile/install.sh plumbing all stay in place.	2026-04-13 11:00:29 +02:00
Ettore Di Giacinto	706cf5d43c	feat(sam.cpp): add sam.cpp detection backend (#9288) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-09 21:49:11 +02:00
Ettore Di Giacinto	85be4ff03c	feat(api): add ollama compatibility (#9284) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-09 14:15:14 +02:00
Richard Palethorpe	557d0f0f04	feat(api): Allow coding agents to interactively discover how to control and configure LocalAI (#9084) ... Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-04-04 15:14:35 +02:00
Ettore Di Giacinto	b7e3589875	fix(anthropic): show null index when not present, default to 0 (#9225) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-04 15:13:17 +02:00
Ettore Di Giacinto	59108fbe32	feat: add distributed mode (#9124) ... * feat: add distributed mode (experimental) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix data races, mutexes, transactions Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactorings Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix events and tool stream in agent chat Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * use ginkgo Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(cron): compute correctly time boundaries avoiding re-triggering Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * enhancements, refactorings Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * do not flood of healthy checks Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * do not list obvious backends as text backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * tests fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring and consolidation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Drop redundant healthcheck Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * enhancements, refactorings Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-30 00:47:27 +02:00
walcz-de	00fcf6936c	fix: implement encoding_format=base64 for embeddings endpoint (#9135) ... The OpenAI Node.js SDK v4+ sends encoding_format=base64 by default. LocalAI previously ignored this parameter and always returned a float JSON array, causing a silent data corruption bug in any Node.js client (AnythingLLM Desktop, LangChain.js, LlamaIndex.TS, …): // What the client does when it expects base64 but receives a float array: Buffer.from(floatArray, 'base64') Node.js treats a non-string first argument as a byte array — each float32 value is truncated to a single byte — and Float32Array then reads those bytes as floats, yielding dims/4 values. Vector databases (Qdrant, pgvector, …) then create collections with the wrong dimension, causing all similarity searches to fail silently. e.g. granite-embedding-107m (384 dims) → 96 stored in Qdrant jina-embeddings-v3 (1024 dims) → 256 stored in Qdrant Changes: - core/schema/prediction.go: add EncodingFormat string field to PredictionOptions so the request parameter is parsed and available throughout the request pipeline - core/schema/openai.go: add EmbeddingBase64 string field to Item; add MarshalJSON so the "embedding" JSON key emits either []float32 or a base64 string depending on which field is populated — all other Item consumers (image, video endpoints) are unaffected - core/http/endpoints/openai/embeddings.go: add floatsToBase64() which packs a float32 slice as little-endian bytes and base64-encodes it; add embeddingItem() helper; both InputToken and InputStrings loops now honour encoding_format=base64 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-25 17:38:07 +01:00
Ettore Di Giacinto	031a36c995	feat: inferencing default, automatic tool parsing fallback and wire min_p (#9092) ... * feat: wire min_p Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: inferencing defaults Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(refactor): re-use iterative parser Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: generate automatically inference defaults from unsloth Instead of trying to re-invent the wheel and maintain here the inference defaults, prefer to consume unsloth ones, and contribute there as necessary. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: apply defaults also to models installed via gallery Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: be consistent and apply fallback to all endpoint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-22 00:57:15 +01:00
Ettore Di Giacinto	f7e8d9e791	feat(quantization): add quantization backend (#9096) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-22 00:56:34 +01:00
Ettore Di Giacinto	d9c1db2b87	feat: add (experimental) fine-tuning support with TRL (#9088) ... * feat: add fine-tuning endpoint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(experimental): add fine-tuning endpoint and TRL support This changeset defines new GRPC signatues for Fine tuning backends, and add TRL backend as initial fine-tuning engine. This implementation also supports exporting to GGUF and automatically importing it to LocalAI after fine-tuning. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * commit TRL backend, stop by killing process Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * move fine-tune to generic features Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * add evals, reorder menu Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fix tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-21 02:08:02 +01:00
Tv	a6d0e29eba	fix(openresponses): do not omit required field ORItemParam.Arguments (#9074) ... See #9047	2026-03-19 22:04:45 +01:00
Tv	8a0edd0809	Always populate ORItemParam.Summary (#9049) ... * fix(openresponses): do not omit required fields summary and id * fix(openresponses): ensure ORItemParam.Summary is never null Normalize Summary to an empty slice at serialization chokepoints (sendSSEEvent, bufferEvent, buildORResponse) so it always serializes as [] instead of null. Closes #9047	2026-03-18 08:45:46 +01:00
Ettore Di Giacinto	8818452d85	feat(ui): MCP Apps, mcp streaming and client-side support (#8947) ... * Revert "fix: Add timeout-based wait for model deletion completion (#8756)" This reverts commit 9e1b0d0c82. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: add mcp prompts and resources Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add client-side MCP Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): allow to authenticate MCP servers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add MCP Apps Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: update AGENTS Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: allow to collapse navbar, save state in storage Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add MCP button also to home page Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): populate string content Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-11 07:30:49 +01:00
Ettore Di Giacinto	a026277ab9	feat(mlx-distributed): add new MLX-distributed backend (#8801) ... * feat(mlx-distributed): add new MLX-distributed backend Add new MLX distributed backend with support for both TCP and RDMA for model sharding. This implementation ties in the discovery implementation already in place, and re-uses the same P2P mechanism for the TCP MLX-distributed inferencing. The Auto-parallel implementation is inspired by Exo's ones (who have been added to acknowledgement for the great work!) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * expose a CLI to facilitate backend starting Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: make manual rank0 configurable via model configs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add missing features from mlx backend Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Apply suggestion from @mudler Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2026-03-09 17:29:32 +01:00
BitToby	96efa4fce0	feat: add WebSocket mode support for the response api (#8676) ... * feat: add WebSocket mode support for the response api Signed-off-by: bittoby <218712309+bittoby@users.noreply.github.com> * test: add e2e tests for WebSocket Responses API Signed-off-by: bittoby <218712309+bittoby@users.noreply.github.com> --------- Signed-off-by: bittoby <218712309+bittoby@users.noreply.github.com>	2026-03-06 10:36:59 +00:00
Ettore Di Giacinto	983db7bedc	feat(ui): add model size estimation (#8684) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-02-28 23:03:47 +01:00
Copilot	3ac7301f31	Add sample_rate support to TTS API via post-processing resampling (#8650) ... * Initial plan * Add TTS sample_rate support via AudioResample post-processing Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>	2026-02-25 16:36:27 +01:00
Lukas Schaefer	ed0bfb8732	fix: rename json_verbose to verbose_json (#8627) ... Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>	2026-02-23 17:57:06 +00:00
Ettore Di Giacinto	53276d28e7	feat(musicgen): add ace-step and UI interface (#8396) ... * feat(musicgen): add ace-step and UI interface Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Correctly handle model dir Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Drop auto-download Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add to models, fixup UIs icons Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Update docs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * l4t13 is incompatbile Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * avoid pinning version for cuda12 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Drop l4t12 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-02-05 12:04:53 +01:00
Dream	10a1e6c74d	feat(whisperx): add whisperx backend for transcription with speaker diarization (#8299) ... * feat(proto): add speaker field to TranscriptSegment for diarization Add speaker field to the gRPC TranscriptSegment message and map it through the Go schema, enabling backends to return speaker labels. Signed-off-by: eureka928 <meobius123@gmail.com> * feat(whisperx): add whisperx backend for transcription with diarization Add Python gRPC backend using WhisperX for speech-to-text with word-level timestamps, forced alignment, and speaker diarization via pyannote-audio when HF_TOKEN is provided. Signed-off-by: eureka928 <meobius123@gmail.com> * feat(whisperx): register whisperx backend in Makefile Signed-off-by: eureka928 <meobius123@gmail.com> * feat(whisperx): add whisperx meta and image entries to index.yaml Signed-off-by: eureka928 <meobius123@gmail.com> * ci(whisperx): add build matrix entries for CPU, CUDA 12/13, and ROCm Signed-off-by: eureka928 <meobius123@gmail.com> * fix(whisperx): unpin torch versions and use CPU index for cpu requirements Address review feedback: - Use --extra-index-url for CPU torch wheels to reduce size - Remove torch version pins, let uv resolve compatible versions Signed-off-by: eureka928 <meobius123@gmail.com> * fix(whisperx): pin torch ROCm variant to fix CI build failure Signed-off-by: eureka928 <meobius123@gmail.com> * fix(whisperx): pin torch CPU variant to fix uv resolution failure Pin torch==2.8.0+cpu so uv resolves the CPU wheel from the extra index instead of picking torch==2.8.0+cu128 from PyPI, which pulls unresolvable CUDA dependencies. Signed-off-by: eureka928 <meobius123@gmail.com> * fix(whisperx): use unsafe-best-match index strategy to fix uv resolution failure uv's default first-match strategy finds torch on PyPI before checking the extra index, causing it to pick torch==2.8.0+cu128 instead of the CPU variant. This makes whisperx's transitive torch dependency unresolvable. Using unsafe-best-match lets uv consider all indexes. Signed-off-by: eureka928 <meobius123@gmail.com> * fix(whisperx): drop +cpu local version suffix to fix uv resolution failure PEP 440 ==2.8.0 matches 2.8.0+cpu from the extra index, avoiding the issue where uv cannot locate an explicit +cpu local version specifier. This aligns with the pattern used by all other CPU backends. Signed-off-by: eureka928 <meobius123@gmail.com> * fix(backends): drop +rocm local version suffixes from hipblas requirements to fix uv resolution uv cannot resolve PEP 440 local version specifiers (e.g. +rocm6.4, +rocm6.3) in pinned requirements. The --extra-index-url already points to the correct ROCm wheel index and --index-strategy unsafe-best-match (set in libbackend.sh) ensures the ROCm variant is preferred. Applies the same fix as 7f5d72e8 (which resolved this for +cpu) across all 14 hipblas requirements files. Signed-off-by: eureka928 <meobius123@gmail.com> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: eureka928 <meobius123@gmail.com> * revert: scope hipblas suffix fix to whisperx only Reverts changes to non-whisperx hipblas requirements files per maintainer review — other backends are building fine with the +rocm local version suffix. Signed-off-by: eureka928 <meobius123@gmail.com> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: eureka928 <meobius123@gmail.com> --------- Signed-off-by: eureka928 <meobius123@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-02 16:33:12 +01:00
Andres	b6459ddd57	feat(api): Add transcribe response format request parameter & adjust STT backends (#8318) ... * WIP response format implementation for audio transcriptions (cherry picked from commit e271dd764bbc13846accf3beb8b6522153aa276f) Signed-off-by: Andres Smith <andressmithdev@pm.me> * Rework transcript response_format and add more formats (cherry picked from commit 6a93a8f63e2ee5726bca2980b0c9cf4ef8b7aeb8) Signed-off-by: Andres Smith <andressmithdev@pm.me> * Add test and replace go-openai package with official openai go client (cherry picked from commit f25d1a04e46526429c89db4c739e1e65942ca893) Signed-off-by: Andres Smith <andressmithdev@pm.me> * Fix faster-whisper backend and refactor transcription formatting to also work on CLI Signed-off-by: Andres Smith <andressmithdev@pm.me> (cherry picked from commit 69a93977d5e113eb7172bd85a0f918592d3d2168) Signed-off-by: Andres Smith <andressmithdev@pm.me> --------- Signed-off-by: Andres Smith <andressmithdev@pm.me> Co-authored-by: nanoandrew4 <nanoandrew4@gmail.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2026-02-01 17:33:17 +01:00